[New Feature] Alter table support in carbondata

[manishgupta88]

Hi All,

Currently in carbondata after a table is created once it becomes immutable.
Deletion or addition of column is not supported because of which the same
table and data cannot be used again. To add more flexibility to the
carbondata system alter table support needs to be added to carbondata
system.


Please refer the design document at below location and share your
suggestions.


https://drive.google.com/file/d/0B1DnrpMgGOu9dGxYMzVJUXEyZG8/view?usp=sharing


Regards

Manish Gupta

[Liang Chen]

Hi

Thanks for you started this discussion for alter table feature.

A couple of comments:
1.For "change of data type" , whether only support from INT to BIGINT, or
not ?
2.Whether support adjust the order of columns for MDK , and make compaction
to resort data as per the new order of columns , or not ?

Regards
Liang

2017-03-09 21:00 GMT+08:00 manish gupta <[hidden email]>:

> Hi All,
>
> Currently in carbondata after a table is created once it becomes immutable.
> Deletion or addition of column is not supported because of which the same
> table and data cannot be used again. To add more flexibility to the
> carbondata system alter table support needs to be added to carbondata
> system.
>
>
> Please refer the design document at below location and share your
> suggestions.
>
>
> https://drive.google.com/file/d/0B1DnrpMgGOu9dGxYMzVJUXEyZG8/
> view?usp=sharing
>
>
> Regards
>
> Manish Gupta
>

--
Regards
Liang

[David CaiQiang]

Hi manish,
    Better to keeping consistent with existing SQL syntax.

    ALTER TABLE [database_name].test_change CHANGE col_name col_name changed_column_type
   => ALTER TABLE [database_name].test_change ALTER/MODIFY col_name col_name changed_column_type

Best Regards
David QiangCai

[manishgupta88]

Hi QiangCai,

Syntax for change data type is kept similar to hive alter table change data
type syntax.

Please let me know your point of reference. I will look at it and modify if
required.

Regards
Manish Gupta

On Fri, Mar 10, 2017 at 7:26 AM, QiangCai <[hidden email]> wrote:

> Hi manish,
>     Better to keeping consistent with existing SQL syntax.
>
>     ALTER TABLE [database_name].test_change CHANGE col_name col_name
> changed_column_type
>    => ALTER TABLE [database_name].test_change ALTER/MODIFY col_name
> col_name
> changed_column_type
>
> Best Regards
> David QiangCai
>
>
>
> --
> View this message in context: http://apache-carbondata-
> mailing-list-archive.1130556.n5.nabble.com/New-Feature-
> Alter-table-support-in-carbondata-tp8542p8567.html
> Sent from the Apache CarbonData Mailing List archive mailing list archive
> at Nabble.com.
>

[manishgupta88]

Hi Liang,

Thanks for reviewing the document. Please find my answers inline.

1.For "change of data type" , whether only support from INT to BIGINT, or
not ?
- Yes as part of initial feature development we will support only
 upcasting the datatype  INT to BIGINT and decimal datatype change from
lower to higher precision.

2.Whether support adjust the order of columns for MDK , and make compaction
to resort data as per the new order of columns , or not ?
- Currently addition of column is supported only at the end. From my
opinion once this is stable, we can further extend it to support reordering
of columns to change the MDKEY order.

Regards
Manish Gupta

On Fri, Mar 10, 2017 at 5:20 AM, Liang Chen <[hidden email]> wrote:

> Hi
>
> Thanks for you started this discussion for alter table feature.
>
> A couple of comments:
> 1.For "change of data type" , whether only support from INT to BIGINT, or
> not ?
> 2.Whether support adjust the order of columns for MDK , and make compaction
> to resort data as per the new order of columns , or not ?
>
> Regards
> Liang
>
> 2017-03-09 21:00 GMT+08:00 manish gupta <[hidden email]>:
>
> > Hi All,
> >
> > Currently in carbondata after a table is created once it becomes
> immutable.
> > Deletion or addition of column is not supported because of which the same
> > table and data cannot be used again. To add more flexibility to the
> > carbondata system alter table support needs to be added to carbondata
> > system.
> >
> >
> > Please refer the design document at below location and share your
> > suggestions.
> >
> >
> > https://drive.google.com/file/d/0B1DnrpMgGOu9dGxYMzVJUXEyZG8/
> > view?usp=sharing
> >
> >
> > Regards
> >
> > Manish Gupta
> >
>
>
>
> --
> Regards
> Liang
>

[kumarvishal09]

Hi Manish,

Please add impact area for block pruning and data query.

-Regards
Kumar Vishal

On Fri, Mar 10, 2017 at 12:56 PM, manish gupta <[hidden email]>
wrote:

> Hi Liang,
>
> Thanks for reviewing the document. Please find my answers inline.
>
> 1.For "change of data type" , whether only support from INT to BIGINT, or
> not ?
> - Yes as part of initial feature development we will support only
>  upcasting the datatype  INT to BIGINT and decimal datatype change from
> lower to higher precision.
>
> 2.Whether support adjust the order of columns for MDK , and make compaction
> to resort data as per the new order of columns , or not ?
> - Currently addition of column is supported only at the end. From my
> opinion once this is stable, we can further extend it to support reordering
> of columns to change the MDKEY order.
>
> Regards
> Manish Gupta
>
> On Fri, Mar 10, 2017 at 5:20 AM, Liang Chen <[hidden email]>
> wrote:
>
> > Hi
> >
> > Thanks for you started this discussion for alter table feature.
> >
> > A couple of comments:
> > 1.For "change of data type" , whether only support from INT to BIGINT, or
> > not ?
> > 2.Whether support adjust the order of columns for MDK , and make
> compaction
> > to resort data as per the new order of columns , or not ?
> >
> > Regards
> > Liang
> >
> > 2017-03-09 21:00 GMT+08:00 manish gupta <[hidden email]>:
> >
> > > Hi All,
> > >
> > > Currently in carbondata after a table is created once it becomes
> > immutable.
> > > Deletion or addition of column is not supported because of which the
> same
> > > table and data cannot be used again. To add more flexibility to the
> > > carbondata system alter table support needs to be added to carbondata
> > > system.
> > >
> > >
> > > Please refer the design document at below location and share your
> > > suggestions.
> > >
> > >
> > > https://drive.google.com/file/d/0B1DnrpMgGOu9dGxYMzVJUXEyZG8/
> > > view?usp=sharing
> > >
> > >
> > > Regards
> > >
> > > Manish Gupta
> > >
> >
> >
> >
> > --
> > Regards
> > Liang
> >
>

[Venkata Gollamudi]

Hi Manish,

Thanks for working on this. Please raise Jira to track this.

Regards,
Ramana

On Fri, Mar 10, 2017 at 12:21 PM, Kumar Vishal <[hidden email]>
wrote:

> Hi Manish,
>
> Please add impact area for block pruning and data query.
>
> -Regards
> Kumar Vishal
>
> On Fri, Mar 10, 2017 at 12:56 PM, manish gupta <[hidden email]>
> wrote:
>
> > Hi Liang,
> >
> > Thanks for reviewing the document. Please find my answers inline.
> >
> > 1.For "change of data type" , whether only support from INT to BIGINT, or
> > not ?
> > - Yes as part of initial feature development we will support only
> >  upcasting the datatype  INT to BIGINT and decimal datatype change from
> > lower to higher precision.
> >
> > 2.Whether support adjust the order of columns for MDK , and make
> compaction
> > to resort data as per the new order of columns , or not ?
> > - Currently addition of column is supported only at the end. From my
> > opinion once this is stable, we can further extend it to support
> reordering
> > of columns to change the MDKEY order.
> >
> > Regards
> > Manish Gupta
> >
> > On Fri, Mar 10, 2017 at 5:20 AM, Liang Chen <[hidden email]>
> > wrote:
> >
> > > Hi
> > >
> > > Thanks for you started this discussion for alter table feature.
> > >
> > > A couple of comments:
> > > 1.For "change of data type" , whether only support from INT to BIGINT,
> or
> > > not ?
> > > 2.Whether support adjust the order of columns for MDK , and make
> > compaction
> > > to resort data as per the new order of columns , or not ?
> > >
> > > Regards
> > > Liang
> > >
> > > 2017-03-09 21:00 GMT+08:00 manish gupta <[hidden email]>:
> > >
> > > > Hi All,
> > > >
> > > > Currently in carbondata after a table is created once it becomes
> > > immutable.
> > > > Deletion or addition of column is not supported because of which the
> > same
> > > > table and data cannot be used again. To add more flexibility to the
> > > > carbondata system alter table support needs to be added to carbondata
> > > > system.
> > > >
> > > >
> > > > Please refer the design document at below location and share your
> > > > suggestions.
> > > >
> > > >
> > > > https://drive.google.com/file/d/0B1DnrpMgGOu9dGxYMzVJUXEyZG8/
> > > > view?usp=sharing
> > > >
> > > >
> > > > Regards
> > > >
> > > > Manish Gupta
> > > >
> > >
> > >
> > >
> > > --
> > > Regards
> > > Liang
> > >
> >
>

[manishgupta88]

Ok.

Regards
Manish Gupta

On Mon, Mar 13, 2017 at 3:51 PM, Venkata Gollamudi <[hidden email]>
wrote:

> Hi Manish,
>
> Thanks for working on this. Please raise Jira to track this.
>
> Regards,
> Ramana
>
> On Fri, Mar 10, 2017 at 12:21 PM, Kumar Vishal <[hidden email]>
> wrote:
>
> > Hi Manish,
> >
> > Please add impact area for block pruning and data query.
> >
> > -Regards
> > Kumar Vishal
> >
> > On Fri, Mar 10, 2017 at 12:56 PM, manish gupta <
> [hidden email]>
> > wrote:
> >
> > > Hi Liang,
> > >
> > > Thanks for reviewing the document. Please find my answers inline.
> > >
> > > 1.For "change of data type" , whether only support from INT to BIGINT,
> or
> > > not ?
> > > - Yes as part of initial feature development we will support only
> > >  upcasting the datatype  INT to BIGINT and decimal datatype change from
> > > lower to higher precision.
> > >
> > > 2.Whether support adjust the order of columns for MDK , and make
> > compaction
> > > to resort data as per the new order of columns , or not ?
> > > - Currently addition of column is supported only at the end. From my
> > > opinion once this is stable, we can further extend it to support
> > reordering
> > > of columns to change the MDKEY order.
> > >
> > > Regards
> > > Manish Gupta
> > >
> > > On Fri, Mar 10, 2017 at 5:20 AM, Liang Chen <[hidden email]>
> > > wrote:
> > >
> > > > Hi
> > > >
> > > > Thanks for you started this discussion for alter table feature.
> > > >
> > > > A couple of comments:
> > > > 1.For "change of data type" , whether only support from INT to
> BIGINT,
> > or
> > > > not ?
> > > > 2.Whether support adjust the order of columns for MDK , and make
> > > compaction
> > > > to resort data as per the new order of columns , or not ?
> > > >
> > > > Regards
> > > > Liang
> > > >
> > > > 2017-03-09 21:00 GMT+08:00 manish gupta <[hidden email]>:
> > > >
> > > > > Hi All,
> > > > >
> > > > > Currently in carbondata after a table is created once it becomes
> > > > immutable.
> > > > > Deletion or addition of column is not supported because of which
> the
> > > same
> > > > > table and data cannot be used again. To add more flexibility to the
> > > > > carbondata system alter table support needs to be added to
> carbondata
> > > > > system.
> > > > >
> > > > >
> > > > > Please refer the design document at below location and share your
> > > > > suggestions.
> > > > >
> > > > >
> > > > > https://drive.google.com/file/d/0B1DnrpMgGOu9dGxYMzVJUXEyZG8/
> > > > > view?usp=sharing
> > > > >
> > > > >
> > > > > Regards
> > > > >
> > > > > Manish Gupta
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Regards
> > > > Liang
> > > >
> > >
> >
>

[gururajshetty]

Hi Manish,

We need to document in details the usage scenario and the sytax for the
same. We can also provide examples on the usage.

So shall i raise a parallel JIRA for the document update?

Regards,
Gururaj


On Mon, Mar 13, 2017 at 4:10 PM, manish gupta <[hidden email]>
wrote:

> Ok.
>
> Regards
> Manish Gupta
>
> On Mon, Mar 13, 2017 at 3:51 PM, Venkata Gollamudi <[hidden email]>
> wrote:
>
> > Hi Manish,
> >
> > Thanks for working on this. Please raise Jira to track this.
> >
> > Regards,
> > Ramana
> >
> > On Fri, Mar 10, 2017 at 12:21 PM, Kumar Vishal <
> [hidden email]>
> > wrote:
> >
> > > Hi Manish,
> > >
> > > Please add impact area for block pruning and data query.
> > >
> > > -Regards
> > > Kumar Vishal
> > >
> > > On Fri, Mar 10, 2017 at 12:56 PM, manish gupta <
> > [hidden email]>
> > > wrote:
> > >
> > > > Hi Liang,
> > > >
> > > > Thanks for reviewing the document. Please find my answers inline.
> > > >
> > > > 1.For "change of data type" , whether only support from INT to
> BIGINT,
> > or
> > > > not ?
> > > > - Yes as part of initial feature development we will support only
> > > >  upcasting the datatype  INT to BIGINT and decimal datatype change
> from
> > > > lower to higher precision.
> > > >
> > > > 2.Whether support adjust the order of columns for MDK , and make
> > > compaction
> > > > to resort data as per the new order of columns , or not ?
> > > > - Currently addition of column is supported only at the end. From my
> > > > opinion once this is stable, we can further extend it to support
> > > reordering
> > > > of columns to change the MDKEY order.
> > > >
> > > > Regards
> > > > Manish Gupta
> > > >
> > > > On Fri, Mar 10, 2017 at 5:20 AM, Liang Chen <[hidden email]
> >
> > > > wrote:
> > > >
> > > > > Hi
> > > > >
> > > > > Thanks for you started this discussion for alter table feature.
> > > > >
> > > > > A couple of comments:
> > > > > 1.For "change of data type" , whether only support from INT to
> > BIGINT,
> > > or
> > > > > not ?
> > > > > 2.Whether support adjust the order of columns for MDK , and make
> > > > compaction
> > > > > to resort data as per the new order of columns , or not ?
> > > > >
> > > > > Regards
> > > > > Liang
> > > > >
> > > > > 2017-03-09 21:00 GMT+08:00 manish gupta <[hidden email]
> >:
> > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > Currently in carbondata after a table is created once it becomes
> > > > > immutable.
> > > > > > Deletion or addition of column is not supported because of which
> > the
> > > > same
> > > > > > table and data cannot be used again. To add more flexibility to
> the
> > > > > > carbondata system alter table support needs to be added to
> > carbondata
> > > > > > system.
> > > > > >
> > > > > >
> > > > > > Please refer the design document at below location and share your
> > > > > > suggestions.
> > > > > >
> > > > > >
> > > > > > https://drive.google.com/file/d/0B1DnrpMgGOu9dGxYMzVJUXEyZG8/
> > > > > > view?usp=sharing
> > > > > >
> > > > > >
> > > > > > Regards
> > > > > >
> > > > > > Manish Gupta
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Regards
> > > > > Liang
> > > > >
> > > >
> > >
> >
>

[manishgupta88]

Hi Guru,

Yes you can raise a jira for document updation.

Regards
Manish Gupta

On Tue, Mar 14, 2017 at 1:41 PM, Gururaj Shetty <[hidden email]>
wrote:

> Hi Manish,
>
> We need to document in details the usage scenario and the sytax for the
> same. We can also provide examples on the usage.
>
> So shall i raise a parallel JIRA for the document update?
>
> Regards,
> Gururaj
>
>
> On Mon, Mar 13, 2017 at 4:10 PM, manish gupta <[hidden email]>
> wrote:
>
> > Ok.
> >
> > Regards
> > Manish Gupta
> >
> > On Mon, Mar 13, 2017 at 3:51 PM, Venkata Gollamudi <
> [hidden email]>
> > wrote:
> >
> > > Hi Manish,
> > >
> > > Thanks for working on this. Please raise Jira to track this.
> > >
> > > Regards,
> > > Ramana
> > >
> > > On Fri, Mar 10, 2017 at 12:21 PM, Kumar Vishal <
> > [hidden email]>
> > > wrote:
> > >
> > > > Hi Manish,
> > > >
> > > > Please add impact area for block pruning and data query.
> > > >
> > > > -Regards
> > > > Kumar Vishal
> > > >
> > > > On Fri, Mar 10, 2017 at 12:56 PM, manish gupta <
> > > [hidden email]>
> > > > wrote:
> > > >
> > > > > Hi Liang,
> > > > >
> > > > > Thanks for reviewing the document. Please find my answers inline.
> > > > >
> > > > > 1.For "change of data type" , whether only support from INT to
> > BIGINT,
> > > or
> > > > > not ?
> > > > > - Yes as part of initial feature development we will support only
> > > > >  upcasting the datatype  INT to BIGINT and decimal datatype change
> > from
> > > > > lower to higher precision.
> > > > >
> > > > > 2.Whether support adjust the order of columns for MDK , and make
> > > > compaction
> > > > > to resort data as per the new order of columns , or not ?
> > > > > - Currently addition of column is supported only at the end. From
> my
> > > > > opinion once this is stable, we can further extend it to support
> > > > reordering
> > > > > of columns to change the MDKEY order.
> > > > >
> > > > > Regards
> > > > > Manish Gupta
> > > > >
> > > > > On Fri, Mar 10, 2017 at 5:20 AM, Liang Chen <
> [hidden email]
> > >
> > > > > wrote:
> > > > >
> > > > > > Hi
> > > > > >
> > > > > > Thanks for you started this discussion for alter table feature.
> > > > > >
> > > > > > A couple of comments:
> > > > > > 1.For "change of data type" , whether only support from INT to
> > > BIGINT,
> > > > or
> > > > > > not ?
> > > > > > 2.Whether support adjust the order of columns for MDK , and make
> > > > > compaction
> > > > > > to resort data as per the new order of columns , or not ?
> > > > > >
> > > > > > Regards
> > > > > > Liang
> > > > > >
> > > > > > 2017-03-09 21:00 GMT+08:00 manish gupta <
> [hidden email]
> > >:
> > > > > >
> > > > > > > Hi All,
> > > > > > >
> > > > > > > Currently in carbondata after a table is created once it
> becomes
> > > > > > immutable.
> > > > > > > Deletion or addition of column is not supported because of
> which
> > > the
> > > > > same
> > > > > > > table and data cannot be used again. To add more flexibility to
> > the
> > > > > > > carbondata system alter table support needs to be added to
> > > carbondata
> > > > > > > system.
> > > > > > >
> > > > > > >
> > > > > > > Please refer the design document at below location and share
> your
> > > > > > > suggestions.
> > > > > > >
> > > > > > >
> > > > > > > https://drive.google.com/file/d/0B1DnrpMgGOu9dGxYMzVJUXEyZG8/
> > > > > > > view?usp=sharing
> > > > > > >
> > > > > > >
> > > > > > > Regards
> > > > > > >
> > > > > > > Manish Gupta
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Regards
> > > > > > Liang
> > > > > >
> > > > >
> > > >
> > >
> >
>