[DISCUSSION] Support alter schema for complex types

[akshay_nuthala]

Hi everyone,

Currently carbon only supports altering schema for primitive types.
We can extend it to complex columns too -
1) alter add complex column(array/struct)
2) alter drop complex column(array/struct)
3) alter change datatype/rename complex column(array/struct)

Here phase-1 focuses only on non-nested complex columns.
Phase-2 will later target nested ones.

Please let me know your valid inputs about the same.
Following is the link to the design document.

https://docs.google.com/document/d/1DhhkVXM8rMvOuKDZeccJpFEfO3VkA9C0c7JHCV88NXI/edit

Thanks,
Akshay




--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/

[Indhumathi]

+1. It is an important feature needs to be supported.

Since, you have mentioned alter drop struct child column, are you gonna
handle below scenarios also in phase 1?
1. support rename/change struct primitive child members ?
2. add new primitive child members in an existing struct ?

If so, please update design for those DDL commands.

Regards,
Indhumathi M



--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/

[akashrn5]

Hi,

+1 for feature. Thanks for proposing it as now most of the use case from
user perspective involves complex columns.

I have reviewed the doc and given comments, please work on it, then can be
reviewed again.

Regards,
Akash R



--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/

[akshay_nuthala]

No, these and other nested level operations will be taken care in the next
phase.



--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/

[kunalkapoor]

+1

On Fri, Mar 26, 2021 at 6:19 PM akshay_nuthala <[hidden email]>
wrote:

> No, these and other nested level operations will be taken care in the next
> phase.
>
>
>
> --
> Sent from:
> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
>

[Ajantha Bhat]

Hi Akshay,
The mail description and document content are not matching. For
single-level struct also document says cannot support.
So, please list down all the work that need to be done in points and
then divide which is supported in phase1 and which is supported in phase 2
clearly in the summary section of the document.

Also in the query flow, after adding the column, for previously loaded
segments what will be the output NULL or empty complex type ?
you can refer hive behavior for this.  Hope schema evolution (column drift)
also intact with complex column support.

Thanks,
Ajantha

On Tue, Mar 30, 2021 at 11:18 AM Kunal Kapoor <[hidden email]>
wrote:

> +1
>
> On Fri, Mar 26, 2021 at 6:19 PM akshay_nuthala <[hidden email]>
> wrote:
>
> > No, these and other nested level operations will be taken care in the
> next
> > phase.
> >
> >
> >
> > --
> > Sent from:
> > http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
> >
>

[maheshrajus]

Hi,

+1 For the feature. Agreed with Ajantha.

Thanks & Regards
Mahesh Raju Somalaraju

On Thu, Mar 18, 2021 at 10:00 PM akshay_nuthala <[hidden email]>
wrote:

> Hi everyone,
>
> Currently carbon only supports altering schema for primitive types.
> We can extend it to complex columns too -
> 1) alter add complex column(array/struct)
> 2) alter drop complex column(array/struct)
> 3) alter change datatype/rename complex column(array/struct)
>
> Here phase-1 focuses only on non-nested complex columns.
> Phase-2 will later target nested ones.
>
> Please let me know your valid inputs about the same.
> Following is the link to the design document.
>
>
> https://docs.google.com/document/d/1DhhkVXM8rMvOuKDZeccJpFEfO3VkA9C0c7JHCV88NXI/edit
>
> Thanks,
> Akshay
>
>
>
>
> --
> Sent from:
> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
>

[akshay_nuthala]

Handled comments



--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/