[jira] [Updated] (CARBONDATA-3347) support SORT_COLUMNS modification

[Akash R Nilugal (Jira)]

     [ https://issues.apache.org/jira/browse/CARBONDATA-3347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

QiangCai updated CARBONDATA-3347:
---------------------------------
    Description:
*Background*

Now SORT_COLUMNS can’t be modified after the table is created. If we want to modify SORT_COLUMNS on this table, we need to create a new table and migrate data. If the data is huge, the migration will take a long time and even impact the user business.

SORT_SCOPE in table properties can be modified now. And we can specify new SORT_SCOPE during data loading. Carbon index file will mark whether this segment is sorted or not. So the different segments maybe have different SORT_SCOPE.

*Mo**tivation*

After the table is created, the user can adjust SORT_SCOPE/SORT_COLUMNS according to their business. History segments will still use old SORT_SCOPE/SORT_COLUMNS, but the user also can resort old segments one by one if need.

But we still suggest the user give a proper SORT_SCOPE/SORT_COLUMNS when they create the table because the modification will take many resources to resort data of old segments.

 

please check design doc for more detail.

[^sort_columns modification_v2.pdf]

  was:
*Background*

Now SORT_COLUMNS can’t be modified after the table is created. If we want to modify SORT_COLUMNS on this table, we need to create a new table and migrate data. If the data is huge, the migration will take a long time and even impact the user business.

SORT_SCOPE in table properties can be modified now. And we can specify new SORT_SCOPE during data loading. Carbon index file will mark whether this segment is sorted or not. So the different segments maybe have different SORT_SCOPE.

*Mo**tivation*

After the table is created, the user can adjust SORT_SCOPE/SORT_COLUMNS according to their business. History segments will still use old SORT_SCOPE/SORT_COLUMNS, but the user also can resort old segments one by one if need.

But we still suggest the user give a proper SORT_SCOPE/SORT_COLUMNS when they create the table because the modification will take many resources to resort data of old segments.

 

please check design doc for more detail.

[^sort_columns modification.pdf]

[^sort_columns modification_v2.pdf]

> support SORT_COLUMNS modification
> ---------------------------------
>
>                 Key: CARBONDATA-3347
>                 URL: https://issues.apache.org/jira/browse/CARBONDATA-3347
>             Project: CarbonData
>          Issue Type: New Feature
>          Components: spark-integration
>            Reporter: QiangCai
>            Assignee: QiangCai
>            Priority: Major
>         Attachments: sort_columns modification.pdf, sort_columns modification_v2.pdf
>
>
> *Background*
> Now SORT_COLUMNS can’t be modified after the table is created. If we want to modify SORT_COLUMNS on this table, we need to create a new table and migrate data. If the data is huge, the migration will take a long time and even impact the user business.
> SORT_SCOPE in table properties can be modified now. And we can specify new SORT_SCOPE during data loading. Carbon index file will mark whether this segment is sorted or not. So the different segments maybe have different SORT_SCOPE.
> *Mo**tivation*
> After the table is created, the user can adjust SORT_SCOPE/SORT_COLUMNS according to their business. History segments will still use old SORT_SCOPE/SORT_COLUMNS, but the user also can resort old segments one by one if need.
> But we still suggest the user give a proper SORT_SCOPE/SORT_COLUMNS when they create the table because the modification will take many resources to resort data of old segments.
>  
> please check design doc for more detail.
> [^sort_columns modification_v2.pdf]

--
This message was sent by Atlassian JIRA
(v7.6.3#76005)