[Discussion] is it necessary to support SORT_COLUMNS modification

[David CaiQiang]

Hi all,
    Let's discuss whether it is necessary to support SORT_COLUMNS
modification.
    *Background*
    "SORT_COLUMNS" is a table level property, and we can't change it after
creating a table.
    *Motivation*
     When we want to optimize the query performance and found that it needs
to change SORT_COLUMNS, Carbon should support changing SORT_COLUMNS.
SORT_COLUMNS just like the main data index and impact the data layout.  At
the same time, we can re-sort old segment data by new SORT_COLUMNS.
     *Modification*
     1. loading data use table level "SORT_COLUMNS" and store it as a
segment level property
     2. query should use segment level property to read data files
     3. only compacting segments with same "SORT_COLUMNS"
     4. convert old segment one by one by new SORT_COLUMNS and refresh
DataMap if needed
     5. show segment command output SORT_COLUMNS  of each segment
   
    Any suggestions and questions?



-----
Best Regards
David Cai
--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/

[yaojinguo]

nice feature. I think range_column feature can benefit a lot from this
feature.



--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/

[xm_zzc]

Nice feature. Just one suggestion: we can support to convert a specified old
segment by new SORT_COLUMNS, for example: resort table table_name for
segment 0.



--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/

[David CaiQiang]

please check JIRA and find the design doc:
https://issues.apache.org/jira/browse/CARBONDATA-3347



-----
Best Regards
David Cai
--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/