[GitHub] [carbondata] ajantha-bhat opened a new pull request #3924: [WIP] Allow SI creation on first dimension column

[GitBox]

ajantha-bhat opened a new pull request #3924:
URL: https://github.com/apache/carbondata/pull/3924


    ### Why is this PR needed?
   
   
    ### What changes were proposed in this PR?
   
       
    ### Does this PR introduce any user interface change?
    - No
    - Yes. (please explain the change and update document)
   
    ### Is any new testcase added?
    - No
    - Yes
   
       
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3924:
URL: https://github.com/apache/carbondata/pull/3924#issuecomment-691886172


   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4058/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3924:
URL: https://github.com/apache/carbondata/pull/3924#issuecomment-691889246


   Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2320/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

ajantha-bhat commented on pull request #3924:
URL: https://github.com/apache/carbondata/pull/3924#issuecomment-691960921


   @jack86596 : You can give me more scenario example to add if you have for this.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

ajantha-bhat commented on pull request #3924:
URL: https://github.com/apache/carbondata/pull/3924#issuecomment-693325597


   @QiangCai : please check and merge this


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

QiangCai commented on pull request #3924:
URL: https://github.com/apache/carbondata/pull/3924#issuecomment-706004725


   I don't understand the advantage of this change.
   maybe we need to improve the use logic of both SI and the main index.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

QiangCai commented on pull request #3924:
URL: https://github.com/apache/carbondata/pull/3924#issuecomment-706004725


   I don't understand the advantage of this change.
   maybe we need to improve the use logic of both SI and the main index.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

ajantha-bhat commented on pull request #3924:
URL: https://github.com/apache/carbondata/pull/3924#issuecomment-706872243


   retest this please


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

jack86596 commented on pull request #3924:
URL: https://github.com/apache/carbondata/pull/3924#issuecomment-706907380


   The main idea of this PR is to improve the performance of following query: select * from table where col1 = 'xxx' or col2 = 'yyy'; where sort_columns order is col1,...,col2. Currently we can only create SI on col2, but if only 1 column SI is created another is not, above query will still just scan the main table directly, will not use col2 SI, leads to bad performance.
   So we have two options to improve this kind of query:
   1. create SI on col1 also, this is the solution of this PR
   2. if only one SI is created, still use this SI, other columns use main index to prune, this will be done later.
   
   And for no_sort table, first column is not sorted either, main index is worse than SI, under this scenario, we need this PR to create SI on the first column.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

ajantha-bhat commented on pull request #3924:
URL: https://github.com/apache/carbondata/pull/3924#issuecomment-706911344


   @jack86596 : Thanks for adding the user problem.
   @QiangCai : The scenario is mentioned above, if we don't create the SI on first column, the task launched for scanning main table is more (as the data files in main table is more), so if we support SI for first column and other column the task launched is very less. which will improve concurrent performance.
   
   Also I agree that if user creates SI with just first column, may be SI is no use. But when multiple column SI creation, This can be useful.
   
   Please check and merge.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3924:
URL: https://github.com/apache/carbondata/pull/3924#issuecomment-706932861


   Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2619/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3924:
URL: https://github.com/apache/carbondata/pull/3924#issuecomment-706945830


   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4369/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

QiangCai commented on pull request #3924:
URL: https://github.com/apache/carbondata/pull/3924#issuecomment-707080395


   LGTM


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

asfgit closed pull request #3924:
URL: https://github.com/apache/carbondata/pull/3924


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]