[GitHub] [carbondata] nihal0107 opened a new pull request #4132: [WIP] Fixed insert failure when partition column present in local sort scope

[GitBox]

nihal0107 opened a new pull request #4132:
URL: https://github.com/apache/carbondata/pull/4132


    ### Why is this PR needed?
   
   
    ### What changes were proposed in this PR?
   
       
    ### Does this PR introduce any user interface change?
    - No
    - Yes. (please explain the change and update document)
   
    ### Is any new testcase added?
    - No
    - Yes
   
       
   


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4132:
URL: https://github.com/apache/carbondata/pull/4132#issuecomment-839569543


   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5343/
   


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4132:
URL: https://github.com/apache/carbondata/pull/4132#issuecomment-839571855


   Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/3598/
   


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4132:
URL: https://github.com/apache/carbondata/pull/4132#issuecomment-839636531


   Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/3600/
   


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

nihal0107 commented on pull request #4132:
URL: https://github.com/apache/carbondata/pull/4132#issuecomment-839675661


   retest this please


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4132:
URL: https://github.com/apache/carbondata/pull/4132#issuecomment-839676485


   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5345/
   


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4132:
URL: https://github.com/apache/carbondata/pull/4132#issuecomment-839729852


   Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/3601/
   


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4132:
URL: https://github.com/apache/carbondata/pull/4132#issuecomment-839731885


   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5346/
   


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

ajantha-bhat commented on a change in pull request #4132:
URL: https://github.com/apache/carbondata/pull/4132#discussion_r634252117



##########
File path: processing/src/main/java/org/apache/carbondata/processing/sort/sortdata/SortParameters.java
##########
@@ -532,7 +556,7 @@ public static SortParameters createSortParameters(CarbonDataLoadConfiguration co
       parameters.setNoDictSortDataType(noDictSortAndNoSortDataTypes.get("noDictSortDataTypes"));
       parameters.setNoDictNoSortDataType(noDictSortAndNoSortDataTypes.get("noDictNoSortDataTypes"));
       // keep partition columns in the end for table spec by getting rearranged tale spec
-      TableSpec tableSpec = new TableSpec(configuration.getTableSpec().getCarbonTable(), true);
+      TableSpec tableSpec = new TableSpec(configuration.getTableSpec().getCarbonTable(), false);

Review comment:
       a) Inside TableSpec, If the true part is dead code, please remove it.
   b) remove the above comments, it's not applicable anymore.
   c) Please check all the test case with `carbon.enable.bad.record.handling.for.insert` true also.
     partition column, same sort column (all the type, measure & dim) [ local sort, global, no sort]
     partition column without sort columns
     partition with other sort columns (all the type, measure & dim) [ local sort, global, no sort]




--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

ajantha-bhat commented on a change in pull request #4132:
URL: https://github.com/apache/carbondata/pull/4132#discussion_r635772654



##########
File path: processing/src/main/java/org/apache/carbondata/processing/sort/sortdata/SortParameters.java
##########
@@ -532,7 +556,7 @@ public static SortParameters createSortParameters(CarbonDataLoadConfiguration co
       parameters.setNoDictSortDataType(noDictSortAndNoSortDataTypes.get("noDictSortDataTypes"));
       parameters.setNoDictNoSortDataType(noDictSortAndNoSortDataTypes.get("noDictNoSortDataTypes"));
       // keep partition columns in the end for table spec by getting rearranged tale spec
-      TableSpec tableSpec = new TableSpec(configuration.getTableSpec().getCarbonTable(), true);
+      TableSpec tableSpec = new TableSpec(configuration.getTableSpec().getCarbonTable(), false);

Review comment:
       Some more scenario is that we can check like,
   iv) more than one partition column (one int, one string) and more than one sort column for above scenarios.
   v) static partition and dynamic partition together




--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

ajantha-bhat commented on a change in pull request #4132:
URL: https://github.com/apache/carbondata/pull/4132#discussion_r634252117



##########
File path: processing/src/main/java/org/apache/carbondata/processing/sort/sortdata/SortParameters.java
##########
@@ -532,7 +556,7 @@ public static SortParameters createSortParameters(CarbonDataLoadConfiguration co
       parameters.setNoDictSortDataType(noDictSortAndNoSortDataTypes.get("noDictSortDataTypes"));
       parameters.setNoDictNoSortDataType(noDictSortAndNoSortDataTypes.get("noDictNoSortDataTypes"));
       // keep partition columns in the end for table spec by getting rearranged tale spec
-      TableSpec tableSpec = new TableSpec(configuration.getTableSpec().getCarbonTable(), true);
+      TableSpec tableSpec = new TableSpec(configuration.getTableSpec().getCarbonTable(), false);

Review comment:
       a) Inside TableSpec, If the true part is dead code, please remove it.
   b) remove the above comments, it's not applicable anymore.
   c) Please check all the test case with `carbon.enable.bad.record.handling.for.insert` true also.
     i) partition column, same sort column (all the type, measure & dim) [ local sort, global, no sort]
     ii) partition column without sort columns
     iii) partition with other sort columns (all the type, measure & dim) [ local sort, global, no sort]




--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4132:
URL: https://github.com/apache/carbondata/pull/4132#issuecomment-848667256


   Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/3696/
   


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4132:
URL: https://github.com/apache/carbondata/pull/4132#issuecomment-848667879


   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5441/
   


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4132:
URL: https://github.com/apache/carbondata/pull/4132#issuecomment-848746379


   Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/3697/
   


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4132:
URL: https://github.com/apache/carbondata/pull/4132#issuecomment-848748645


   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5442/
   


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

nihal0107 commented on a change in pull request #4132:
URL: https://github.com/apache/carbondata/pull/4132#discussion_r640303241



##########
File path: processing/src/main/java/org/apache/carbondata/processing/sort/sortdata/SortParameters.java
##########
@@ -532,7 +556,7 @@ public static SortParameters createSortParameters(CarbonDataLoadConfiguration co
       parameters.setNoDictSortDataType(noDictSortAndNoSortDataTypes.get("noDictSortDataTypes"));
       parameters.setNoDictNoSortDataType(noDictSortAndNoSortDataTypes.get("noDictNoSortDataTypes"));
       // keep partition columns in the end for table spec by getting rearranged tale spec
-      TableSpec tableSpec = new TableSpec(configuration.getTableSpec().getCarbonTable(), true);
+      TableSpec tableSpec = new TableSpec(configuration.getTableSpec().getCarbonTable(), false);

Review comment:
       Rework is done and tested above-mentioned scenario. It is working fine now. @ajantha-bhat please review.




--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

kunal642 commented on a change in pull request #4132:
URL: https://github.com/apache/carbondata/pull/4132#discussion_r642823135



##########
File path: processing/src/main/java/org/apache/carbondata/processing/sort/sortdata/SortParameters.java
##########
@@ -532,7 +556,7 @@ public static SortParameters createSortParameters(CarbonDataLoadConfiguration co
       parameters.setNoDictSortDataType(noDictSortAndNoSortDataTypes.get("noDictSortDataTypes"));
       parameters.setNoDictNoSortDataType(noDictSortAndNoSortDataTypes.get("noDictNoSortDataTypes"));
       // keep partition columns in the end for table spec by getting rearranged tale spec
-      TableSpec tableSpec = new TableSpec(configuration.getTableSpec().getCarbonTable(), true);
+      TableSpec tableSpec = new TableSpec(configuration.getTableSpec().getCarbonTable(), false);

Review comment:
       @nihal0107 i dont see test cases for many of the scenarios mentioned.




--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

nihal0107 commented on a change in pull request #4132:
URL: https://github.com/apache/carbondata/pull/4132#discussion_r642835796



##########
File path: processing/src/main/java/org/apache/carbondata/processing/sort/sortdata/SortParameters.java
##########
@@ -532,7 +556,7 @@ public static SortParameters createSortParameters(CarbonDataLoadConfiguration co
       parameters.setNoDictSortDataType(noDictSortAndNoSortDataTypes.get("noDictSortDataTypes"));
       parameters.setNoDictNoSortDataType(noDictSortAndNoSortDataTypes.get("noDictNoSortDataTypes"));
       // keep partition columns in the end for table spec by getting rearranged tale spec
-      TableSpec tableSpec = new TableSpec(configuration.getTableSpec().getCarbonTable(), true);
+      TableSpec tableSpec = new TableSpec(configuration.getTableSpec().getCarbonTable(), false);

Review comment:
       For some of the scenario test cases were already present like:
   a) partition column without sort columns
   b) partition with other sort columns (all the type, measure & dim) [ local sort, global, no sort]
   c) static partition and dynamic partition together
   
   and test case which I have added covered for some of flow like:
   a) partition column, same sort column (all the type, measure & dim) [ local sort, global, no sort]
   b) more than one partition column (one int, one string) and more than one sort column for above scenarios.
   




--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

kunal642 commented on a change in pull request #4132:
URL: https://github.com/apache/carbondata/pull/4132#discussion_r642933824



##########
File path: processing/src/main/java/org/apache/carbondata/processing/sort/sortdata/SortParameters.java
##########
@@ -532,7 +556,7 @@ public static SortParameters createSortParameters(CarbonDataLoadConfiguration co
       parameters.setNoDictSortDataType(noDictSortAndNoSortDataTypes.get("noDictSortDataTypes"));
       parameters.setNoDictNoSortDataType(noDictSortAndNoSortDataTypes.get("noDictNoSortDataTypes"));
       // keep partition columns in the end for table spec by getting rearranged tale spec
-      TableSpec tableSpec = new TableSpec(configuration.getTableSpec().getCarbonTable(), true);
+      TableSpec tableSpec = new TableSpec(configuration.getTableSpec().getCarbonTable(), false);

Review comment:
       ok
   




--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

kunal642 commented on pull request #4132:
URL: https://github.com/apache/carbondata/pull/4132#issuecomment-851973452


   LGTM


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]