[GitHub] [carbondata] ajantha-bhat opened a new pull request #3654: [CARBONDATA-3734] Fix insert to partition table when parition column is in sort column

[GitBox]

ajantha-bhat opened a new pull request #3654: [CARBONDATA-3734] Fix insert to partition table when parition column is in sort column
URL: https://github.com/apache/carbondata/pull/3654
 
 
    ### Why is this PR needed?
    When partition column is a sort column.
   
   a) currently sort columns won't be the head of attributes, so need to add logic as per that, as partition columns will be in the end for global sort.
   
   b) While rearranging the data fields at executor, need to keep partition column in the end even though it is in sort column.
   
    ### What changes were proposed in this PR?
   prepare sort columns for global sort based on the table sort columns.
   Keep partition in the end, even though it is in sort columns for data field attributes
   
    ### Does this PR introduce any user interface change?
    - No
   
    ### Is any new testcase added?
    - No (already this scenario present in mv test case)
       
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3654: [CARBONDATA-3734] Fix insert failure on partition table when parition column is in sort column
URL: https://github.com/apache/carbondata/pull/3654#issuecomment-593966689
 
 
   Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/588/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3654: [CARBONDATA-3734] Fix insert failure on partition table when parition column is in sort column
URL: https://github.com/apache/carbondata/pull/3654#issuecomment-594008203
 
 
   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2294/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

jackylk commented on a change in pull request #3654: [CARBONDATA-3734] Fix insert failure on partition table when parition column is in sort column
URL: https://github.com/apache/carbondata/pull/3654#discussion_r388763834
 
 

 ##########
 File path: integration/spark/src/main/scala/org/apache/spark/sql/execution/command/management/CommonLoadUtils.scala
 ##########
 @@ -503,7 +503,20 @@ object CommonLoadUtils {
       if (numPartitions <= 0) {
         numPartitions = partitionsLen
       }
-      val sortColumns = attributes.take(table.getSortColumns.size())
+      val sortColumns =
+      if (table.isHivePartitionTable) {
+        // In case of partition column as sort column, attribute will not have it in the front.
+        // so need to look up the attribute and prepare
+        var sortColsAttr: Seq[AttributeReference] = Seq.empty
 
 Review comment:
   use ArrayBuffer instead of Seq to make it mutable

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

ajantha-bhat commented on a change in pull request #3654: [CARBONDATA-3734] Fix insert failure on partition table when parition column is in sort column
URL: https://github.com/apache/carbondata/pull/3654#discussion_r388805463
 
 

 ##########
 File path: integration/spark/src/main/scala/org/apache/spark/sql/execution/command/management/CommonLoadUtils.scala
 ##########
 @@ -503,7 +503,20 @@ object CommonLoadUtils {
       if (numPartitions <= 0) {
         numPartitions = partitionsLen
       }
-      val sortColumns = attributes.take(table.getSortColumns.size())
+      val sortColumns =
+      if (table.isHivePartitionTable) {
+        // In case of partition column as sort column, attribute will not have it in the front.
+        // so need to look up the attribute and prepare
+        var sortColsAttr: Seq[AttributeReference] = Seq.empty
 
 Review comment:
   ok. done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3654: [CARBONDATA-3734] Fix insert failure on partition table when parition column is in sort column
URL: https://github.com/apache/carbondata/pull/3654#issuecomment-595724290
 
 
   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2355/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3654: [CARBONDATA-3734] Fix insert failure on partition table when parition column is in sort column
URL: https://github.com/apache/carbondata/pull/3654#issuecomment-595727040
 
 
   Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/651/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

jackylk commented on issue #3654: [CARBONDATA-3734] Fix insert failure on partition table when parition column is in sort column
URL: https://github.com/apache/carbondata/pull/3654#issuecomment-597017497
 
 
   LGTM

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

asfgit closed pull request #3654: [CARBONDATA-3734] Fix insert failure on partition table when parition column is in sort column
URL: https://github.com/apache/carbondata/pull/3654
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services