[GitHub] [carbondata] Indhumathi27 opened a new pull request #3714: [WIP] Skip Validate partition info in Indexserver count star flow

[GitBox]

Indhumathi27 opened a new pull request #3714: [WIP] Skip Validate partition info in Indexserver count star flow
URL: https://github.com/apache/carbondata/pull/3714
 
 
    ### Why is this PR needed?
   
   
    ### What changes were proposed in this PR?
   
       
    ### Does this PR introduce any user interface change?
    - No
    - Yes. (please explain the change and update document)
   
    ### Is any new testcase added?
    - No
    - Yes
   
       
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3714: [WIP] Skip Validate partition info in Indexserver count star flow
URL: https://github.com/apache/carbondata/pull/3714#issuecomment-614484075
 
 
   Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1039/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3714: [WIP] Skip Validate partition info in Indexserver count star flow
URL: https://github.com/apache/carbondata/pull/3714#issuecomment-614484559
 
 
   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2752/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3714: [CARBONDATA-3773] Skip Validate partition info in Indexserver count star flow
URL: https://github.com/apache/carbondata/pull/3714#issuecomment-614579089
 
 
   Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1040/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3714: [CARBONDATA-3773] Skip Validate partition info in Indexserver count star flow
URL: https://github.com/apache/carbondata/pull/3714#issuecomment-614579309
 
 
   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2753/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3714: [CARBONDATA-3773] Skip Validate partition info in Indexserver count star flow
URL: https://github.com/apache/carbondata/pull/3714#issuecomment-615366557
 
 
   Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1057/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3714: [CARBONDATA-3773] Skip Validate partition info in Indexserver count star flow
URL: https://github.com/apache/carbondata/pull/3714#issuecomment-615382731
 
 
   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2770/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

jackylk commented on a change in pull request #3714: [CARBONDATA-3773] Skip Validate partition info in Indexserver count star flow
URL: https://github.com/apache/carbondata/pull/3714#discussion_r410554671
 
 

 ##########
 File path: core/src/main/java/org/apache/carbondata/core/index/TableIndex.java
 ##########
 @@ -591,14 +591,14 @@ public void onEvent(Event event, OperationContext opContext) {
    * @throws IOException
    */
   public Map<String, Long> getBlockRowCount(List<Segment> allsegments,
-      final List<PartitionSpec> partitions, TableIndex defaultIndex)
+      final List<PartitionSpec> partitions, TableIndex defaultIndex, boolean isIUDFlow)
 
 Review comment:
   It is not good idea to add such parameter, which hides the actual behavior of this function
   
   Can you refactor it

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3714: [CARBONDATA-3773] Skip Validate partition info in Indexserver count star flow
URL: https://github.com/apache/carbondata/pull/3714#issuecomment-615557946
 
 
   Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1063/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3714: [CARBONDATA-3773] Skip Validate partition info in Indexserver count star flow
URL: https://github.com/apache/carbondata/pull/3714#issuecomment-615558235
 
 
   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2776/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3714: [CARBONDATA-3773] Skip Validate partition info in Indexserver count star flow
URL: https://github.com/apache/carbondata/pull/3714#issuecomment-615559089
 
 
   Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1064/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3714: [CARBONDATA-3773] Skip Validate partition info in Indexserver count star flow
URL: https://github.com/apache/carbondata/pull/3714#issuecomment-615559314
 
 
   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2777/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

jackylk commented on issue #3714: [CARBONDATA-3773] Skip Validate partition info in Indexserver count star flow
URL: https://github.com/apache/carbondata/pull/3714#issuecomment-615559727
 
 
   much better now, please fix the comment

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3714: [CARBONDATA-3773] Skip Validate partition info in Indexserver count star flow
URL: https://github.com/apache/carbondata/pull/3714#issuecomment-615657225
 
 
   Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1065/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3714: [CARBONDATA-3773] Skip Validate partition info in Indexserver count star flow
URL: https://github.com/apache/carbondata/pull/3714#issuecomment-615683118
 
 
   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2778/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

Indhumathi27 commented on a change in pull request #3714: [CARBONDATA-3773] Skip Validate partition info in Indexserver count star flow
URL: https://github.com/apache/carbondata/pull/3714#discussion_r410664981
 
 

 ##########
 File path: core/src/main/java/org/apache/carbondata/core/index/TableIndex.java
 ##########
 @@ -591,14 +591,14 @@ public void onEvent(Event event, OperationContext opContext) {
    * @throws IOException
    */
   public Map<String, Long> getBlockRowCount(List<Segment> allsegments,
-      final List<PartitionSpec> partitions, TableIndex defaultIndex)
+      final List<PartitionSpec> partitions, TableIndex defaultIndex, boolean isIUDFlow)
 
 Review comment:
   done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

jackylk commented on a change in pull request #3714: [CARBONDATA-3773] Skip Validate partition info in Indexserver count star flow
URL: https://github.com/apache/carbondata/pull/3714#discussion_r410669065
 
 

 ##########
 File path: core/src/main/java/org/apache/carbondata/core/index/TableIndex.java
 ##########
 @@ -595,6 +595,29 @@ public void onEvent(Event event, OperationContext opContext) {
       throws IOException {
     List<Segment> segments = getCarbonSegments(allsegments);
     Map<String, Long> blockletToRowCountMap = new HashMap<>();
+    for (Segment segment : segments) {
+      List<CoarseGrainIndex> indexes = defaultIndex.getIndexFactory().getIndexes(segment);
+      for (CoarseGrainIndex index : indexes) {
+        if (null != partitions) {
+          // if it has partitioned index but there is no partitioned information stored, it means
+          // partitions are dropped so return empty list.
+          if (!index.validatePartitionInfo(partitions)) {
+            return new HashMap<>();
+          }
+        }
+        index.getRowCountForEachBlock(segment, partitions, blockletToRowCountMap);
+      }
+    }
+    return blockletToRowCountMap;
+  }
+
+  /**
+   * Prune the index of the given segments and return the Map of blocklet path and row count
+   */
+  public Map<String, Long> getRowCountForBlock(List<Segment> allsegments,
 
 Review comment:
   Please give a more proper name and modify comment

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

Indhumathi27 commented on a change in pull request #3714: [CARBONDATA-3773] Skip Validate partition info in Indexserver count star flow
URL: https://github.com/apache/carbondata/pull/3714#discussion_r410675473
 
 

 ##########
 File path: core/src/main/java/org/apache/carbondata/core/index/TableIndex.java
 ##########
 @@ -595,6 +595,29 @@ public void onEvent(Event event, OperationContext opContext) {
       throws IOException {
     List<Segment> segments = getCarbonSegments(allsegments);
     Map<String, Long> blockletToRowCountMap = new HashMap<>();
+    for (Segment segment : segments) {
+      List<CoarseGrainIndex> indexes = defaultIndex.getIndexFactory().getIndexes(segment);
+      for (CoarseGrainIndex index : indexes) {
+        if (null != partitions) {
+          // if it has partitioned index but there is no partitioned information stored, it means
+          // partitions are dropped so return empty list.
+          if (!index.validatePartitionInfo(partitions)) {
+            return new HashMap<>();
+          }
+        }
+        index.getRowCountForEachBlock(segment, partitions, blockletToRowCountMap);
+      }
+    }
+    return blockletToRowCountMap;
+  }
+
+  /**
+   * Prune the index of the given segments and return the Map of blocklet path and row count
+   */
+  public Map<String, Long> getRowCountForBlock(List<Segment> allsegments,
 
 Review comment:
   done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3714: [CARBONDATA-3773] Skip Validate partition info in Indexserver count star flow
URL: https://github.com/apache/carbondata/pull/3714#issuecomment-615849253
 
 
   Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1072/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3714: [CARBONDATA-3773] Skip Validate partition info in Indexserver count star flow
URL: https://github.com/apache/carbondata/pull/3714#issuecomment-615852334
 
 
   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2785/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services