[GitHub] [carbondata] nihal0107 opened a new pull request #4137: [CARBONDATA-4188] Fixed select query with small table page size after alter add column.

[GitBox]

nihal0107 opened a new pull request #4137:
URL: https://github.com/apache/carbondata/pull/4137


    ### Why is this PR needed?
   Select query on table with long string data type and small page size throws ArrayIndexOutOfBoudException after alter add columns.
   
   Query fails because after changing the schema, the number of rows set in bitsetGroup(RestructureIncludeFilterExecutorImpl.applyFilter()) for pages is not correct.
   
    ### What changes were proposed in this PR?
   Set the correct number of rows inside every page of bitsetGroup.
       
    ### Does this PR introduce any user interface change?
    - No
    - Yes. (please explain the change and update document)
   
    ### Is any new testcase added?
    - No
   
       
   


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4137:
URL: https://github.com/apache/carbondata/pull/4137#issuecomment-842201073


   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5379/
   


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4137:
URL: https://github.com/apache/carbondata/pull/4137#issuecomment-842203437


   Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/3634/
   


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4137:
URL: https://github.com/apache/carbondata/pull/4137#issuecomment-842392198


   Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/3636/
   


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4137:
URL: https://github.com/apache/carbondata/pull/4137#issuecomment-842392672


   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5381/
   


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4137:
URL: https://github.com/apache/carbondata/pull/4137#issuecomment-842201073






--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

Indhumathi27 commented on a change in pull request #4137:
URL: https://github.com/apache/carbondata/pull/4137#discussion_r634101068



##########
File path: core/src/main/java/org/apache/carbondata/core/scan/filter/executer/RestructureIncludeFilterExecutorImpl.java
##########
@@ -49,10 +48,15 @@ public RestructureIncludeFilterExecutorImpl(DimColumnResolvedFilterInfo dimColum
   @Override
   public BitSetGroup applyFilter(RawBlockletColumnChunks rawBlockletColumnChunks,
       boolean useBitsetPipeLine) {
-    int numberOfRows = rawBlockletColumnChunks.getDataBlock().numRows();
-    return FilterUtil.createBitSetGroupWithDefaultValue(
-        rawBlockletColumnChunks.getDataBlock().numberOfPages(),
-        numberOfRows, isDefaultValuePresentInFilterValues);
+    int pageCount = rawBlockletColumnChunks.getDataBlock().numberOfPages();

Review comment:
       Check all other filters also and handle




--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4137:
URL: https://github.com/apache/carbondata/pull/4137#issuecomment-842930344


   Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/3643/
   


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4137:
URL: https://github.com/apache/carbondata/pull/4137#issuecomment-842931177


   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5387/
   


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4137:
URL: https://github.com/apache/carbondata/pull/4137#issuecomment-843000460


   Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/3645/
   


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4137:
URL: https://github.com/apache/carbondata/pull/4137#issuecomment-843002244


   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5390/
   


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

nihal0107 commented on a change in pull request #4137:
URL: https://github.com/apache/carbondata/pull/4137#discussion_r634208120



##########
File path: core/src/main/java/org/apache/carbondata/core/scan/filter/executer/RestructureIncludeFilterExecutorImpl.java
##########
@@ -49,10 +48,15 @@ public RestructureIncludeFilterExecutorImpl(DimColumnResolvedFilterInfo dimColum
   @Override
   public BitSetGroup applyFilter(RawBlockletColumnChunks rawBlockletColumnChunks,
       boolean useBitsetPipeLine) {
-    int numberOfRows = rawBlockletColumnChunks.getDataBlock().numRows();
-    return FilterUtil.createBitSetGroupWithDefaultValue(
-        rawBlockletColumnChunks.getDataBlock().numberOfPages(),
-        numberOfRows, isDefaultValuePresentInFilterValues);
+    int pageCount = rawBlockletColumnChunks.getDataBlock().numberOfPages();

Review comment:
       done




--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

Indhumathi27 commented on pull request #4137:
URL: https://github.com/apache/carbondata/pull/4137#issuecomment-844173997


   LGTM, @kunal642 can review once


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

kunal642 commented on a change in pull request #4137:
URL: https://github.com/apache/carbondata/pull/4137#discussion_r635787217



##########
File path: core/src/test/java/org/apache/carbondata/core/scan/filter/FilterUtilTest.java
##########
@@ -223,17 +223,6 @@ public void testCheckIfRightExpressionRequireEvaluationWithExpressionIsInstanceO
             DataTypes.STRING) instanceof ColumnFilterInfo);
   }
 
-  @Test public void testCreateBitSetGroupWithDefaultValue() {

Review comment:
       Instead of removing, change the test case to use the new method




--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

nihal0107 commented on a change in pull request #4137:
URL: https://github.com/apache/carbondata/pull/4137#discussion_r635809496



##########
File path: core/src/test/java/org/apache/carbondata/core/scan/filter/FilterUtilTest.java
##########
@@ -223,17 +223,6 @@ public void testCheckIfRightExpressionRequireEvaluationWithExpressionIsInstanceO
             DataTypes.STRING) instanceof ColumnFilterInfo);
   }
 
-  @Test public void testCreateBitSetGroupWithDefaultValue() {

Review comment:
       done




--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4137:
URL: https://github.com/apache/carbondata/pull/4137#issuecomment-844848021


   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5406/
   


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4137:
URL: https://github.com/apache/carbondata/pull/4137#issuecomment-844858018


   Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/3662/
   


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

nihal0107 commented on pull request #4137:
URL: https://github.com/apache/carbondata/pull/4137#issuecomment-844858368


   retest this please.


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4137:
URL: https://github.com/apache/carbondata/pull/4137#issuecomment-844943242


   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5408/
   


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4137:
URL: https://github.com/apache/carbondata/pull/4137#issuecomment-844943992


   Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/3664/
   


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]