[GitHub] carbondata pull request #1751: [WIP]Measure null

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1751
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1643/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1751
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2879/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1751
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1647/



---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1751#discussion_r162053147
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/indexstore/blockletindex/BlockletDataMap.java ---
    @@ -196,6 +201,10 @@ private DataMapRowImpl loadToUnsafe(DataFileFooter fileFooter,
               unsafeMemorySummaryDMStore.getSchema()[taskMinMaxOrdinal], maxValues,
               TASK_MAX_VALUES_INDEX, false);
           ordinal++;
    +      taskMinMaxOrdinal++;
    +
    +      summaryRow.setByteArray(minMaxIndex.getNullValues().toByteArray(), taskMinMaxOrdinal);
    --- End diff --
   
    We need to calculate null bits based on whole blocks and then set to the summary


---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1751#discussion_r162124619
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/scan/filter/executer/RowLevelRangeGrtrThanEquaToFilterExecuterImpl.java ---
    @@ -149,13 +154,17 @@ private boolean isScanRequired(byte[] blockMaxValue, byte[][] filterValues) {
         return isScanRequired;
       }
     
    -  private boolean isScanRequired(byte[] maxValue, Object[] filterValue,
    +  private boolean isScanRequired(byte[] maxValue, boolean nullValue, Object[] filterValue,
           DataType dataType) {
         Object value = DataTypeUtil.getMeasureObjectFromDataType(maxValue, dataType);
         for (int i = 0; i < filterValue.length; i++) {
           // TODO handle min and max for null values.
           if (filterValue[i] == null) {
    -        return true;
    +        if (nullValue == true) {
    +          return true;
    +        } else {
    +          return false;
    +        }
    --- End diff --
   
    you no need use if condition, just return nullValue


---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1751#discussion_r162124981
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/scan/filter/executer/RowLevelRangeLessThanEqualFilterExecuterImpl.java ---
    @@ -150,14 +155,17 @@ private boolean isScanRequired(byte[] blockMinValue, byte[][] filterValues) {
         return isScanRequired;
       }
     
    -  private boolean isScanRequired(byte[] minValue, Object[] filterValue,
    +  private boolean isScanRequired(byte[] minValue, boolean nullValue, Object[] filterValue,
           DataType dataType) {
         Object value =
             DataTypeUtil.getMeasureObjectFromDataType(minValue, dataType);
         for (int i = 0; i < filterValue.length; i++) {
    -      // TODO handle min and max for null values.
           if (filterValue[i] == null) {
    -        return true;
    +        if (nullValue == true) {
    +          return true;
    +        } else {
    +          return false;
    +        }
    --- End diff --
   
    Just return nullValue no need if condition


---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1751#discussion_r162125695
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/scan/filter/executer/RowLevelRangeLessThanFiterExecuterImpl.java ---
    @@ -153,13 +158,17 @@ private boolean isScanRequired(byte[] blockMinValue, byte[][] filterValues) {
         return isScanRequired;
       }
     
    -  private boolean isScanRequired(byte[] minValue, Object[] filterValue,
    +  private boolean isScanRequired(byte[] minValue, boolean nullValue, Object[] filterValue,
           DataType dataType) {
         Object value = DataTypeUtil.getMeasureObjectFromDataType(minValue, dataType);
         for (int i = 0; i < filterValue.length; i++) {
           // TODO handle min and max for null values.
           if (filterValue[i] == null) {
    -        return true;
    +        if (nullValue == true) {
    +          return true;
    +        } else {
    +          return false;
    +        }
    --- End diff --
   
    just return nullValue


---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1751#discussion_r162126330
 
    --- Diff: hadoop/src/test/java/org/apache/carbondata/hadoop/test/util/StoreCreator.java ---
    @@ -520,7 +522,19 @@ public static String readCurrentTime() {
       }
     
       public static void main(String[] args) {
    -    StoreCreator.createCarbonStore();
    +
    +    SimpleDateFormat parser = null;
    +    parser = new SimpleDateFormat("yyyy-MM-dd");
    +    parser.setTimeZone(TimeZone.getTimeZone("IST"));
    +
    +    String v = "2011-11-11";
    +    try {
    +      long value = parser.parse(v).getTime() * 1000L;
    +      System.out.print(value);
    +    } catch (ParseException pe) {
    +
    +    }
    +    // StoreCreator.createCarbonStore();
    --- End diff --
   
    why this class i changed


---

[qiuchenjian-2]

Github user sounakr commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1751#discussion_r162289924
 
    --- Diff: hadoop/src/test/java/org/apache/carbondata/hadoop/test/util/StoreCreator.java ---
    @@ -520,7 +522,19 @@ public static String readCurrentTime() {
       }
     
       public static void main(String[] args) {
    -    StoreCreator.createCarbonStore();
    +
    +    SimpleDateFormat parser = null;
    +    parser = new SimpleDateFormat("yyyy-MM-dd");
    +    parser.setTimeZone(TimeZone.getTimeZone("IST"));
    +
    +    String v = "2011-11-11";
    +    try {
    +      long value = parser.parse(v).getTime() * 1000L;
    +      System.out.print(value);
    +    } catch (ParseException pe) {
    +
    +    }
    +    // StoreCreator.createCarbonStore();
    --- End diff --
   
    Removed


---

[qiuchenjian-2]

Github user sounakr commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1751#discussion_r162289956
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/scan/filter/executer/RowLevelRangeLessThanFiterExecuterImpl.java ---
    @@ -153,13 +158,17 @@ private boolean isScanRequired(byte[] blockMinValue, byte[][] filterValues) {
         return isScanRequired;
       }
     
    -  private boolean isScanRequired(byte[] minValue, Object[] filterValue,
    +  private boolean isScanRequired(byte[] minValue, boolean nullValue, Object[] filterValue,
           DataType dataType) {
         Object value = DataTypeUtil.getMeasureObjectFromDataType(minValue, dataType);
         for (int i = 0; i < filterValue.length; i++) {
           // TODO handle min and max for null values.
           if (filterValue[i] == null) {
    -        return true;
    +        if (nullValue == true) {
    +          return true;
    +        } else {
    +          return false;
    +        }
    --- End diff --
   
    Done


---

[qiuchenjian-2]

Github user sounakr commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1751#discussion_r162289986
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/scan/filter/executer/RowLevelRangeLessThanEqualFilterExecuterImpl.java ---
    @@ -150,14 +155,17 @@ private boolean isScanRequired(byte[] blockMinValue, byte[][] filterValues) {
         return isScanRequired;
       }
     
    -  private boolean isScanRequired(byte[] minValue, Object[] filterValue,
    +  private boolean isScanRequired(byte[] minValue, boolean nullValue, Object[] filterValue,
           DataType dataType) {
         Object value =
             DataTypeUtil.getMeasureObjectFromDataType(minValue, dataType);
         for (int i = 0; i < filterValue.length; i++) {
    -      // TODO handle min and max for null values.
           if (filterValue[i] == null) {
    -        return true;
    +        if (nullValue == true) {
    +          return true;
    +        } else {
    +          return false;
    +        }
    --- End diff --
   
    Done


---

[qiuchenjian-2]

Github user sounakr commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1751#discussion_r162290027
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/scan/filter/executer/RowLevelRangeGrtrThanEquaToFilterExecuterImpl.java ---
    @@ -149,13 +154,17 @@ private boolean isScanRequired(byte[] blockMaxValue, byte[][] filterValues) {
         return isScanRequired;
       }
     
    -  private boolean isScanRequired(byte[] maxValue, Object[] filterValue,
    +  private boolean isScanRequired(byte[] maxValue, boolean nullValue, Object[] filterValue,
           DataType dataType) {
         Object value = DataTypeUtil.getMeasureObjectFromDataType(maxValue, dataType);
         for (int i = 0; i < filterValue.length; i++) {
           // TODO handle min and max for null values.
           if (filterValue[i] == null) {
    -        return true;
    +        if (nullValue == true) {
    +          return true;
    +        } else {
    +          return false;
    +        }
    --- End diff --
   
    Done


---

[qiuchenjian-2]

Github user sounakr commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1751#discussion_r162290297
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/indexstore/blockletindex/BlockletDataMap.java ---
    @@ -196,6 +201,10 @@ private DataMapRowImpl loadToUnsafe(DataFileFooter fileFooter,
               unsafeMemorySummaryDMStore.getSchema()[taskMinMaxOrdinal], maxValues,
               TASK_MAX_VALUES_INDEX, false);
           ordinal++;
    +      taskMinMaxOrdinal++;
    +
    +      summaryRow.setByteArray(minMaxIndex.getNullValues().toByteArray(), taskMinMaxOrdinal);
    --- End diff --
   
    Added new routine addTaskNullValues which is going to OR all the Null Bitsets from all the blocklets and place the final value into the summary row. Like the one done for minMax.


---

[qiuchenjian-2]

Github user sounakr commented on the issue:

    https://github.com/apache/carbondata/pull/1751
 
    Retest this please


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1751
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2968/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1751
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2971/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1751
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2959/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1751
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1730/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1751
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2992/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1751
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1762/



---