[GitHub] carbondata pull request #1751: [WIP]Measure null

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1751
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1485/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1751
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2846/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1751
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1491/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1751
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2724/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1751
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2850/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1751
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2851/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1751
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2732/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1751
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1501/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1751
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2859/



---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1751#discussion_r161366903
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/statistics/SimpleStatsResult.java ---
    @@ -25,6 +25,8 @@
     
       Object getMax();
     
    +  byte getNull();
    --- End diff --
   
    SimpleStatsResult is the statistics for a ColumnPage, what does getNull() means for a page?


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1751#discussion_r161366923
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/statistics/SimpleStatsResult.java ---
    @@ -25,6 +25,8 @@
     
       Object getMax();
     
    +  byte getNull();
    --- End diff --
   
    SimpleStatsResult is the statistics for a ColumnPage, what does getNull() means for a page?


---

[qiuchenjian-2]

Github user sounakr commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1751#discussion_r161430749
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/statistics/SimpleStatsResult.java ---
    @@ -25,6 +25,8 @@
     
       Object getMax();
     
    +  byte getNull();
    --- End diff --
   
    getNull() will returns Null value present in page or not.


---

[qiuchenjian-2]

Github user sounakr commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1751#discussion_r161431169
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/indexstore/blockletindex/BlockletDataMap.java ---
    @@ -728,9 +746,9 @@ private DataMapRow convertToRow(IndexKey key) {
     
       private List<String> getPartitions() {
         DataMapRow unsafeRow = unsafeMemorySummaryDMStore.getUnsafeRow(0);
    -    if (unsafeRow.getColumnCount() > 2) {
    +    if (unsafeRow.getColumnCount() > 3) {
           List<String> partitions = new ArrayList<>();
    -      DataMapRow row = unsafeRow.getRow(2);
    +      DataMapRow row = unsafeRow.getRow(3);
    --- End diff --
   
    Done


---

[qiuchenjian-2]

Github user sounakr commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1751#discussion_r161431217
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/indexstore/blockletindex/BlockletDataRefNodeWrapper.java ---
    @@ -99,6 +99,10 @@ public BlockletDataRefNodeWrapper(List<TableBlockInfo> blockInfos, int index,
         return null;
       }
     
    +  @Override public byte[] getColumnsNullValue() {
    --- End diff --
   
    Done


---

[qiuchenjian-2]

Github user sounakr commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1751#discussion_r161431288
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/scan/filter/executer/FilterExecuter.java ---
    @@ -46,7 +46,7 @@ boolean applyFilter(RowIntf value, int dimOrdinalMax)
        * @param blockMinValue
        * @return BitSet
        */
    -  BitSet isScanRequired(byte[][] blockMaxValue, byte[][] blockMinValue);
    +  BitSet isScanRequired(byte[][] blockMaxValue, byte[][] blockMinValue, byte[] blockNullValue);
    --- End diff --
   
    Done


---

[qiuchenjian-2]

Github user sounakr commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1751#discussion_r161431299
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/scan/filter/executer/ImplicitColumnFilterExecutor.java ---
    @@ -43,5 +43,6 @@ BitSet isFilterValuesPresentInBlockOrBlocklet(byte[][] maxValue, byte[][] minVal
        *
        * @return
        */
    -  Boolean isFilterValuesPresentInAbstractIndex(byte[][] maxValue, byte[][] minValue);
    +  Boolean isFilterValuesPresentInAbstractIndex(byte[][] maxValue, byte[][] minValue,
    --- End diff --
   
    Done


---

[qiuchenjian-2]

Github user sounakr commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1751#discussion_r161431317
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/scan/filter/executer/ImplicitColumnFilterExecutor.java ---
    @@ -34,7 +34,7 @@
        * @return
        */
       BitSet isFilterValuesPresentInBlockOrBlocklet(byte[][] maxValue, byte[][] minValue,
    -      String uniqueBlockPath);
    +      byte[] nullValue, String uniqueBlockPath);
    --- End diff --
   
    Done


---

[qiuchenjian-2]

Github user sounakr commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1751#discussion_r161431335
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/statistics/TablePageStatistics.java ---
    @@ -71,6 +82,10 @@ private void updateMeasureMinMax(EncodedColumnPage[] measures) {
         return dimensionMaxValue;
       }
     
    +  public byte[] getDimentionNullValue() {
    --- End diff --
   
    Rectified


---

[qiuchenjian-2]

Github user sounakr commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1751#discussion_r161431568
 
    --- Diff: examples/spark2/src/main/scala/org/apache/carbondata/examples/CarbonSessionExample.scala ---
    @@ -62,9 +62,9 @@ object CarbonSessionExample {
     
         spark.sql(
           s"""
    -         | SELECT *
    -         | FROM carbon_table
    -         | WHERE stringfield = 'spark' AND decimalField > 40
    +        | SELECT *
    --- End diff --
   
    Rectified


---

[qiuchenjian-2]

Github user sounakr commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1751#discussion_r161431581
 
    --- Diff: examples/spark2/src/main/scala/org/apache/carbondata/examples/ExampleUtils.scala ---
    @@ -39,8 +39,8 @@ object ExampleUtils {
         val warehouse = s"$rootPath/examples/spark2/target/warehouse"
     
         CarbonProperties.getInstance()
    -      .addProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT, "yyyy/MM/dd HH:mm:ss")
    -      .addProperty(CarbonCommonConstants.CARBON_DATE_FORMAT, "yyyy/MM/dd")
    +      .addProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT, "yyyy-MM-dd HH:mm:ss")
    --- End diff --
   
    Done


---