[GitHub] carbondata pull request #2984: [CARBONDATA-3165]Protection of Bloom Null Exc...

[qiuchenjian-2]

GitHub user qiuchenjian opened a pull request:

    https://github.com/apache/carbondata/pull/2984

    [CARBONDATA-3165]Protection of Bloom Null Exception

    Problem：
    24274.0 (TID 664711) | org.apache.spark.internal.Logging$class.logError(Logging.scala:91)
    java.lang.NullPointerException
            at java.util.ArrayList.<init>(ArrayList.java:177)
            at org.apache.carbondata.datamap.bloom.BloomCoarseGrainDataMap.prune(BloomCoarseGrainDataMap.java:230)
            at org.apache.carbondata.core.datamap.TableDataMap.prune(TableDataMap.java:379)
            at org.apache.carbondata.core.datamap.DistributableDataMapFormat$1.initialize(DistributableDataMapFormat.java:108)
            at org.apache.carbondata.spark.rdd.DataMapPruneRDD.internalCompute(SparkDataMapJob.scala:77)
            at org.apache.carbondata.spark.rdd.CarbonRDD.compute(CarbonRDD.scala:82)
            at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
            at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
            at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
            at org.apache.spark.scheduler.Task.run(Task.scala:99)
            at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
            at java.lang.Thread.run(Thread.java:748)
   
    Solution：
    when filteredShard is empty, return arraylist<>(), and protect null exception of other scenes
   
     - [ ] Any interfaces changed?
     
     - [ ] Any backward compatibility impacted?
     
     - [ ] Document update required?
   
     - [ ] Testing done
            protection of null exception, existing test cases are fine
           
     - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
   


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/qiuchenjian/carbondata CARBONDATA-3165

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/2984.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2984
   
----
commit fe7f87d2a5e049c030d800f845a8d3be13323873
Author: qiuchenjian <807169000@...>
Date:   2018-12-12T08:43:31Z

    [CARBONDATA-3165]Protection of Bloom Null Exception

----


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2984
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1718/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2984
 
    Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9978/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2984
 
    Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1930/



---

[qiuchenjian-2]

Github user QiangCai commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2984#discussion_r241265601
 
    --- Diff: datamap/bloom/src/main/java/org/apache/carbondata/datamap/bloom/BloomCoarseGrainDataMap.java ---
    @@ -227,6 +231,12 @@ private String getAncestorTablePath(CarbonTable currentTable) {
             }
           }
         }
    +    if (hitBlocklets == null) {
    +      LOGGER.warn(String.format("HitBlocklets is null int bloom filter prune method! " +
    --- End diff --
   
    ```suggestion
          LOGGER.warn(String.format("HitBlocklets is empty in bloom filter prune method. " +
    ```


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2984
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1728/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2984
 
    Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9988/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2984
 
    Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1940/



---

[qiuchenjian-2]

Github user qiuchenjian commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2984#discussion_r241298414
 
    --- Diff: datamap/bloom/src/main/java/org/apache/carbondata/datamap/bloom/BloomCoarseGrainDataMap.java ---
    @@ -227,6 +231,12 @@ private String getAncestorTablePath(CarbonTable currentTable) {
             }
           }
         }
    +    if (hitBlocklets == null) {
    +      LOGGER.warn(String.format("HitBlocklets is null int bloom filter prune method! " +
    --- End diff --
   
    done


---

[qiuchenjian-2]

Github user xuchuanyin commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2984#discussion_r241616785
 
    --- Diff: datamap/bloom/src/main/java/org/apache/carbondata/datamap/bloom/BloomCoarseGrainDataMap.java ---
    @@ -227,6 +231,12 @@ private String getAncestorTablePath(CarbonTable currentTable) {
             }
           }
         }
    +    if (hitBlocklets == null) {
    +      LOGGER.warn(String.format("HitBlocklets is empty in bloom filter prune method. " +
    --- End diff --
   
    Is this a potential problem? If not, why output 'warn' message?


---

[qiuchenjian-2]

Github user qiuchenjian commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2984#discussion_r241617708
 
    --- Diff: datamap/bloom/src/main/java/org/apache/carbondata/datamap/bloom/BloomCoarseGrainDataMap.java ---
    @@ -227,6 +231,12 @@ private String getAncestorTablePath(CarbonTable currentTable) {
             }
           }
         }
    +    if (hitBlocklets == null) {
    +      LOGGER.warn(String.format("HitBlocklets is empty in bloom filter prune method. " +
    --- End diff --
   
    Yes, it will not be executed normally ,when filteredShard's size is processed, but for protection, adding this code


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2984#discussion_r241938979
 
    --- Diff: datamap/bloom/src/main/java/org/apache/carbondata/datamap/bloom/BloomCoarseGrainDataMap.java ---
    @@ -227,6 +231,12 @@ private String getAncestorTablePath(CarbonTable currentTable) {
             }
           }
         }
    +    if (hitBlocklets == null) {
    +      LOGGER.warn(String.format("HitBlocklets is empty in bloom filter prune method. " +
    --- End diff --
   
    can you make it non-null, such as making it an empty set


---

[qiuchenjian-2]

Github user qiuchenjian commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2984#discussion_r241950006
 
    --- Diff: datamap/bloom/src/main/java/org/apache/carbondata/datamap/bloom/BloomCoarseGrainDataMap.java ---
    @@ -227,6 +231,12 @@ private String getAncestorTablePath(CarbonTable currentTable) {
             }
           }
         }
    +    if (hitBlocklets == null) {
    +      LOGGER.warn(String.format("HitBlocklets is empty in bloom filter prune method. " +
    --- End diff --
   
    @jackylk empty sets present all blocklets are pruned, and null presents pruning is not effective,so it should return null.
    Now out datamap lack null processing（exception scene），it need to be enhanced later


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2984
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1967/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2984
 
    Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10219/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2984
 
    Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2240/



---

[qiuchenjian-2]

Github user jackylk commented on the issue:

    https://github.com/apache/carbondata/pull/2984
 
    LGTM


---

[qiuchenjian-2]

Github user asfgit closed the pull request at:

    https://github.com/apache/carbondata/pull/2984


---