[GitHub] carbondata pull request #2897: [WIP] Supporting local dictionary enable by d...

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2897
 
    Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2120/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2897
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1925/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2897
 
    Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2135/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2897
 
    Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10178/



---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2897#discussion_r243810009
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/memory/UnsafeMemoryManager.java ---
    @@ -173,6 +174,7 @@ public synchronized void freeMemoryAll(String taskId) {
               "Freeing offheap working memory of size %d. Current available memory is %d",
               occuppiedMemory, totalMemory - memoryUsed));
         }
    +    ThreadLocalTaskInfo.clearCarbonTaskInfo();
    --- End diff --
   
    Better call explicitly from methods where we call `freeMemoryAll, thread clearing should not be part of it.


---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2897#discussion_r243810091
 
    --- Diff: hadoop/src/main/java/org/apache/carbondata/hadoop/util/CarbonVectorizedRecordReader.java ---
    @@ -68,6 +67,10 @@
       private AbstractDetailQueryResultIterator iterator;
     
       private QueryModel queryModel;
    +  //This holds mapping of  fetch index with respect to project col index.
    +  // it is used when same col is used in projection many times.So need to fetch only that col.
    +  private List<Integer> projectionMapping = new ArrayList<>();
    --- End diff --
   
    Better use  an array to have fast lookups instead of list


---

[qiuchenjian-2]

Github user BJangir commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2897#discussion_r243812352
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/memory/UnsafeMemoryManager.java ---
    @@ -173,6 +174,7 @@ public synchronized void freeMemoryAll(String taskId) {
               "Freeing offheap working memory of size %d. Current available memory is %d",
               occuppiedMemory, totalMemory - memoryUsed));
         }
    +    ThreadLocalTaskInfo.clearCarbonTaskInfo();
    --- End diff --
   
    OK..


---

[qiuchenjian-2]

Github user BJangir commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2897#discussion_r243812841
 
    --- Diff: hadoop/src/main/java/org/apache/carbondata/hadoop/util/CarbonVectorizedRecordReader.java ---
    @@ -68,6 +67,10 @@
       private AbstractDetailQueryResultIterator iterator;
     
       private QueryModel queryModel;
    +  //This holds mapping of  fetch index with respect to project col index.
    +  // it is used when same col is used in projection many times.So need to fetch only that col.
    +  private List<Integer> projectionMapping = new ArrayList<>();
    --- End diff --
   
    During initBatch only it is done(not frequent lookup/creation ).  


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2897
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1928/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2897
 
    Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10181/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2897
 
    Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2138/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2897
 
    LGTM


---

[qiuchenjian-2]

Github user asfgit closed the pull request at:

    https://github.com/apache/carbondata/pull/2897


---