[GitHub] carbondata pull request #3068: [HOTFIX] Fixed NPE during query with Local Di...

[qiuchenjian-2]

GitHub user kumarvishal09 opened a pull request:

    https://github.com/apache/carbondata/pull/3068

    [HOTFIX] Fixed NPE during query with Local Dictionary

    **Problem:**
    Query is failing with NPE when some blocklet encoded with local dictionary and some without local dictionary.
    **Root Cause:**
    This is coming because in carbonvectorProxy setDictionary with null it is not setting the dictionary to null because of this it is treated like a local dictionary column but column is not encoded with dictionary.
    **Solution:**
    Set dictionary to null
     - [ ] Any interfaces changed?
     
     - [ ] Any backward compatibility impacted?
     
     - [ ] Document update required?
   
     - [ ] Testing done
            Please provide details on
            - Whether new unit test cases have been added or why no new tests are required?
            - How it is tested? Please attach test report.
            - Is it a performance related change? Please attach the performance test report.
            - Any additional information to help reviewers in testing this change.
           
     - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
   


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/kumarvishal09/incubator-carbondata master_102019

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/3068.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #3068
   
----
commit 7fdc042bcf2c7bc05135a809cab8ccd45dfbc01c
Author: kumarvishal09 <kumarvishal1802@...>
Date:   2019-01-11T09:44:53Z

    fixed NPE in LocalDictionary Query

----


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/3068
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2269/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/3068
 
    Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2488/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/3068
 
    Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10527/



---

[qiuchenjian-2]

Github user qiuchenjian commented on the issue:

    https://github.com/apache/carbondata/pull/3068
 
    why does one segment have some blocklet encoded with local dictionary and some without local dictionary ?


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/3068
 
    > why does one segment have some blocklet encoded with local dictionary and some without local dictionary ?
   
    It is because carbon generates a dictionary based on the column value count threshold, so once it reaches that threshold it stops generating the dictionary. There are scenarios where some blocks/blocklets are with in threshold and some are not, thats why some blocks has local dictionary and some don't have


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/3068
 
    LGTM


---

[qiuchenjian-2]

Github user qiuchenjian commented on the issue:

    https://github.com/apache/carbondata/pull/3068
 
    @ravipesala I remember if the local dictionary value count reach threshold, it will go back to original value， right？
    Does some nodes's local dictionary reach the threshold, bug other's not, so this scene appeared ？（some shards have local dictionary, some shards don't have  local dictionary）


---