[GitHub] incubator-carbondata pull request #752: [CARBONDATA-879] Clear driver side d...

[qiuchenjian-2]

GitHub user Sephiroth-Lin opened a pull request:

    https://github.com/apache/incubator-carbondata/pull/752

    [CARBONDATA-879] Clear driver side dict cache for CarbonDictionaryDecoder

   

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/Sephiroth-Lin/incubator-carbondata carbondata-879

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-carbondata/pull/752.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #752
   
----
commit 07c4c71bcea737d9db87ac426c1fe077801e3b24
Author: Sephiroth-Lin <[hidden email]>
Date:   2017-04-06T13:01:13Z

    Clear driver side dict cache for CarbonDictionaryDecoder

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/752
 
    Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user Sephiroth-Lin commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/752
 
    @kumarvishal09 @QiangCai @watermen Please help review, tks


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user chenliang613 commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/752
 
    add to whitelist


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/752
 
    Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1493/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user QiangCai commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/752
 
    LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/752
 
    @Sephiroth-Lin Actually clearing the cache at the time of codegen generation does not help as it just returns the code string to spark , Spark compiles it and executes as part of WholeCodeGen. We need to find a way to clear inside the code gen.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user Sephiroth-Lin commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/752
 
    @ravipesala Yes, inside the code gen, then it will clear for executor side, but for driver side after code gen we also need to clear the access count. From my test, if we don't clear it, then we can't remove it from memory if we don't have enough memory to load another cache.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/752
 
    @Sephiroth-Lin Yes you are right, but I feel it is not required to create dictionary while doing codegen, so better avoid dictionary creation while doing codegen. just create only at executor side.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---