[GitHub] carbondata pull request #2595: [Documentation] [Unsafe Configuration] Added ...

[qiuchenjian-2]

GitHub user manishgupta88 opened a pull request:

    https://github.com/apache/carbondata/pull/2595

    [Documentation] [Unsafe Configuration] Added carbon.unsafe.driver.working.memory.in.mb parameter to differentiate between driver and executor unsafe memory

    Added carbon.unsafe.driver.working.memory.in.mb parameter to differentiate between driver and executor unsafe memory
   
     - [ ] Any interfaces changed?
    No
     - [ ] Any backward compatibility impacted?
     No
     - [ ] Document update required?
    Yes. Updated
     - [ ] Testing done
    Verified manually      
     - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
    NA


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/manishgupta88/carbondata unsafe_driver_property

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/2595.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2595
   
----
commit 8196be0a50d9d0f38452cfdf5fbe0b0485973e87
Author: manishgupta88 <tomanishgupta18@...>
Date:   2018-08-01T14:08:30Z

    Added carbon.unsafe.driver.working.memory.in.mb parameter to differentiate between driver and executor unsafe memory

----


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2595
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7710/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2595
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6436/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2595
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6103/



---

[qiuchenjian-2]

Github user xuchuanyin commented on the issue:

    https://github.com/apache/carbondata/pull/2595
 
    Why driver needs this unsafe memory?


---

[qiuchenjian-2]

Github user manishgupta88 commented on the issue:

    https://github.com/apache/carbondata/pull/2595
 
    @xuchuanyin ....Usually in production scenarios driver memory will be less than the executor memory. Now we are using unsafe for caching block/blocklet dataMap in driver.  Current unsafe memory configured fo executor is getting used for driver also which is not a good idea.
    Therefore it is required to separate out driver and executor unsafe memory.
    You can observe the same in spark configuration also that spark has given different parameters for configuring driver and executor memory overhead to control the unsafe memory usage.
    spark.yarn.driver.memoryOverhead and spark.yarn.executor.memoryOverhead


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2595
 
    LGTM


---

[qiuchenjian-2]

Github user asfgit closed the pull request at:

    https://github.com/apache/carbondata/pull/2595


---