[jira] [Commented] (CARBONDATA-1624) If SORT_SCOPE is non-GLOBAL_SORT with Spark, set 'carbon.number.of.cores.while.loading' dynamically as per the available executor cores

[Akash R Nilugal (Jira)]

    [ https://issues.apache.org/jira/browse/CARBONDATA-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16221856#comment-16221856 ]

Zhichao  Zhang commented on CARBONDATA-1624:
--------------------------------------------

Hi [~ravi.pesala], why does it impact other loads?  I will implement this as following steps:
1. Add a parameter called 'carbon.number.of.cores.in.executor' to CarbonCommonConstants;
2. In 'CarbonSession.getOrCreateCarbonSession', get the core number of executor from SparkConf (using sparkContext.getConf.get("spark.executor.cores")) , and then set this number to CarbonProperties, the key is 'carbon.number.of.cores.in.executor';
3. Change CarbonProperties.getNumberOfCores(), use the value of 'carbon.number.of.cores.in.executor' as the default value of 'carbon.number.of.cores.while.loading'; if the value of 'carbon.number.of.cores.in.executor' is null or empty, use 'NUM_CORES_DEFAULT_VAL' as the default value.

Are the above steps OK?

Do you mean that it can't change the below code snippet to 'CarbonProperties.getNumberOfCores()'?

> If SORT_SCOPE is non-GLOBAL_SORT with Spark, set 'carbon.number.of.cores.while.loading' dynamically as per the available executor cores
> ----------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CARBONDATA-1624
>                 URL: https://issues.apache.org/jira/browse/CARBONDATA-1624
>             Project: CarbonData
>          Issue Type: Improvement
>          Components: data-load, spark-integration
>    Affects Versions: 1.3.0
>            Reporter: Zhichao  Zhang
>            Assignee: Zhichao  Zhang
>            Priority: Minor
>
> If we are using carbondata + spark to load data, we can set
> carbon.number.of.cores.while.loading to the  number of executor cores.
> For example, when set the number of executor cores to 6, it shows that there are at
> least 6 cores per node for loading data, so we can set
> carbon.number.of.cores.while.loading to 6 automatically.

--
This message was sent by Atlassian JIRA
(v6.4.14#64029)