[
https://issues.apache.org/jira/browse/CARBONDATA-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16221627#comment-16221627 ]
Zhichao Zhang commented on CARBONDATA-1624:
--------------------------------------------
I can implement this feature.
By the way, I find there are many same code snippets to get 'NUM_CORES_LOADING', but there is a method called 'getNumberOfCores' in CarbonProperties to get 'NUM_CORES_LOADING' too, I think we can use method 'CarbonProperties.getNumberOfCores' uniformly to get 'NUM_CORES_LOADING' .
{code:java}
Integer.parseInt(CarbonProperties.getInstance()
.getProperty(CarbonCommonConstants.NUM_CORES_LOADING,
CarbonCommonConstants.NUM_CORES_DEFAULT_VAL));
} catch (NumberFormatException e) {
thread_pool_size = Integer.parseInt(CarbonCommonConstants.NUM_CORES_DEFAULT_VAL);
}
{code}
right?
> If SORT_SCOPE is non-GLOBAL_SORT with Spark, set 'carbon.number.of.cores.while.loading' dynamically as per the available executor cores
> ----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CARBONDATA-1624
> URL:
https://issues.apache.org/jira/browse/CARBONDATA-1624> Project: CarbonData
> Issue Type: Improvement
> Components: data-load, spark-integration
> Affects Versions: 1.3.0
> Reporter: Zhichao Zhang
> Assignee: Zhichao Zhang
> Priority: Minor
>
> If we are using carbondata + spark to load data, we can set
> carbon.number.of.cores.while.loading to the number of executor cores.
> For example, when set the number of executor cores to 6, it shows that there are at
> least 6 cores per node for loading data, so we can set
> carbon.number.of.cores.while.loading to 6 automatically.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
Locked
