Posted by
xuchuanyin on
Nov 09, 2018; 6:25am
URL: http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/DISCUSSION-refining-usage-of-numberofcores-in-CarbonProperties-tp68049p68070.html
I think you are talking about 2 problems.
The first is that：We have some configurations about numberOfCores such as
cores for Loading/Compaction/AlterPartition, currently they use the same
method to get the configured value which means they all are actually using
the numberOfCoresWhileLoading. And your PR#2907 is to fix this problem.
But there is another problem: For data loading, in the document we say it
will use 2 cores by default if the numberOfCoresWhileLoading is not
configured. But actually in our code, we are using the
numberOfCoresForCurrentExecutor as the default value. And this problem is
not resolved in your PR#2907 yet.
To fix the 2nd problem, my suggestion is as below:
1. You can update the document for the default value of
numOfCoresWhileLoading. The default value will be the
'spark.executor.cores'.
2. At the same time I think you should also optimize the default value of
numOfCoresWhileCompation. To keep the behavior the same, the default value
should also be 'spark.executor.cores'. -- This requires the modification of
the document as well as the code.
--
Sent from:
http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/