Re: Enhancement on compaction performance

Posted by xuchuanyin on Nov 08, 2018; 8:27am
URL: http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/Enhancement-on-compaction-performance-tp67996p68035.html

Hi, all:

The previous experiment uses 3 huawei ecs instances as workers each with 16
cores and 32GB. Spark executor use 12 cores and 24GB. Using 74GB LineItem in
100GB TPCH.

Today I run another experiment using 1 huawei RH2288 machine with 32 cores
and 128GB. Spark executor use 30 cores and 90GB. Using 7.3GB LineItem in
10GB TPCH. And the results are as below:

Code Branch Prefetch Batch Size (default 100) Load1 (s) Load2 (s) Load3 (s)
Compact 3 Loads (s) Time Reduced Perf Enhanced
master NA 100 147.4 142.3 144.6 201.4 Baseline Baseline
master NA 32000 140.8 138.7 141.6 196.2 2.6% 2.7%
PR2906 enable 100 143.9 142.5 146.2 99.9 50.4% 101.6%
PR2906 enable 32000 142.1 139.3 136.9 98.3 51.2% 104.9%
PR2906 disable 100 146.7 137.4 139.6 200.6 0.4% 0.4%
PR2906 disable 32000 145.2 145.0 139.7 195.7 2.8% 2.9%

It also shows this PR will not decrease the compaction performance if
disabled and will enhance the performance if enabled.




--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/