Re: Enhancement on compaction performance

Posted by xuchuanyin on Nov 09, 2018; 6:16am
URL: http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/Enhancement-on-compaction-performance-tp67996p68067.html

Oh, I didn't notice the memory consumption at that time.

We all know that the resource utilization is low during compaction.
Using prefetch means that We are doing query background and it will surely
consume more resources.
Current size of prefetch is controlled by the 'carbon.detail.batch.size' and
by default is 100 which means extra 100 rows will be kept in memory before
it is retrieved.
So the memory overhead consists the memory consumed by the query plus the
memory of the #carbon.detail.batch.size records.





--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/