Re: Carbon over-use cluster resources

Posted by David CaiQiang on May 08, 2020; 1:58am
URL: http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/Carbon-over-use-cluster-resources-tp94332p95840.html

Hi, manhua
  Now no_sort reuse the loading flow of local_sort. It is not a good
solution and led to the situation which you have mentioned. In my opinion,
we need to adjust the loading flow of no_sort, maybe like global_sort
finally.

  In addition, the producer-consumer pattern in data encoding and
compression also can be optimized for no_sort and global_sort, maybe just
prefetch one page and process it instead of using a thread pool.



-----
Best Regards
David Cai
--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/