Re: [Discussion]Query Regarding Task launch mechanism for data load operations

Posted by David CaiQiang on Aug 14, 2020; 9:55am
URL: http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/Discussion-Query-Regarding-Task-launch-mechanism-for-data-load-operations-tp98711p98714.html

This mechanism will work fine for LOCAL_SORT loading of big data and the
small cluster with big executor.

If it doesn't match these conditions, better consider a new solution to
adapter the generic scenario.

I suggest re-factoring NO_SORT, maybe we can check and improve the
global_sort solution.

The solution should support both NO_SORT and GLOBAL_SORT, and automatically
determines the number of partitions to avoid small file issue.




-----
Best Regards
David Cai
--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/