Re: [Discussion] About carbon.si.segment.merge feature

Posted by David CaiQiang on Nov 09, 2020; 7:05am
URL: http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/Discussion-About-carbon-si-segment-merge-feature-tp103161p103174.html

hi Ajantha,
  Agree to remove "carbon.si.segment.merge"  

  1. dynamic decide the number for the loading tasks
  Before loading the SI segment, it is easy to estimate the total size of
this SI segment.
  So better to dynamic decide the number for the loading tasks to avoid
small carbon files in the SI segment.
 
  2. can we use global_sort for SI by default?
  SI is used to speed up filter query, global_sort can do better.
  We need global_sort for SI.
 
  3. use reindex instead of refresh index
  If Refresh index is only used to merge small files, reindex will be
better(should implement point 1).
  So, can we remove Refresh index too?
 
 



-----
Best Regards
David Cai
--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/