How to reduce driver memory usage of carbon index

Posted by yaojinguo on Apr 11, 2018; 7:55am
URL: http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/How-to-reduce-driver-memory-usage-of-carbon-index-tp44863.html

Hi community ,
  I am using CarbonData1.3 + Spark2.1, I find a potential bottleneck when
using Carbondata. As
I know, CarbonData loads all of the carbonindex files and turn these files
to DataMap or SegmentIndex (for early version)which contains startkey
,endkey,min/max value of each column. If I have one table with 200 columns
which contains 1000 segments, each segment has 2000 carbondata files, assume
each column occupies just 10 bytes, then you need at least 20GB memory to
store min/max values only. Any suggestion to resolve this problem?




--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/