Re: Question about RLE and DELTA encoding

Posted by k.ashok on Dec 15, 2016; 2:55am
URL: http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/Question-about-RLE-and-DELTA-encoding-tp4441p4451.html

Hi Hao Jiang
Regarding your first question why RLE is controlled by aggKeyBlock.
There is dictionary and no-dictionary column type in carbon.
carbon sort the column data and then store it. Due to sorting index will get shuffled. Hence
for no dictionary data RLE is applied on index and not on data because it is no dictionary data.
thus in BlockIndexerStorageForInt@compressMyOwnWay, RLE happens on index. compressDataMyOwnWay
is done only for dictionary data.

Regarding your second question
Measure data are not sorted and hence sequential delta may be either big or small
for e.g
if data is 2,-3,4,-6 then sequential delta will be(-5,7,-10,-6)
Other then max min delta, we do type conversion also to reduce storage space