Re: Disable Adaptive encoding for Double and Float by default

Posted by ravipesala on
URL: http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/Disable-Adaptive-encoding-for-Double-and-Float-by-default-tp93914p93915.html

Hi ,

It increases the store size.  Can you give me performance figures with and
without these changes.  And also provide how much store size impact if we
disable it.


Regards,
Ravindra.

On Wed, 25 Mar 2020 at 1:51 PM, Ajantha Bhat <[hidden email]> wrote:

> Hi all,
>
> I have done insert into flow profiling using JMC with the latest code
> [with new optimized insert flow]
>
> It seems for *2.5GB* carbon to carbon insert, double and float stats
> collector has used *68.36 GB* [*25%* of TLAB (Thread local allocation
> buffer)]
>
> [image: Screenshot from 2020-03-25 11-18-04.png]
> *The problem is for every value of double and float in every row, we call *
> *PrimitivePageStatsCollector.getDecimalCount()**Which makes new objects
> every time.*
>
> So, I want to disable Adaptive encoding for float and double by default.
> *I will make this configurable.*
> If some user has a well-sorted double or float column and wants to apply
> adaptive encoding on that, they can enable it to reduce store size.
>
> Thanks,
> Ajantha
>

--
Thanks & Regards,
Ravi