Re: [Discussion] Minimize the Btree size and unify the driver and executor Btrees.

Posted by ravipesala on May 18, 2017; 6:55am
URL: http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/Discussion-Minimize-the-Btree-size-and-unify-the-driver-and-executor-Btrees-tp12771p12863.html

Hi Jacky,

Default blocklet size currently is 64 MB, so if block size is 256 MB then
at most blocklets per block is 4.

Regards,
Ravindra.

On 17 May 2017 at 19:59, Jacky Li <[hidden email]> wrote:

> +1 for both proposal 1 & 2,
>
> For point 2, do you have idea how many blocklet within one block roughly?
> This will help to estimate the length of array in driver side.
>
> Regards,
> Jacky
>
> > 在 2017年5月17日，下午7:33，Ravindra Pesala <[hidden email]> 写道：
> >
> > Hi,
> >
> > *1. Current problem.*
> > 1.There is more size taking on java heap to create Btree for index file.
> > It is because we create multiple objects for each leaf node so it takes
> > more memory inside heap than actual size of index file. while doing LRU
> > cache also we are considering only index file size instead of objects
> size
> > so it impacts the eviction process of LRU cache.
> > 2. Currently we load one btree on driver side to find the blocks and load
> > another btree on executor side to find the blocklets. After we have
> > increased the blocklet size to 128 mb and decrease the table_block size
> to
> > 256 mb the number of nodes inside driver side btree and executor side
> btree
> > is not much different. So it would be overhead to read the same
> information
> > twice.
> > And also chances of loading btree on each executor is more for every
> query
> > because there is no guarantee that same block goes to same executor every
> > time. It will be worse in case of dynamic containers.
> >
> > *2. Proposed solution.*
> > 1. To reduce the java heap for Btree , we can remove the Btree data
> > structure and use simple single array and do binary search on it. And
> also
> > we should move this cached arrays to unsafe (offheap/onheap) to reduce
> the
> > burden on GC.
> > 2. Unify the btree to single Btree instead of 2 and load at driver side.
> > So that only one lookup can be done to find the blocklets directly. And
> > executors are not required to load the btree for every query.
> >    We can consider moving this to separate metadata service eventually
> > once the memory footprint get reduced.
> >
> > First I will consider point 1 reduce the btree size after that I consider
> > merging of btrees.
> >
> > Please comment on it.
> >
> > --
> > Thanks & Regards,
> > Ravindra.
>
>
>
>

--
Thanks & Regards,
Ravi