Re: [Discussion] Carbon Local Dictionary Support

Posted by kumarvishal09 on Jun 05, 2018; 5:48am
URL: http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/Discussion-Carbon-Local-Dictionary-Support-tp51447p51456.html

Hi Xuchuanyin,

I am working on design document, and all the points you have mentioned I
have already captured. I will share once it is finished.

-Regards
Kumar Vishal

On Tue, Jun 5, 2018 at 9:22 AM, xuchuanyin <[hidden email]> wrote:

> Hi, Kumar：
>   Local dictionary will be nice feature and other formats like parquet all
> support this.
>
>   My concern is that: How will you implement this feature?
>
>   1. What's the scope of the `local`? Page level (for all containing rows),
> Blocklet level (for all containing pages), Block level(for all containing
> blocklets)?
>
>   2. Where will you store the local dictionary?
>
>   3. How do you decide to enable the local dictionary for a column?
>
>   4. Have you considered to fall back to plain encoding if the local
> dictionary encoding consumes more space?
>
>   5. Will you still work on V3 format or start a new V4 (or v3.1) version?
>
>   Anyway, I'm concerning about the data loading performance. Please pay
> attention to it while you are implementing this feature.
>
>
>
> --
> Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.
> n5.nabble.com/
>