Posted by
kumarvishal09 on
Dec 07, 2018; 10:25am
URL: http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/SDK-supports-LOCAL-DICTIONARY-INCLUDE-and-LOCAL-DICTIONARY-EXCLUDE-tp69870p69901.html
@Raghunandan subramanya <
[hidden email]>
We have tested with *80 string columns with 10 high cardinality
columns(fallback happened for these columns)*, please find the stats:
*Test result is with 1 billion records 385 Gb size*
*1. Load time without local dictionary:* 66 minutes
*2. Load time without fallback local dictionary:* 72 minutes
*3. Load time with fallback local dictionary:* 74 minutes
*Without fallback local dictionary:* 9.09% degradation
*With fallback local dictionary:* 13.63%
-Regards
Kumar Vishal
On Fri, Dec 7, 2018 at 12:59 PM Raghunandan S <
[hidden email]> wrote:
> @kumar vishal what is the fallback performance if more number of columns
> need to fallback. Would it not increase the overhead of generating
> temporary dictionary and discarding it?
>
> On Fri, 7 Dec 2018, 12:56 pm ravipesala, <
[hidden email]> wrote:
>
> >
> > I agree with @kumarvishal , better not add more options as it confuses
> the
> > user. We better fallback automatically depends on the size of the
> > dictionary.
> >
> >
> >
> > --
> > Sent from:
> >
http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/> >
>
kumar vishal