[DISCUSSION]implement delta encoding for numeric type column in SORT_COLUMNS

[David CaiQiang]

Hi all,

    Now we plan to implement delta encoding for the numeric type column in SORT_COLUMNS.
   
    1. use delta encoding to encode the numeric type data

    2. write present metadata to the page header, to record the null value.

    3. improve the compression on no-dictionary string column.
        use RLE to compress the array of the lengths in LV encoding.

    Any thoughts, comments and questions ?

Best Regards
David QiangCai

[Jacky Li]

> 在 2017年4月5日，下午6:31，QiangCai <[hidden email]> 写道：
>
> Hi all,
>
>    Now we plan to implement delta encoding for the numeric type column in
> SORT_COLUMNS.
>
>    1. use delta encoding to encode the numeric type data
>

I think the adaptive data type conversion still apply here, right?

>    2. write present metadata to the page header, to record the null value.
>
>    3. improve the compression on no-dictionary string column.
>        use RLE to compress the array of the lengths in LV encoding.
>

I think local dictionary is more suitable for this case

>    Any thoughts, comments and questions ?
>
> Best Regards
> David QiangCai
>
>
> --
> View this message in context: http://apache-carbondata-mailing-list-archive.1130556.n5.nabble.com/DISCUSSION-implement-delta-encoding-for-numeric-type-column-in-SORT-COLUMNS-tp10124.html
> Sent from the Apache CarbonData Mailing List archive mailing list archive at Nabble.com.

[Liang Chen]

Hi David

Thanks for your starting this new feature's discussion.
Can you explain what are the major benefits after doing delta encoding for
the numeric type column.

Regards
Liang

2017-04-05 16:01 GMT+05:30 QiangCai <[hidden email]>:

> Hi all,
>
>     Now we plan to implement delta encoding for the numeric type column in
> SORT_COLUMNS.
>
>     1. use delta encoding to encode the numeric type data
>
>     2. write present metadata to the page header, to record the null value.
>
>     3. improve the compression on no-dictionary string column.
>         use RLE to compress the array of the lengths in LV encoding.
>
>     Any thoughts, comments and questions ?
>
> Best Regards
> David QiangCai
>
>
>
> --
> View this message in context: http://apache-carbondata-
> mailing-list-archive.1130556.n5.nabble.com/DISCUSSION-
> implement-delta-encoding-for-numeric-type-column-in-SORT-
> COLUMNS-tp10124.html
> Sent from the Apache CarbonData Mailing List archive mailing list archive
> at Nabble.com.
>

--
Regards
Liang