Re: Support Zstd as Column Compressor

Posted by Jacky Li on Sep 13, 2018; 1:45am
URL: http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/Support-Zstd-as-Column-Compressor-tp60417p62355.html

Great, thanks for your effort.

For the lz4 task,I checked lz4 compressor (lz4-java), and found it needs the decompressed size before decompressing the data. In CarbonData V3 format, we have stored the uncompressed size of data page in ChunkCompressionMeta.total_uncompressed_size in the data file for every page.
So to implement Lz4Compressor in carbon, I think we need to use this information in the file and maybe compressor interface need to changed to add this parameter to unCompressXXX interface so that lz4 can use it.

Regards,
Jacky

> 在 2018年9月12日，下午8:35，xuchuanyin <[hidden email]> 写道：
>
> As a result of the latest implementation, I store the compressor name in the
> thrift and the old enum for compression_codec has been deprecated. This
> makes it easier to support other compressors. Take LZ4 for example, the
> following changes are required:
> 1 Implement Lz4Compressor
> 2 Add Lz4Compressor to the compressor factory as native supported compressor
>
>
> --
> Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
>