Re: Support Zstd as Column Compressor

Posted by xuchuanyin on Sep 13, 2018; 2:33am
URL: http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/Support-Zstd-as-Column-Compressor-tp60417p62363.html

Snappy and Zstd both know the decompress size of the content since they stored that size along with the compressed content. But LZ4 didn't do this, you can refer to the issue#26 in the lz4-java github page.




To work around this, You can store the original size in metadata for decompression. But I would like to store the size along with the compressed content to keep the interface stable.


1. While compressing, you can return the compressed content prefixed with original size


2. While decompressing, you can extract the original size and init the dest buffer




Please let me know your choice.