RE: Proposal to integrate QATCodec into Carbondata

Posted by Xu, Cheng A on Nov 06, 2018; 12:03am
URL: http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/Proposal-to-integrate-QATCodec-into-Carbondata-tp64916p67782.html

Yes, we have the performance number against Snappy. It's included in our proposal. The performance various depending on workloads.

> For the sort workload (input, intermediate data, output are all compression-enabled, 3TB data scale, 5 workers, 2 replica for data) with Map Reduce, using QATCodec brings 7.29% performance gain and 7.5% better compression ratio. For the sort workload (input and intermediate data are compression-enabled, 3TB data scale) with Spark, it brings 14.3% performance gain, 7.5% better compression ratio. Also we measured in Hive on MR with TPCx-BB workload [3] (3TB data scale), it brings 12.98% performance gain, 13.65% better compression ratio.

Thanks
Ferdinand Xu

-----Original Message-----
From: brijoobopanna [mailto:[hidden email]]
Sent: Monday, November 5, 2018 5:45 PM
To: [hidden email]
Subject: Re: Proposal to integrate QATCodec into Carbondata

Thanks por proposing this QATCodec

If any performance benchmarks are already available wrt Snappy or ZSTD



--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/