http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/Discussion-Compression-for-sort-temp-files-in-Carbomdata-tp31747p32613.html
I think this is a good proposal. GC is performance killer for carbon since carbon need to sort data when loading.
> 在 2017年12月20日，上午11:47，徐传印 <
[hidden email]> 写道：
>
> Hi, dev:
>
> PS: Sorry for the bad format in previous two letters, please refer to this one.
>
> Recently I found the bug in compressing sort temp file and tried to fix this bug in PR#1632 (
https://github.com/apache/carbondata/pull/1632). In this PR, Carbondata will compress the records in batch and write the compressed content to file if we turn on this feature. However, I found that the GC performance is terrible. In my scenario, about half of the time were wasted in GC. And the overall performance is worse than before.
>
> I think the problem may lie in compressing the records by batch. Instead of this, I propose to compress the sort temp file in file level, not in record-batch level.
>
> 1. Compared with uncompressed ones, compressing the file in record-batch level leads to different layout of file. And it also affects the reading/writing behavior.
>
> (The compressed: |total_entry_number|batch_entry_numer|compressed_length|compressed_content|batch_entry_numer|compressed_length|compressed_content|...;
>
> The uncompressed: |total_entry_number|record|record|...;)
>
> 2. During compressing/uncompressing the record-batch, we have to store the bytes in temporary memory. If the size is big, it directly goes into JVM old generation, which will cause FULL GC frequently. I also tried to reuse this temporary memory, but it can only be reusable in file level -- We need to allocate the memory for each file. If the number of intermediate files are big, frequent FULL GC is still inevitable.
>
> If the size is small, we will need to store more `batch_entry_numer`(described in point1).
>
> Note that, the size is rowSize*batchSize. In previous implementation, Carbondata use 2MB bytes to store one row.
>
> 3. Using file level compression will simply the code since CompressedStream is also an Stream, which will not affect the behavior in reading/writing compressed/uncompressed files.
>
> 4. After I used file level compression, the GC problem disappeared. Since my cluster has crashed, I didn't get the actual performace enhanced. But seeing from the Carbondata maven tests, the most time consuming module `Spark Common Test` takes less time to complete comparing with uncompressed.
>
> Time consumed in `Spark Common Test` module:
>
> | Compressor | Time Consumed |
> | --- | --- |
> | None | 19:25min |
> | SNAPPY | 18:38min |
> | LZ4 | 19:12min |
> | GZIP | 20:32min |
> | BZIP2 | 21:10min |
>
>
> In conclusion, I think file level compression is better and I plan to remove the record-batch leve compression related code in Carbondata.