[GitHub] carbondata pull request #2525: [CARBONDATA-2756] refactored code to use ZSTD...

[qiuchenjian-2]

GitHub user kunal642 opened a pull request:

    https://github.com/apache/carbondata/pull/2525

    [CARBONDATA-2756] refactored code to use ZSTD compression using Reflection

    1. refactored code to use ZSTD compression using Reflection
    2. add license
   
    Be sure to do all of the following checklist to help us incorporate
    your contribution quickly and easily:
   
     - [ ] Any interfaces changed?
     
     - [ ] Any backward compatibility impacted?
     
     - [ ] Document update required?
   
     - [ ] Testing done
            Please provide details on
            - Whether new unit test cases have been added or why no new tests are required?
            - How it is tested? Please attach test report.
            - Is it a performance related change? Please attach the performance test report.
            - Any additional information to help reviewers in testing this change.
           
     - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
   


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/kunal642/carbondata zstd_fix

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/2525.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2525
   
----
commit 1cada8b2b23f846d31ab80c1592346804583d17b
Author: kunal642 <kunalkapoor642@...>
Date:   2018-07-19T06:47:04Z

    refactored code to use ZSTD compression using Reflection

----


---

[qiuchenjian-2]

Github user KanakaKumar commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2525#discussion_r203637393
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/datastore/filesystem/LocalCarbonFile.java ---
    @@ -293,7 +291,12 @@ public boolean delete() {
         } else if ("LZ4".equalsIgnoreCase(compressor)) {
           inputStream = new LZ4BlockInputStream(new FileInputStream(path));
         } else if ("ZSTD".equalsIgnoreCase(compressor)) {
    -      inputStream = new ZstdInputStream(new FileInputStream(path));
    +      try {
    +        inputStream = (InputStream) Class.forName("com.github.luben.zstd.ZstdInputStream")
    +            .getConstructors()[0].newInstance(new FileInputStream(path));
    +      } catch (ReflectiveOperationException e) {
    +        throw new RuntimeException(e);
    --- End diff --
   
    Throw IOException with a hint to  "Ensure ZSTD lib is in class path or not"


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2525
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7305/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2525
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6071/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2525
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7316/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2525
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6080/



---

[qiuchenjian-2]

Github user brijoobopanna commented on the issue:

    https://github.com/apache/carbondata/pull/2525
 
    retest this please


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2525
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7339/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2525
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6103/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2525
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5926/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2525
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7347/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2525
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6111/



---

[qiuchenjian-2]

Github user manishgupta88 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2525#discussion_r203991119
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/datastore/filesystem/LocalCarbonFile.java ---
    @@ -375,7 +381,15 @@ public DataOutputStream getDataOutputStream(String path, FileFactory.FileType fi
         } else if ("ZSTD".equalsIgnoreCase(compressor)) {
           // compression level 1 is cost-effective for sort temp file
           // which is not used for storage
    -      outputStream = new ZstdOutputStream(new FileOutputStream(path), 1);
    +      FileOutputStream fileOutputStream = new FileOutputStream(path);
    +      try {
    +        outputStream = (OutputStream) Class.forName("com.github.luben.zstd.ZstdOutputStream")
    +            .getConstructor(OutputStream.class, int.class).newInstance(fileOutputStream, 1);
    +      } catch (ReflectiveOperationException e) {
    +        throw new IOException(e);
    +      } finally {
    +        fileOutputStream.close();
    --- End diff --
   
    dont close the stream here....it should be closed by the caller after its usage is complete
    Only in case of failure you close in catch block and catch for Throwable...same comment is applicable for above code changes also..


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2525
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7359/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2525
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7382/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2525
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6143/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2525
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5958/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2525
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7396/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2525
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6160/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2525
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7400/



---