[GitHub] carbondata pull request #2647: [WIP] Added Spark FileFormat interface implem...

[qiuchenjian-2]

GitHub user ravipesala opened a pull request:

    https://github.com/apache/carbondata/pull/2647

    [WIP] Added Spark FileFormat interface implementation in Carbon

    Be sure to do all of the following checklist to help us incorporate
    your contribution quickly and easily:
   
     - [ ] Any interfaces changed?
     
     - [ ] Any backward compatibility impacted?
     
     - [ ] Document update required?
   
     - [ ] Testing done
            Please provide details on
            - Whether new unit test cases have been added or why no new tests are required?
            - How it is tested? Please attach test report.
            - Is it a performance related change? Please attach the performance test report.
            - Any additional information to help reviewers in testing this change.
           
     - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
   


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/ravipesala/incubator-carbondata spark-fileformat

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/2647.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2647
   
----
commit 4df28b3ac90fbd26ff674c10dd4ae3d3a95f956a
Author: ravipesala <ravi.pesala@...>
Date:   2018-08-22T06:02:21Z

    Added Spark FileFormat interface implementation in Carbon

----


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2647
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6316/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2647
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7966/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2647
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6690/



---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2647#discussion_r211872774
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/readcommitter/LatestFilesReadCommittedScope.java ---
    @@ -66,7 +68,23 @@ public LatestFilesReadCommittedScope(String path, String segmentId) {
        * @param path carbon file path
        */
       public LatestFilesReadCommittedScope(String path) {
    -    this(path, null);
    +    this(path, (String) null);
    +  }
    +
    +  /**
    +   * a new constructor with path
    +   *
    +   * @param path carbon file path
    +   */
    --- End diff --
   
    missing parameter doc


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2647#discussion_r211872899
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/readcommitter/LatestFilesReadCommittedScope.java ---
    @@ -36,12 +36,14 @@
      */
     @InterfaceAudience.Internal
     @InterfaceStability.Stable
    -public class LatestFilesReadCommittedScope implements ReadCommittedScope {
    +public class LatestFilesReadCommittedScope
    +    implements ReadCommittedScope {
    --- End diff --
   
    no need to modify


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2647#discussion_r211873452
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/readcommitter/LatestFilesReadCommittedScope.java ---
    @@ -66,7 +68,23 @@ public LatestFilesReadCommittedScope(String path, String segmentId) {
        * @param path carbon file path
        */
       public LatestFilesReadCommittedScope(String path) {
    -    this(path, null);
    +    this(path, (String) null);
    +  }
    +
    +  /**
    +   * a new constructor with path
    +   *
    +   * @param path carbon file path
    +   */
    +  public LatestFilesReadCommittedScope(String path, String[] subFolders) {
    +    Objects.requireNonNull(path);
    +    this.carbonFilePath = path;
    +    this.subFolders = subFolders;
    +    try {
    +      takeCarbonIndexFileSnapShot();
    +    } catch (IOException ex) {
    +      throw new RuntimeException("Error while taking index snapshot", ex);
    --- End diff --
   
    Can we use more specific exception


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2647#discussion_r211873872
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/readcommitter/LatestFilesReadCommittedScope.java ---
    @@ -66,7 +68,23 @@ public LatestFilesReadCommittedScope(String path, String segmentId) {
        * @param path carbon file path
        */
       public LatestFilesReadCommittedScope(String path) {
    -    this(path, null);
    +    this(path, (String) null);
    +  }
    +
    +  /**
    +   * a new constructor with path
    +   *
    +   * @param path carbon file path
    +   */
    +  public LatestFilesReadCommittedScope(String path, String[] subFolders) {
    +    Objects.requireNonNull(path);
    +    this.carbonFilePath = path;
    +    this.subFolders = subFolders;
    --- End diff --
   
    suggest change `subFolders` to `dataFolders` and add javadoc to explain it contains carbondata files and carbonindex files in these folders


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2647
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6319/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2647
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6693/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2647
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7969/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2647
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6324/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2647
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6699/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2647
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7975/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2647
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6326/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2647
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6701/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2647
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7977/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2647
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6331/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2647
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6332/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2647
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7983/



---