[GitHub] carbondata pull request #2207: [WIP] Support flat folder for managed carbon ...

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2207
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5161/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2207
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6324/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2207
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5278/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2207
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6329/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2207
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5167/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2207
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5280/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2207
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5281/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2207
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6337/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2207
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5175/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2207
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5287/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2207
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6340/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2207
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5178/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2207
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5290/



---

[qiuchenjian-2]

Github user jackylk commented on the issue:

    https://github.com/apache/carbondata/pull/2207
 
    LGTM


---

[qiuchenjian-2]

Github user kunal642 commented on the issue:

    https://github.com/apache/carbondata/pull/2207
 
    @ravipesala I think we can avoid writing the Path directory if flat_folder is true.


---

[qiuchenjian-2]

Github user kunal642 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2207#discussion_r195635301
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/metadata/schema/table/CarbonTable.java ---
    @@ -1007,6 +1007,19 @@ public boolean canAllow(CarbonTable carbonTable, TableOperation operation) {
         return indexColumn;
       }
     
    +  /**
    +   * Whether this table supports flat folder structure, it means all data files directly written
    +   * under table path
    +   */
    +  public boolean isSupportFlatFolder() {
    +    boolean supportFlatFolder = Boolean.parseBoolean(CarbonCommonConstants.DEFAULT_FLAT_FOLDER);
    +    Map<String, String> tblProps = getTableInfo().getFactTable().getTableProperties();
    +    if (tblProps.containsKey(CarbonCommonConstants.FLAT_FOLDER)) {
    --- End diff --
   
    Can the user set FLAT_FOLDER property in CarbonProperties(applicable for all tables) instead of specifying for each table??


---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2207#discussion_r195644961
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/metadata/schema/table/CarbonTable.java ---
    @@ -1007,6 +1007,19 @@ public boolean canAllow(CarbonTable carbonTable, TableOperation operation) {
         return indexColumn;
       }
     
    +  /**
    +   * Whether this table supports flat folder structure, it means all data files directly written
    +   * under table path
    +   */
    +  public boolean isSupportFlatFolder() {
    +    boolean supportFlatFolder = Boolean.parseBoolean(CarbonCommonConstants.DEFAULT_FLAT_FOLDER);
    +    Map<String, String> tblProps = getTableInfo().getFactTable().getTableProperties();
    +    if (tblProps.containsKey(CarbonCommonConstants.FLAT_FOLDER)) {
    --- End diff --
   
    Because of old tables also the property will be applicable if property set at System level, that causes hybrid storage like some data in segment folders and some data under table path. thats why it is not set at system level


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2207
 
    @kunal642 The empty Fact folder is created because first the segment is written inside this folder like the normal flow and in the end while creating segment file it moves the data to the flat folder. We cannot simply delete the folder as the write path for segment needs to be changed and it has lot of impact on current code.


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2207
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6349/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2207
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5188/



---