[GitHub] carbondata pull request #1575: [CARBONDATA-1698]Adding support for table lev...

[qiuchenjian-2]

GitHub user Xaprice opened a pull request:

    https://github.com/apache/carbondata/pull/1575

    [CARBONDATA-1698]Adding support for table level compaction configuration

    Adding support for table level compaction configuration

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/Xaprice/carbondata master

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/1575.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1575
   
----
commit 0a6ba166795872b41c8fa3fa8a5e1a2e5faa81b0
Author: 周瑾 <[hidden email]>
Date:   2017-11-27T02:30:17Z

    add support for table level compaction properties

commit fa8e847cf26f3b8daa067af792b84f5666ad3920
Author: Jin Zhou <[hidden email]>
Date:   2017-11-27T09:05:08Z

    [CARBONDATA-1698]Adding support for table level compaction configuration

commit f50cd67caf4ea9280d16f34dfe984e218634824c
Author: Jin Zhou <[hidden email]>
Date:   2017-11-27T09:45:53Z

    [CARBONDATA-1698]Adding table level compaction configuration

commit 763e22ce95b829f6a5cb43fa92a523137807a7db
Author: Jin Zhou <[hidden email]>
Date:   2017-11-27T09:46:04Z

    Merge branch 'master' of https://github.com/apache/carbondata

----


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1575
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1489/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1575
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1907/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1575
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1908/



---

[qiuchenjian-2]

Github user chenliang613 commented on the issue:

    https://github.com/apache/carbondata/pull/1575
 
    retest this please


---

[qiuchenjian-2]

Github user chenliang613 commented on the issue:

    https://github.com/apache/carbondata/pull/1575
 
    @Xaprice please add the checklist as per the below template:
   
    Be sure to do all of the following checklist to help us incorporate
    your contribution quickly and easily:
   
     - [ ] Any interfaces changed?
     
     - [ ] Any backward compatibility impacted?
     
     - [ ] Document update required?
   
     - [ ] Testing done
            Please provide details on
            - Whether new unit test cases have been added or why no new tests are required?
            - How it is tested? Please attach test report.
            - Is it a performance related change? Please attach the performance test report.
            - Any additional information to help reviewers in testing this change.
           
     - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1575
 
    Build Failed with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/393/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1575
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1523/



---

[qiuchenjian-2]

Github user chenliang613 commented on the issue:

    https://github.com/apache/carbondata/pull/1575
 
    retest this please



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1575
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1524/



---

[qiuchenjian-2]

Github user jackylk commented on the issue:

    https://github.com/apache/carbondata/pull/1575
 
    Can you describe how this feature can be used by user? What SQL syntax?


---

[qiuchenjian-2]

Github user Xaprice commented on the issue:

    https://github.com/apache/carbondata/pull/1575
 
    Be sure to do all of the following checklist to help us incorporate
    your contribution quickly and easily:
   
    - [x] Any interfaces changed?
          **no**
    - [x] Any backward compatibility impacted?
          **no**
    - [x]  Document update required?
          **Yes, data-management-on-carbondata.md has been updated**
    - [x] Testing done
        Please provide details on
        - Whether new unit test cases have been added or why no new tests are required?
            **new unit test cases added**
        - How it is tested? Please attach test report.
            **unit test and tested on cluster with 7 nodes**
        - Is it a performance related change? Please attach the performance test report.
            **no**
        - Any additional information to help reviewers in testing this change.
            **no**
   
    - [x] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
        **NOT RELATED**


---

[qiuchenjian-2]

Github user Xaprice commented on the issue:

    https://github.com/apache/carbondata/pull/1575
 
    @jackylk
    User can create table by SQL below:
    ```
     CREATE TABLE tableWithCompactionOptions(
         intField INT,
         stringField STRING
         )
     STORED BY 'carbondata'
     TBLPROPERTIES('MAJOR_COMPACTION_SIZE'='10240',
     'AUTO_LOAD_MERGE'='true',
     'COMPACTION_LEVEL_THRESHOLD'='5,6',
     'COMPACTION_PRESERVE_SEGMENTS'='10',
     'ALLOWED_COMPACTION_DAYS'='5')
    ```
    Thus user can specify compaction configurations in table level. The configurations are all optional, if not specified, corresponding configurations in carbon.properties will be used. Related document has been updated.
   



---

[qiuchenjian-2]

Github user Xaprice commented on the issue:

    https://github.com/apache/carbondata/pull/1575
 
    retest this please


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1575
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1616/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1575
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2009/



---

[qiuchenjian-2]

Github user chenliang613 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1575#discussion_r154353706
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java ---
    @@ -863,6 +863,16 @@
       public static final String TABLE_BLOCKSIZE = "table_blocksize";
       // set in column level to disable inverted index
       public static final String NO_INVERTED_INDEX = "no_inverted_index";
    +  // table property name of major compaction size
    +  public static final String TBL_PROP_MAJOR_COMPACTION_SIZE = "major_compaction_size";
    --- End diff --
   
    suggest keeping the consistent name format , how about MAJOR_COMPACTION_SIZE,  remove "TBL_PROP"


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1575
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1672/



---

[qiuchenjian-2]

Github user Xaprice commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1575#discussion_r154612395
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java ---
    @@ -863,6 +863,16 @@
       public static final String TABLE_BLOCKSIZE = "table_blocksize";
       // set in column level to disable inverted index
       public static final String NO_INVERTED_INDEX = "no_inverted_index";
    +  // table property name of major compaction size
    +  public static final String TBL_PROP_MAJOR_COMPACTION_SIZE = "major_compaction_size";
    --- End diff --
   
    TBL_PROPs removed


---

[qiuchenjian-2]

Github user Xaprice commented on the issue:

    https://github.com/apache/carbondata/pull/1575
 
    retest this please


---