[GitHub] carbondata pull request #1361: [CARBONDATA-1481]Compaction support global so...

[qiuchenjian-2]

GitHub user xubo245 opened a pull request:

    https://github.com/apache/carbondata/pull/1361

    [CARBONDATA-1481]Compaction support global sort

    We should add some test cases and evaluate performance for Compaction support global_sort.
   
    Test cases:
    1. compaction type: major and monor
    2. parameter: global_sort_partitions
    3. scort_column: data type
    4. data size
    5. load times
    6. clean files and delete by segment.id
   
    performance evaluation:
    It should faster for querying after loading 1000 times and compaction
   
    This PR is based on https://github.com/apache/carbondata/pull/1321

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/xubo245/carbondata compactionSupportGlobalSort

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/1361.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1361
   
----
commit 528deb7157f84b52890aea2751290793d1ae793f
Author: chenerlu <[hidden email]>
Date:   2017-09-04T12:54:55Z

    Unify the sort column and sort scope in create table command

commit 0982d96efaef6d39a69922d2e3e3d7e274cf900d
Author: chenerlu <[hidden email]>
Date:   2017-09-11T09:29:48Z

    move sort scope from segment level to table level

commit 506096dc6edb2c504092725033613fdf4f8c6984
Author: chenerlu <[hidden email]>
Date:   2017-09-11T15:59:50Z

    move sort scope from segment level to table level

commit 2094f70658a54a75fd9bdb20781cb4c243a3506a
Author: chenerlu <[hidden email]>
Date:   2017-09-11T16:57:06Z

    move sort scope from segment level to table level

commit 8944905cec30a900d32c94aea30456f20e0ae968
Author: chenerlu <[hidden email]>
Date:   2017-09-11T17:10:07Z

    Fix scala stype

commit cad37cd9cdc1c62dfb8bc961011760dfc4e41f84
Author: chenerlu <[hidden email]>
Date:   2017-09-12T00:28:43Z

    fix compile error

commit 867b7b35b8b9da8891d07e12adbedf949125258d
Author: chenerlu <[hidden email]>
Date:   2017-09-12T01:59:10Z

    fix TestCreateTableWithSortScope

commit eb21c48481631995444138ff9d52d43eee45ead3
Author: xubo245 <[hidden email]>
Date:   2017-09-13T12:15:39Z

    compaction supprt global_sort

----


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1361
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/175/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1361
 
    Build Failed with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/51/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1361
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/805/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1361
 
    Build Failed with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/56/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1361
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/180/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1361
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/810/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1361
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/191/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1361
 
    Build Failed with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/68/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1361
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/822/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1361
 
    Build Failed with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/69/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1361
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/192/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1361
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/823/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1361
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/824/



---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1361#discussion_r139307748
 
    --- Diff: integration/spark-common-cluster-test/src/test/scala/org/apache/carbondata/cluster/sdv/generated/BatchSortLoad3TestCase.scala ---
    @@ -112,33 +112,15 @@ class BatchSortLoad3TestCase extends QueryTest with BeforeAndAfterAll {
         sql(s"""drop table if exists t3""").collect
       }
     
    -
    -  //Batch_sort_Loading_001-01-01-01_001-TC_056
    -  test("Batch_sort_Loading_001-01-01-01_001-TC_056", Include) {
    --- End diff --
   
    why removing it?


---

[qiuchenjian-2]

Github user jackylk commented on the issue:

    https://github.com/apache/carbondata/pull/1361
 
    please rebase to master


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1361
 
    Build Success with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/81/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1361
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/204/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1361
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/836/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1361
 
    Build Success with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/82/



---