[GitHub] carbondata pull request #1736: [CARBONDATA-1904][CARBONDATA-1905] Support au...

[qiuchenjian-2]

GitHub user QiangCai opened a pull request:

    https://github.com/apache/carbondata/pull/1736

    [CARBONDATA-1904][CARBONDATA-1905] Support auto handoff and close streaming

    1. auto handoff streaming segment
    2. alter streaming table to normal table
       alter table <tableName> compact 'close_streaming'
   
     - [ ] Any interfaces changed?
     
     - [ ] Any backward compatibility impacted?
     
     - [ ] Document update required?
   
     - [ ] Testing done
            Please provide details on
            - Whether new unit test cases have been added or why no new tests are required?
            - How it is tested? Please attach test report.
            - Is it a performance related change? Please attach the performance test report.
            - Any additional information to help reviewers in testing this change.
           
     - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
   


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/QiangCai/carbondata auto_compaction

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/1736.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1736
   
----
commit bfc37e358a0d872cc9ed57fa46f992bb2ffe126e
Author: QiangCai <qiangcai@...>
Date:   2017-12-28T10:43:43Z

    support auto handoff and close streaming

----


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1736
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2407/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1736
 
    Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1183/



---

[qiuchenjian-2]

Github user QiangCai commented on the issue:

    https://github.com/apache/carbondata/pull/1736
 
    retest this please


---

[qiuchenjian-2]

Github user QiangCai commented on the issue:

    https://github.com/apache/carbondata/pull/1736
 
    retest sdv please


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1736
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2423/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1736
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2604/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1736
 
    Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1199/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1736
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2614/



---

[qiuchenjian-2]

Github user QiangCai commented on the issue:

    https://github.com/apache/carbondata/pull/1736
 
    retest this please


---

[qiuchenjian-2]

Github user watermen commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1736#discussion_r159030981
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java ---
    @@ -1459,6 +1459,13 @@
        */
       public static final String HANDOFF_SIZE = "carbon.streaming.segment.max.size";
     
    +  /**
    +   * enable auto handoff streaming segment
    +   */
    +  public static final String ENABLE_AUTO_HANDOFF = "carbon.enable.auto.handoff";
    --- End diff --
   
    I think `carbon.streaming.auto.handoff.enabled` is better.


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1736#discussion_r159031130
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java ---
    @@ -1459,6 +1459,13 @@
        */
       public static final String HANDOFF_SIZE = "carbon.streaming.segment.max.size";
     
    +  /**
    +   * enable auto handoff streaming segment
    --- End diff --
   
    Can you describe more how this feature is working, and you can copy the description to document in `/doc` folder


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1736#discussion_r159034646
 
    --- Diff: integration/spark2/src/test/scala/org/apache/spark/carbondata/TestStreamingTableOperation.scala ---
    @@ -114,6 +114,14 @@ class TestStreamingTableOperation extends QueryTest with BeforeAndAfterAll {
     
         // 14. finish streaming
         createTable(tableName = "stream_table_finish", streaming = true, withBatchLoad = true)
    --- End diff --
   
    please add a test case to *reopen* the stream property after closing stream


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1736
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2432/



---

[qiuchenjian-2]

Github user watermen commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1736#discussion_r159039529
 
    --- Diff: streaming/src/main/scala/org/apache/spark/sql/execution/streaming/CarbonAppendableStreamSink.scala ---
    @@ -80,6 +80,12 @@ class CarbonAppendableStreamSink(
         CarbonProperties.getInstance().getHandoffSize
       )
     
    +  // auto handoff
    +  private val enableAutoHandoff = hadoopConf.getBoolean(
    +    CarbonCommonConstants.ENABLE_AUTO_HANDOFF,
    --- End diff --
   
    Write the name in doc.


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1736
 
    Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1206/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1736
 
    Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1217/



---

[qiuchenjian-2]

Github user QiangCai commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1736#discussion_r159196561
 
    --- Diff: streaming/src/main/scala/org/apache/spark/sql/execution/streaming/CarbonAppendableStreamSink.scala ---
    @@ -80,6 +80,12 @@ class CarbonAppendableStreamSink(
         CarbonProperties.getInstance().getHandoffSize
       )
     
    +  // auto handoff
    +  private val enableAutoHandoff = hadoopConf.getBoolean(
    +    CarbonCommonConstants.ENABLE_AUTO_HANDOFF,
    --- End diff --
   
    ok


---

[qiuchenjian-2]

Github user QiangCai commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1736#discussion_r159197657
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java ---
    @@ -1459,6 +1459,13 @@
        */
       public static final String HANDOFF_SIZE = "carbon.streaming.segment.max.size";
     
    +  /**
    +   * enable auto handoff streaming segment
    +   */
    +  public static final String ENABLE_AUTO_HANDOFF = "carbon.enable.auto.handoff";
    --- End diff --
   
    ok


---

[qiuchenjian-2]

Github user QiangCai commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1736#discussion_r159197778
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java ---
    @@ -1459,6 +1459,13 @@
        */
       public static final String HANDOFF_SIZE = "carbon.streaming.segment.max.size";
     
    +  /**
    +   * enable auto handoff streaming segment
    --- End diff --
   
    ok, I will raise pr to finish streaming doc


---