[GitHub] carbondata pull request #1638: [CARBONDATA-1879] Support alter table to chan...

[qiuchenjian-2]

GitHub user QiangCai opened a pull request:

    https://github.com/apache/carbondata/pull/1638

    [CARBONDATA-1879] Support alter table to change the status of the streaming segment

    Support new SQL command to change the status of the segment from "streaming" to "streaming finish":
    Alter table <dbname.tablename> finish streaming
   
    1. If streaming is running, the command will throw an exception.
    2. If there isn't streaming segment, do nothing.
   
     - [x] Any interfaces changed?
     add new sql command:
      Alter table <dbname.tablename> finish streaming
     - [x] Any backward compatibility impacted?
      no
     - [x] Document update required?
      required
     - [x] Testing done
            Please provide details on
            - Whether new unit test cases have been added or why no new tests are required?
               added
            - How it is tested? Please attach test report.
            - Is it a performance related change? Please attach the performance test report.
            - Any additional information to help reviewers in testing this change.
           
     - [x] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
      NA


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/QiangCai/carbondata finish_streaming

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/1638.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1638
   
----
commit fc5318348dea70f57e967a9a38a8cd25c139b7f9
Author: QiangCai <[hidden email]>
Date:   2017-12-10T13:01:56Z

    alter table finish streaming

----


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1638
 
    Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/626/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1638
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1854/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1638
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2206/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1638
 
    Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/658/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1638
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1889/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1638
 
    Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/664/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1638
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1893/



---

[qiuchenjian-2]

Github user jackylk commented on the issue:

    https://github.com/apache/carbondata/pull/1638
 
    retest this please


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1638
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1900/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1638
 
    Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/671/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1638
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2232/



---

[qiuchenjian-2]

Github user QiangCai commented on the issue:

    https://github.com/apache/carbondata/pull/1638
 
    retest this please


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1638
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1906/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1638
 
    Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/677/



---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1638#discussion_r156596190
 
    --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonAlterTableFinishStreaming.scala ---
    @@ -0,0 +1,37 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.sql.execution.command.management
    +
    +import org.apache.spark.sql.{CarbonEnv, Row, SparkSession}
    +import org.apache.spark.sql.execution.command.DataCommand
    +
    +import org.apache.carbondata.streaming.segment.StreamSegment
    +
    +/**
    + * This command will try to change the status of the segment from "streaming" to "streaming finish"
    + */
    +case class CarbonAlterTableFinishStreaming(
    +    dbName: Option[String],
    +    tableName: String)
    +  extends DataCommand {
    --- End diff --
   
    This is not data command


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1638#discussion_r156596322
 
    --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/parser/CarbonSpark2SqlParser.scala ---
    @@ -129,6 +129,12 @@ class CarbonSpark2SqlParser extends CarbonDDLSqlParser {
             CarbonAlterTableCompactionCommand(altertablemodel)
         }
     
    +  protected lazy val alterTableFinishStreaming: Parser[LogicalPlan] =
    --- End diff --
   
    add syntax description comment


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1638#discussion_r156597163
 
    --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonAlterTableFinishStreaming.scala ---
    @@ -0,0 +1,37 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.sql.execution.command.management
    +
    +import org.apache.spark.sql.{CarbonEnv, Row, SparkSession}
    +import org.apache.spark.sql.execution.command.DataCommand
    +
    +import org.apache.carbondata.streaming.segment.StreamSegment
    +
    +/**
    + * This command will try to change the status of the segment from "streaming" to "streaming finish"
    + */
    +case class CarbonAlterTableFinishStreaming(
    +    dbName: Option[String],
    +    tableName: String)
    +  extends DataCommand {
    --- End diff --
   
    This should be metadata command


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1638#discussion_r156597691
 
    --- Diff: streaming/src/main/java/org/apache/carbondata/streaming/segment/StreamSegment.java ---
    @@ -180,6 +182,70 @@ public static String close(CarbonTable table, String segmentId)
         }
       }
     
    +  /**
    +   * change the status of the segment from "streaming" to "streaming finish"
    +   */
    +  public static void finishStreaming(CarbonTable carbonTable) throws Exception {
    +    ICarbonLock lock = CarbonLockFactory.getCarbonLockObj(
    +        carbonTable.getTableInfo().getOrCreateAbsoluteTableIdentifier(),
    +        LockUsage.TABLE_STATUS_LOCK);
    +    try {
    +      if (lock.lockWithRetries()) {
    +        ICarbonLock streamingLock = CarbonLockFactory.getCarbonLockObj(
    +            carbonTable.getTableInfo().getOrCreateAbsoluteTableIdentifier(),
    +            LockUsage.STREAMING_LOCK);
    +        try {
    +          if (streamingLock.lockWithRetries()) {
    +            LoadMetadataDetails[] details =
    +                SegmentStatusManager.readLoadMetadata(carbonTable.getMetaDataFilepath());
    +            boolean updated = false;
    +            for (LoadMetadataDetails detail : details) {
    +              if (SegmentStatus.STREAMING == detail.getSegmentStatus()) {
    +                detail.setLoadEndTime(System.currentTimeMillis());
    +                detail.setSegmentStatus(SegmentStatus.STREAMING_FINISH);
    +                updated = true;
    +              }
    +            }
    +            if (updated) {
    +              CarbonTablePath tablePath =
    +                  CarbonStorePath.getCarbonTablePath(carbonTable.getAbsoluteTableIdentifier());
    +              SegmentStatusManager.writeLoadDetailsIntoFile(
    +                  tablePath.getTableStatusFilePath(), details);
    +            }
    +          } else {
    +            String msg = "Failed to finish streaming, because streaming is locked for table " +
    +                carbonTable.getDatabaseName() + "." + carbonTable.getTableName();
    +            LOGGER.error(msg);
    +            throw new Exception(msg);
    +          }
    +        } finally {
    +          if (streamingLock.unlock()) {
    +            LOGGER.info("Table unlocked successfully after streaming finished" + carbonTable
    +                .getDatabaseName() + "." + carbonTable.getTableName());
    +          } else {
    +            LOGGER.error("Unable to unlock Table lock for table " +
    +                carbonTable.getDatabaseName() + "." + carbonTable.getTableName() +
    +                " during streaming finished");
    +          }
    +        }
    +      } else {
    +        String msg = "Failed to acquire table status lock of " +
    +            carbonTable.getDatabaseName() + "." + carbonTable.getTableName();
    +        LOGGER.error(msg);
    +        throw new Exception(msg);
    --- End diff --
   
    change to IOException


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1638#discussion_r156599142
 
    --- Diff: streaming/src/main/java/org/apache/carbondata/streaming/segment/StreamSegment.java ---
    @@ -180,6 +182,70 @@ public static String close(CarbonTable table, String segmentId)
         }
       }
     
    +  /**
    +   * change the status of the segment from "streaming" to "streaming finish"
    +   */
    +  public static void finishStreaming(CarbonTable carbonTable) throws Exception {
    +    ICarbonLock lock = CarbonLockFactory.getCarbonLockObj(
    +        carbonTable.getTableInfo().getOrCreateAbsoluteTableIdentifier(),
    +        LockUsage.TABLE_STATUS_LOCK);
    +    try {
    +      if (lock.lockWithRetries()) {
    +        ICarbonLock streamingLock = CarbonLockFactory.getCarbonLockObj(
    +            carbonTable.getTableInfo().getOrCreateAbsoluteTableIdentifier(),
    +            LockUsage.STREAMING_LOCK);
    --- End diff --
   
    This lock should be acquired first because it is acquired firstly in handoff flow


---