[GitHub] carbondata pull request #2045: [CARBONDATA-2230]Add a path into table path t...

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2045
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2902/



---

[qiuchenjian-2]

Github user zzcclp commented on the issue:

    https://github.com/apache/carbondata/pull/2045
 
    retest this please.


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2045
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3807/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2045
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2903/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2045
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4147/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2045
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3808/



---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2045#discussion_r173642041
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/locks/HdfsFileLock.java ---
    @@ -39,16 +39,18 @@
        */
       private String location;
     
    -  private DataOutputStream dataOutputStream;
    +  private String locationPath;
    --- End diff --
   
    change to `lockFilePath`


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2045#discussion_r173642070
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/locks/HdfsFileLock.java ---
    @@ -39,16 +39,18 @@
        */
       private String location;
    --- End diff --
   
    Is this required anymore?


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2045#discussion_r173642089
 
    --- Diff: integration/spark2/src/main/scala/org/apache/spark/util/AlterTableUtil.scala ---
    @@ -114,6 +114,7 @@ object AlterTableUtil {
         val lockLocation = tablePath
         locks.zip(locksAcquired).foreach { case (carbonLock, lockType) =>
           val lockFilePath = lockLocation + CarbonCommonConstants.FILE_SEPARATOR +
    --- End diff --
   
    add a static function in `CarbonTablePath` to return the lockFilePath


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2045#discussion_r173642153
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/statusmanager/SegmentStatusManager.java ---
    @@ -834,6 +836,40 @@ private static void writeLoadMetadata(AbsoluteTableIdentifier identifier,
         }
       }
     
    +  /**
    +   * Currently the segment lock files are not deleted immediately when unlock,
    +   * so it needs to delete expired lock files before delete loads.
    +   */
    +  private static void deleteExpiredSegmentLockFiles(CarbonTable carbonTable) {
    +    LoadMetadataDetails[] details =
    +        SegmentStatusManager.readLoadMetadata(carbonTable.getMetadataPath());
    +    if (details != null && details.length > 0) {
    +      AbsoluteTableIdentifier absoluteTableIdentifier = carbonTable.getAbsoluteTableIdentifier();
    +      long segmentLockFilesPreservTime =
    +          CarbonProperties.getInstance().getSegmentLockFilesPreserveHours();
    +      long currTime = System.currentTimeMillis();
    +      for (LoadMetadataDetails oneRow : details) {
    +        if (oneRow.getVisibility().equalsIgnoreCase("false") ||
    +            SegmentStatus.SUCCESS == oneRow.getSegmentStatus() ||
    +            SegmentStatus.LOAD_FAILURE == oneRow.getSegmentStatus() ||
    +            SegmentStatus.LOAD_PARTIAL_SUCCESS == oneRow.getSegmentStatus() ||
    +            SegmentStatus.COMPACTED == oneRow.getSegmentStatus()) {
    +          String location = absoluteTableIdentifier.getTablePath() +
    --- End diff --
   
    add a static function in CarbonTablePath to return the lockFilePath


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2045#discussion_r173642168
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/statusmanager/SegmentStatusManager.java ---
    @@ -834,6 +836,40 @@ private static void writeLoadMetadata(AbsoluteTableIdentifier identifier,
         }
       }
     
    +  /**
    +   * Currently the segment lock files are not deleted immediately when unlock,
    +   * so it needs to delete expired lock files before delete loads.
    +   */
    +  private static void deleteExpiredSegmentLockFiles(CarbonTable carbonTable) {
    --- End diff --
   
    move this function to CarbonTable


---

[qiuchenjian-2]

Github user zzcclp commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2045#discussion_r173716287
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/locks/HdfsFileLock.java ---
    @@ -39,16 +39,18 @@
        */
       private String location;
     
    -  private DataOutputStream dataOutputStream;
    +  private String locationPath;
    --- End diff --
   
    Done


---

[qiuchenjian-2]

Github user zzcclp commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2045#discussion_r173716459
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/locks/HdfsFileLock.java ---
    @@ -39,16 +39,18 @@
        */
       private String location;
    --- End diff --
   
    I change this to lockFileDir, it need this parameter to mkdir.


---

[qiuchenjian-2]

Github user zzcclp commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2045#discussion_r173716485
 
    --- Diff: integration/spark2/src/main/scala/org/apache/spark/util/AlterTableUtil.scala ---
    @@ -114,6 +114,7 @@ object AlterTableUtil {
         val lockLocation = tablePath
         locks.zip(locksAcquired).foreach { case (carbonLock, lockType) =>
           val lockFilePath = lockLocation + CarbonCommonConstants.FILE_SEPARATOR +
    --- End diff --
   
    Done


---

[qiuchenjian-2]

Github user zzcclp commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2045#discussion_r173716505
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/statusmanager/SegmentStatusManager.java ---
    @@ -834,6 +836,40 @@ private static void writeLoadMetadata(AbsoluteTableIdentifier identifier,
         }
       }
     
    +  /**
    +   * Currently the segment lock files are not deleted immediately when unlock,
    +   * so it needs to delete expired lock files before delete loads.
    +   */
    +  private static void deleteExpiredSegmentLockFiles(CarbonTable carbonTable) {
    +    LoadMetadataDetails[] details =
    +        SegmentStatusManager.readLoadMetadata(carbonTable.getMetadataPath());
    +    if (details != null && details.length > 0) {
    +      AbsoluteTableIdentifier absoluteTableIdentifier = carbonTable.getAbsoluteTableIdentifier();
    +      long segmentLockFilesPreservTime =
    +          CarbonProperties.getInstance().getSegmentLockFilesPreserveHours();
    +      long currTime = System.currentTimeMillis();
    +      for (LoadMetadataDetails oneRow : details) {
    +        if (oneRow.getVisibility().equalsIgnoreCase("false") ||
    +            SegmentStatus.SUCCESS == oneRow.getSegmentStatus() ||
    +            SegmentStatus.LOAD_FAILURE == oneRow.getSegmentStatus() ||
    +            SegmentStatus.LOAD_PARTIAL_SUCCESS == oneRow.getSegmentStatus() ||
    +            SegmentStatus.COMPACTED == oneRow.getSegmentStatus()) {
    +          String location = absoluteTableIdentifier.getTablePath() +
    --- End diff --
   
    Done


---

[qiuchenjian-2]

Github user zzcclp commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2045#discussion_r173716693
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/statusmanager/SegmentStatusManager.java ---
    @@ -834,6 +836,40 @@ private static void writeLoadMetadata(AbsoluteTableIdentifier identifier,
         }
       }
     
    +  /**
    +   * Currently the segment lock files are not deleted immediately when unlock,
    +   * so it needs to delete expired lock files before delete loads.
    +   */
    +  private static void deleteExpiredSegmentLockFiles(CarbonTable carbonTable) {
    --- End diff --
   
    It's better to move this function to CarbonLockUtil.
    Done


---

[qiuchenjian-2]

Github user zzcclp commented on the issue:

    https://github.com/apache/carbondata/pull/2045
 
    after (PR2051)[https://github.com/apache/carbondata/pull/2051] merged, i will rebase and commit the changes.


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2045
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2952/



---

[qiuchenjian-2]

Github user zzcclp commented on the issue:

    https://github.com/apache/carbondata/pull/2045
 
    retest this please


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2045
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4197/



---