[GitHub] carbondata pull request #2045: [CARBONDATA-2230]Add a path into table path t...

[qiuchenjian-2]

Github user zzcclp commented on the issue:

    https://github.com/apache/carbondata/pull/2045
 
    retest this please


---

[qiuchenjian-2]

Github user zzcclp commented on the issue:

    https://github.com/apache/carbondata/pull/2045
 
    retest sdv please


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2045
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3117/



---

[qiuchenjian-2]

Github user zzcclp commented on the issue:

    https://github.com/apache/carbondata/pull/2045
 
    retest this please


---

[qiuchenjian-2]

Github user zzcclp commented on the issue:

    https://github.com/apache/carbondata/pull/2045
 
    retest sdv please


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2045
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3889/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2045
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3131/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2045
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4351/



---

[qiuchenjian-2]

Github user zzcclp commented on the issue:

    https://github.com/apache/carbondata/pull/2045
 
    @ravipesala @jackylk please help to review this, thanks.


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2045
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4370/



---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2045#discussion_r175009993
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/locks/CarbonLockUtil.java ---
    @@ -107,4 +114,36 @@ public static int getLockProperty(String property, int defaultValue) {
         }
       }
     
    +  /**
    +   * Currently the segment lock files are not deleted immediately when unlock,
    +   * so it needs to delete expired lock files before delete loads.
    +   */
    +  public static void deleteExpiredSegmentLockFiles(CarbonTable carbonTable) {
    +    LoadMetadataDetails[] details =
    +        SegmentStatusManager.readLoadMetadata(carbonTable.getMetadataPath());
    +    if (details != null && details.length > 0) {
    +      AbsoluteTableIdentifier absoluteTableIdentifier = carbonTable.getAbsoluteTableIdentifier();
    +      long segmentLockFilesPreservTime =
    +          CarbonProperties.getInstance().getSegmentLockFilesPreserveHours();
    +      long currTime = System.currentTimeMillis();
    +      for (LoadMetadataDetails oneRow : details) {
    +        if (oneRow.getVisibility().equalsIgnoreCase("false") ||
    +            SegmentStatus.SUCCESS == oneRow.getSegmentStatus() ||
    +            SegmentStatus.LOAD_FAILURE == oneRow.getSegmentStatus() ||
    +            SegmentStatus.LOAD_PARTIAL_SUCCESS == oneRow.getSegmentStatus() ||
    +            SegmentStatus.COMPACTED == oneRow.getSegmentStatus()) {
    +          String location = CarbonTablePath
    +              .getSegmentLockFilePath(absoluteTableIdentifier.getTablePath(),
    +                  oneRow.getLoadName());
    +          CarbonFile carbonFile =
    +              FileFactory.getCarbonFile(location, FileFactory.getFileType(location));
    +          if (carbonFile.exists()) {
    --- End diff --
   
    This operation will call HDFS NameNode, so if there are 1000 lock files, this operation will be for each lock files so it is very time consuming. A better way is to do a CarbonFile.listFile on the lock folder and get all CarbonFile in one NameNode call. So you do not need to read table status (code from line 122 to line 137 is not required)


---

[qiuchenjian-2]

Github user zzcclp commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2045#discussion_r175252906
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/locks/CarbonLockUtil.java ---
    @@ -107,4 +114,36 @@ public static int getLockProperty(String property, int defaultValue) {
         }
       }
     
    +  /**
    +   * Currently the segment lock files are not deleted immediately when unlock,
    +   * so it needs to delete expired lock files before delete loads.
    +   */
    +  public static void deleteExpiredSegmentLockFiles(CarbonTable carbonTable) {
    +    LoadMetadataDetails[] details =
    +        SegmentStatusManager.readLoadMetadata(carbonTable.getMetadataPath());
    +    if (details != null && details.length > 0) {
    +      AbsoluteTableIdentifier absoluteTableIdentifier = carbonTable.getAbsoluteTableIdentifier();
    +      long segmentLockFilesPreservTime =
    +          CarbonProperties.getInstance().getSegmentLockFilesPreserveHours();
    +      long currTime = System.currentTimeMillis();
    +      for (LoadMetadataDetails oneRow : details) {
    +        if (oneRow.getVisibility().equalsIgnoreCase("false") ||
    +            SegmentStatus.SUCCESS == oneRow.getSegmentStatus() ||
    +            SegmentStatus.LOAD_FAILURE == oneRow.getSegmentStatus() ||
    +            SegmentStatus.LOAD_PARTIAL_SUCCESS == oneRow.getSegmentStatus() ||
    +            SegmentStatus.COMPACTED == oneRow.getSegmentStatus()) {
    +          String location = CarbonTablePath
    +              .getSegmentLockFilePath(absoluteTableIdentifier.getTablePath(),
    +                  oneRow.getLoadName());
    +          CarbonFile carbonFile =
    +              FileFactory.getCarbonFile(location, FileFactory.getFileType(location));
    +          if (carbonFile.exists()) {
    --- End diff --
   
    Done


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2045
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4393/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2045
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3161/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2045
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3163/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2045
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4395/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2045
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3165/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2045
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4398/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2045
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4400/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2045
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3167/



---