[GitHub] carbondata pull request #2045: [CARBONDATA-2230]Add a path into table path t...

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2045
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3849/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2045
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4199/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2045
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2954/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2045
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2956/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2045
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4201/



---

[qiuchenjian-2]

Github user zzcclp commented on the issue:

    https://github.com/apache/carbondata/pull/2045
 
    retest this please


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2045
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3852/



---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2045#discussion_r173829328
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/locks/CarbonLockUtil.java ---
    @@ -107,4 +115,37 @@ public static int getLockProperty(String property, int defaultValue) {
         }
       }
     
    +  /**
    +   * Currently the segment lock files are not deleted immediately when unlock,
    +   * so it needs to delete expired lock files before delete loads.
    +   */
    +  public static void deleteExpiredSegmentLockFiles(CarbonTable carbonTable) {
    +    LoadMetadataDetails[] details =
    +        SegmentStatusManager.readLoadMetadata(carbonTable.getMetadataPath());
    +    if (details != null && details.length > 0) {
    +      AbsoluteTableIdentifier absoluteTableIdentifier = carbonTable.getAbsoluteTableIdentifier();
    +      long segmentLockFilesPreservTime =
    +          CarbonProperties.getInstance().getSegmentLockFilesPreserveHours();
    +      long currTime = System.currentTimeMillis();
    +      for (LoadMetadataDetails oneRow : details) {
    +        if (oneRow.getVisibility().equalsIgnoreCase("false") ||
    +            SegmentStatus.SUCCESS == oneRow.getSegmentStatus() ||
    +            SegmentStatus.LOAD_FAILURE == oneRow.getSegmentStatus() ||
    +            SegmentStatus.LOAD_PARTIAL_SUCCESS == oneRow.getSegmentStatus() ||
    +            SegmentStatus.COMPACTED == oneRow.getSegmentStatus()) {
    +          String location = CarbonTablePath
    +              .getLockFilesDirPath(absoluteTableIdentifier.getTablePath()) +
    +              CarbonCommonConstants.FILE_SEPARATOR +
    +              CarbonTablePath.addSegmentPrefix(oneRow.getLoadName()) + LockUsage.LOCK;
    --- End diff --
   
    Put this string concatenate logic into one function in CarbonTablePath


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2045#discussion_r173829652
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/locks/CarbonLockUtil.java ---
    @@ -107,4 +115,37 @@ public static int getLockProperty(String property, int defaultValue) {
         }
       }
     
    +  /**
    +   * Currently the segment lock files are not deleted immediately when unlock,
    +   * so it needs to delete expired lock files before delete loads.
    +   */
    +  public static void deleteExpiredSegmentLockFiles(CarbonTable carbonTable) {
    +    LoadMetadataDetails[] details =
    +        SegmentStatusManager.readLoadMetadata(carbonTable.getMetadataPath());
    +    if (details != null && details.length > 0) {
    +      AbsoluteTableIdentifier absoluteTableIdentifier = carbonTable.getAbsoluteTableIdentifier();
    +      long segmentLockFilesPreservTime =
    +          CarbonProperties.getInstance().getSegmentLockFilesPreserveHours();
    +      long currTime = System.currentTimeMillis();
    +      for (LoadMetadataDetails oneRow : details) {
    +        if (oneRow.getVisibility().equalsIgnoreCase("false") ||
    +            SegmentStatus.SUCCESS == oneRow.getSegmentStatus() ||
    +            SegmentStatus.LOAD_FAILURE == oneRow.getSegmentStatus() ||
    +            SegmentStatus.LOAD_PARTIAL_SUCCESS == oneRow.getSegmentStatus() ||
    +            SegmentStatus.COMPACTED == oneRow.getSegmentStatus()) {
    +          String location = CarbonTablePath
    +              .getLockFilesDirPath(absoluteTableIdentifier.getTablePath()) +
    +              CarbonCommonConstants.FILE_SEPARATOR +
    +              CarbonTablePath.addSegmentPrefix(oneRow.getLoadName()) + LockUsage.LOCK;
    +          CarbonFile carbonFile =
    +              FileFactory.getCarbonFile(location, FileFactory.getFileType(location));
    +          if (carbonFile.exists()) {
    --- End diff --
   
    you can call FileFactory.isFileExist directly


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2045#discussion_r173829976
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/locks/CarbonLockUtil.java ---
    @@ -107,4 +115,37 @@ public static int getLockProperty(String property, int defaultValue) {
         }
       }
     
    +  /**
    +   * Currently the segment lock files are not deleted immediately when unlock,
    +   * so it needs to delete expired lock files before delete loads.
    +   */
    +  public static void deleteExpiredSegmentLockFiles(CarbonTable carbonTable) {
    +    LoadMetadataDetails[] details =
    +        SegmentStatusManager.readLoadMetadata(carbonTable.getMetadataPath());
    +    if (details != null && details.length > 0) {
    +      AbsoluteTableIdentifier absoluteTableIdentifier = carbonTable.getAbsoluteTableIdentifier();
    +      long segmentLockFilesPreservTime =
    +          CarbonProperties.getInstance().getSegmentLockFilesPreserveHours();
    +      long currTime = System.currentTimeMillis();
    +      for (LoadMetadataDetails oneRow : details) {
    +        if (oneRow.getVisibility().equalsIgnoreCase("false") ||
    +            SegmentStatus.SUCCESS == oneRow.getSegmentStatus() ||
    +            SegmentStatus.LOAD_FAILURE == oneRow.getSegmentStatus() ||
    +            SegmentStatus.LOAD_PARTIAL_SUCCESS == oneRow.getSegmentStatus() ||
    +            SegmentStatus.COMPACTED == oneRow.getSegmentStatus()) {
    +          String location = CarbonTablePath
    +              .getLockFilesDirPath(absoluteTableIdentifier.getTablePath()) +
    +              CarbonCommonConstants.FILE_SEPARATOR +
    +              CarbonTablePath.addSegmentPrefix(oneRow.getLoadName()) + LockUsage.LOCK;
    +          CarbonFile carbonFile =
    +              FileFactory.getCarbonFile(location, FileFactory.getFileType(location));
    +          if (carbonFile.exists()) {
    --- End diff --
   
    And add one static function in FileFactory:
    `public static boolean isFileExist(String filePath) throws IOException`


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2045
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4208/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2045
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2963/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2045
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2969/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2045
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4214/



---

[qiuchenjian-2]

Github user zzcclp commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2045#discussion_r173886926
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/locks/CarbonLockUtil.java ---
    @@ -107,4 +115,37 @@ public static int getLockProperty(String property, int defaultValue) {
         }
       }
     
    +  /**
    +   * Currently the segment lock files are not deleted immediately when unlock,
    +   * so it needs to delete expired lock files before delete loads.
    +   */
    +  public static void deleteExpiredSegmentLockFiles(CarbonTable carbonTable) {
    +    LoadMetadataDetails[] details =
    +        SegmentStatusManager.readLoadMetadata(carbonTable.getMetadataPath());
    +    if (details != null && details.length > 0) {
    +      AbsoluteTableIdentifier absoluteTableIdentifier = carbonTable.getAbsoluteTableIdentifier();
    +      long segmentLockFilesPreservTime =
    +          CarbonProperties.getInstance().getSegmentLockFilesPreserveHours();
    +      long currTime = System.currentTimeMillis();
    +      for (LoadMetadataDetails oneRow : details) {
    +        if (oneRow.getVisibility().equalsIgnoreCase("false") ||
    +            SegmentStatus.SUCCESS == oneRow.getSegmentStatus() ||
    +            SegmentStatus.LOAD_FAILURE == oneRow.getSegmentStatus() ||
    +            SegmentStatus.LOAD_PARTIAL_SUCCESS == oneRow.getSegmentStatus() ||
    +            SegmentStatus.COMPACTED == oneRow.getSegmentStatus()) {
    +          String location = CarbonTablePath
    +              .getLockFilesDirPath(absoluteTableIdentifier.getTablePath()) +
    +              CarbonCommonConstants.FILE_SEPARATOR +
    +              CarbonTablePath.addSegmentPrefix(oneRow.getLoadName()) + LockUsage.LOCK;
    --- End diff --
   
    Done


---

[qiuchenjian-2]

Github user zzcclp commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2045#discussion_r173887885
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/locks/CarbonLockUtil.java ---
    @@ -107,4 +115,37 @@ public static int getLockProperty(String property, int defaultValue) {
         }
       }
     
    +  /**
    +   * Currently the segment lock files are not deleted immediately when unlock,
    +   * so it needs to delete expired lock files before delete loads.
    +   */
    +  public static void deleteExpiredSegmentLockFiles(CarbonTable carbonTable) {
    +    LoadMetadataDetails[] details =
    +        SegmentStatusManager.readLoadMetadata(carbonTable.getMetadataPath());
    +    if (details != null && details.length > 0) {
    +      AbsoluteTableIdentifier absoluteTableIdentifier = carbonTable.getAbsoluteTableIdentifier();
    +      long segmentLockFilesPreservTime =
    +          CarbonProperties.getInstance().getSegmentLockFilesPreserveHours();
    +      long currTime = System.currentTimeMillis();
    +      for (LoadMetadataDetails oneRow : details) {
    +        if (oneRow.getVisibility().equalsIgnoreCase("false") ||
    +            SegmentStatus.SUCCESS == oneRow.getSegmentStatus() ||
    +            SegmentStatus.LOAD_FAILURE == oneRow.getSegmentStatus() ||
    +            SegmentStatus.LOAD_PARTIAL_SUCCESS == oneRow.getSegmentStatus() ||
    +            SegmentStatus.COMPACTED == oneRow.getSegmentStatus()) {
    +          String location = CarbonTablePath
    +              .getLockFilesDirPath(absoluteTableIdentifier.getTablePath()) +
    +              CarbonCommonConstants.FILE_SEPARATOR +
    +              CarbonTablePath.addSegmentPrefix(oneRow.getLoadName()) + LockUsage.LOCK;
    +          CarbonFile carbonFile =
    +              FileFactory.getCarbonFile(location, FileFactory.getFileType(location));
    +          if (carbonFile.exists()) {
    --- End diff --
   
    Added static function 'isFileExist(String filePath)' in FileFactory.
    But I didn't use this function directly, it still needs to use 'CarbonFile' to get the last modified time below.


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2045
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4215/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2045
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2970/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2045
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3859/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2045
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3860/



---