Login  Register

[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #4035: [CARBONDATA-4067]: Removing force option in clean files command and changing behaviour when MFD, Compacted and stale Inprogress segments can be deleted

Posted by GitBox on Dec 03, 2020; 3:50pm
URL: http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/GitHub-carbondata-vikramahuja1001-opened-a-new-pull-request-4035-WIP-CleanFiles-Behaviour-Change-tp104048p104209.html


ajantha-bhat commented on a change in pull request #4035:
URL: https://github.com/apache/carbondata/pull/4035#discussion_r535355003



##########
File path: core/src/main/java/org/apache/carbondata/core/util/DeleteLoadFolders.java
##########
@@ -173,42 +176,73 @@ public boolean accept(CarbonFile file) {
   }
 
   private static boolean checkIfLoadCanBeDeleted(LoadMetadataDetails oneLoad,
-      boolean isForceDelete) {
-    if ((SegmentStatus.MARKED_FOR_DELETE == oneLoad.getSegmentStatus() ||
-        SegmentStatus.COMPACTED == oneLoad.getSegmentStatus() ||
-        SegmentStatus.INSERT_IN_PROGRESS == oneLoad.getSegmentStatus() ||
-        SegmentStatus.INSERT_OVERWRITE_IN_PROGRESS == oneLoad.getSegmentStatus())
-        && oneLoad.getVisibility().equalsIgnoreCase("true")) {
-      if (isForceDelete) {
-        return true;
-      }
-      long deletionTime = oneLoad.getModificationOrDeletionTimestamp();
-      return TrashUtil.isTrashRetentionTimeoutExceeded(deletionTime) && CarbonUpdateUtil
-          .isMaxQueryTimeoutExceeded(deletionTime);
+      boolean cleanStaleInProgress, boolean isForceDeletion) {
+    if (oneLoad.getVisibility().equalsIgnoreCase("true")) {
+      return checkLoadDeletionLogic(oneLoad, isForceDeletion, cleanStaleInProgress);
     }
-
     return false;
   }
 
   private static boolean checkIfLoadCanBeDeletedPhysically(LoadMetadataDetails oneLoad,
-      boolean isForceDelete) {
+      boolean isForceDeletion, boolean cleanStaleInProgress) {
     // Check if the segment is added externally and path is set then do not delete it
-    if ((SegmentStatus.MARKED_FOR_DELETE == oneLoad.getSegmentStatus()
-        || SegmentStatus.COMPACTED == oneLoad.getSegmentStatus()) && (oneLoad.getPath() == null
-        || oneLoad.getPath().equalsIgnoreCase("NA"))) {
-      if (isForceDelete) {
-        return true;
-      }
-      long deletionTime = oneLoad.getModificationOrDeletionTimestamp();
-
-      return TrashUtil.isTrashRetentionTimeoutExceeded(deletionTime) && CarbonUpdateUtil
-          .isMaxQueryTimeoutExceeded(deletionTime);
-
+    if (oneLoad.getPath() == null || oneLoad.getPath().equalsIgnoreCase("NA")) {
+      return checkLoadDeletionLogic(oneLoad,  isForceDeletion, cleanStaleInProgress);
     }
-
     return false;
   }
 
+  private static Boolean checkLoadDeletionLogic(LoadMetadataDetails oneLoad,
+      boolean isForceDeletion, boolean cleanStaleInProgress) {
+    /*
+     * if cleanStaleInProgress == false and  force == false, clean MFD and Compacted
+     *  segments will depend on query timeout(1 hr) and trashRetentionTimeout(7 days, default).
+     *  For example:
+     *  If trashRetentionTimeout is 7 days and query timeout is 1 hr--> Delete after 7 days
+     *  If trashRetentionTimeout is 0 days and query timeout is 1 hr--> Delete after 1 hr
+     *
+     * if cleanStaleInProgress == false and  force == true, clean MFD and Compacted
+     *  segments immediately(Do not check for any timeout)
+     *
+     * if cleanStaleInProgress == true and  force == false, clean Stale Inprogress, MFD and
+     *  compacted segments after 7 days(taking carbon.trash.retention.time value)
+     *
+     * if cleanStaleInProgress == true and  force == true, clean MFD, Compacted and
+     *  stale inprogress segments immediately.(Do not check for any timeout)
+     */
+    if (isForceDeletion) {
+      // immediately delete compacted and MFD
+      if (cleanStaleInProgress) {
+        // immediately delete inprogress segments too
+        return SegmentStatus.INSERT_IN_PROGRESS == oneLoad.getSegmentStatus() || SegmentStatus
+            .INSERT_OVERWRITE_IN_PROGRESS == oneLoad.getSegmentStatus() || SegmentStatus
+            .COMPACTED == oneLoad.getSegmentStatus() || SegmentStatus.MARKED_FOR_DELETE ==

Review comment:
       move COMPACTED and MARKED_FOR_DELETE above and return if true.
   check only INSERT_OVERWRITE_IN_PROGRESS, INSERT_IN_PROGRESS in INSERT_IN_PROGRESS




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]