[GitHub] [carbondata] liuhe0702 opened a new pull request #4103: [CARBONDATA-4145] Query fails and the message "File does not exist: xxx.carbondata" is displayed

[GitBox]

liuhe0702 opened a new pull request #4103:
URL: https://github.com/apache/carbondata/pull/4103


   Why is this PR needed?
   If an exception occurs when the refresh index command is executed, a task has been successful. The new query will be failed.
   Reason: After the compaction task is executed successfully, the old carbondata files are deleted. If other exception occurs, the deleted files are missing.
   This PR will fix this issue.
   
   What changes were proposed in this PR?
   When all tasks are successful, the driver deletes the old carbondata files.
   
   Does this PR introduce any user interface change?
   No
   
   Is any new testcase added?
   No


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4103:
URL: https://github.com/apache/carbondata/pull/4103#issuecomment-795183121


   Can one of the admins verify this patch?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

ydvpankaj99 commented on pull request #4103:
URL: https://github.com/apache/carbondata/pull/4103#issuecomment-795233072


   retest this please


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

brijoobopanna commented on pull request #4103:
URL: https://github.com/apache/carbondata/pull/4103#issuecomment-796452540


   add to whitelist


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4103:
URL: https://github.com/apache/carbondata/pull/4103#issuecomment-796525544


   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5546/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4103:
URL: https://github.com/apache/carbondata/pull/4103#issuecomment-796528979


   Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3781/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

liuhe0702 commented on pull request #4103:
URL: https://github.com/apache/carbondata/pull/4103#issuecomment-796563879


   retest this please


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4103:
URL: https://github.com/apache/carbondata/pull/4103#issuecomment-796642629


   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5551/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4103:
URL: https://github.com/apache/carbondata/pull/4103#issuecomment-796647063


   Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3785/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

liuhe0702 commented on pull request #4103:
URL: https://github.com/apache/carbondata/pull/4103#issuecomment-796702647


   retest this please


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4103:
URL: https://github.com/apache/carbondata/pull/4103#issuecomment-796790637


   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5557/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4103:
URL: https://github.com/apache/carbondata/pull/4103#issuecomment-796798043


   Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3791/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

akashrn5 commented on pull request #4103:
URL: https://github.com/apache/carbondata/pull/4103#issuecomment-800030753


   i think this scenario is also handled by #3988 , @ShreelekhyaG please confirm if its handled


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

ShreelekhyaG commented on pull request #4103:
URL: https://github.com/apache/carbondata/pull/4103#issuecomment-800037622


   yes, it is handled in #3988


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

akashrn5 commented on pull request #4103:
URL: https://github.com/apache/carbondata/pull/4103#issuecomment-800041408


   @liuhe0702 please make the jira as duplicate and comment there as its handled in #3988 and close both jira and PR, thanks


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

liuhe0702 closed pull request #4103:
URL: https://github.com/apache/carbondata/pull/4103


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

akashrn5 commented on a change in pull request #4103:
URL: https://github.com/apache/carbondata/pull/4103#discussion_r596897917



##########
File path: integration/spark/src/main/scala/org/apache/spark/sql/secondaryindex/util/SecondaryIndexUtil.scala
##########
@@ -345,6 +353,14 @@ object SecondaryIndexUtil {
     }
   }
 
+
+  private def deleteOldCarbonDataFiles(partition: CarbonSparkPartition): Unit = {

Review comment:
       can u please add a method level comment why this method is used, basically the scenario




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

liuhe0702 commented on a change in pull request #4103:
URL: https://github.com/apache/carbondata/pull/4103#discussion_r596914587



##########
File path: integration/spark/src/main/scala/org/apache/spark/sql/secondaryindex/util/SecondaryIndexUtil.scala
##########
@@ -345,6 +353,14 @@ object SecondaryIndexUtil {
     }
   }
 
+
+  private def deleteOldCarbonDataFiles(partition: CarbonSparkPartition): Unit = {

Review comment:
       done




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

akashrn5 commented on pull request #4103:
URL: https://github.com/apache/carbondata/pull/4103#issuecomment-801997056


   LGTM, can be merged once build passes


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA2 commented on pull request #4103:
URL: https://github.com/apache/carbondata/pull/4103#issuecomment-802140241


   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5591/
   


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]