[GitHub] [carbondata] akkio-97 opened a new pull request #3629: [WIP] Fixed spark 2.4 UT failures

[GitBox]

akkio-97 opened a new pull request #3629: [WIP] Fixed spark 2.4 UT failures
URL: https://github.com/apache/carbondata/pull/3629
 
 
    ### Why is this PR needed?
   
   
    ### What changes were proposed in this PR?
   
       
    ### Does this PR introduce any user interface change?
    - No
    - Yes. (please explain the change and update document)
   
    ### Is any new testcase added?
    - No
    - Yes
   
       
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3629: [WIP] Fixed spark 2.4 UT failures
URL: https://github.com/apache/carbondata/pull/3629#issuecomment-587423316
 
 
   Build Failed  with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/336/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3629: [WIP] Fixed spark 2.4 UT failures
URL: https://github.com/apache/carbondata/pull/3629#issuecomment-587445297
 
 
   Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/338/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3629: [WIP] Fixed spark 2.4 UT failures
URL: https://github.com/apache/carbondata/pull/3629#issuecomment-587480632
 
 
   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2040/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3629: [WIP] Fixed spark 2.4 UT failures
URL: https://github.com/apache/carbondata/pull/3629#issuecomment-588164162
 
 
   Build Failed  with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/354/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3629: [WIP] Fixed spark 2.4 UT failures
URL: https://github.com/apache/carbondata/pull/3629#issuecomment-588171816
 
 
   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2055/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3629: [WIP] Fixed spark 2.4 UT failures
URL: https://github.com/apache/carbondata/pull/3629#issuecomment-588230965
 
 
   Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/358/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3629: [WIP] Fixed spark 2.4 UT failures
URL: https://github.com/apache/carbondata/pull/3629#issuecomment-588273910
 
 
   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2060/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

ydvpankaj99 commented on issue #3629: Fixed spark 2.4 UT failures
URL: https://github.com/apache/carbondata/pull/3629#issuecomment-588619900
 
 
   retest this please

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3629: Fixed spark 2.4 UT failures
URL: https://github.com/apache/carbondata/pull/3629#issuecomment-588634517
 
 
   Build Failed  with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/368/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3629: Fixed spark 2.4 UT failures
URL: https://github.com/apache/carbondata/pull/3629#issuecomment-588668194
 
 
   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2070/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

ajantha-bhat commented on issue #3629: [CARBONDATA-3716] Fixed spark 2.4 UT failures
URL: https://github.com/apache/carbondata/pull/3629#issuecomment-590167339
 
 
   add to whitelist

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

ajantha-bhat commented on issue #3629: [CARBONDATA-3716] Fixed spark 2.4 UT failures
URL: https://github.com/apache/carbondata/pull/3629#issuecomment-590167356
 
 
   retest this please

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3629: [CARBONDATA-3716] Fixed spark 2.4 UT failures
URL: https://github.com/apache/carbondata/pull/3629#issuecomment-590170764
 
 
   Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/420/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3629: [CARBONDATA-3716] Fixed spark 2.4 UT failures
URL: https://github.com/apache/carbondata/pull/3629#issuecomment-590185063
 
 
   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2120/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

akashrn5 commented on a change in pull request #3629: [CARBONDATA-3716] Fixed spark 2.4 UT failures
URL: https://github.com/apache/carbondata/pull/3629#discussion_r383132813
 
 

 ##########
 File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/RefreshCarbonTableCommand.scala
 ##########
 @@ -154,10 +156,20 @@ case class RefreshCarbonTableCommand(
       tableInfo: TableInfo,
       tablePath: String)(sparkSession: SparkSession): Any = {
     val operationContext = new OperationContext
+    var allowCreateTableNonEmptyLocation: String = null
+    val allowCreateTableNonEmptyLocationConf =
+      "spark.sql.legacy.allowCreatingManagedTableUsingNonemptyLocation"
     try {
       val refreshTablePreExecutionEvent: RefreshTablePreExecutionEvent =
         new RefreshTablePreExecutionEvent(sparkSession,
           tableInfo.getOrCreateAbsoluteTableIdentifier())
+      if (SparkUtil.isSparkVersionEqualTo("2.4")) {
+        // During refresh table, when this option is set to true, creating managed tables with
 
 Review comment:
   please give a reference of document of spark , in this comment

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

akashrn5 commented on a change in pull request #3629: [CARBONDATA-3716] Fixed spark 2.4 UT failures
URL: https://github.com/apache/carbondata/pull/3629#discussion_r383133085
 
 

 ##########
 File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/RefreshCarbonTableCommand.scala
 ##########
 @@ -166,6 +178,11 @@ case class RefreshCarbonTableCommand(
       case e: Exception =>
         throw e
     }
 
 Review comment:
   please do it in finally block, else in case of any exception, it will be set as true itself

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

akashrn5 commented on issue #3629: [CARBONDATA-3716] Fixed spark 2.4 UT failures
URL: https://github.com/apache/carbondata/pull/3629#issuecomment-590216899
 
 
   please share the screenshot of the build where all the test cases are passed in 2.4 CI build

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3629: [CARBONDATA-3716] Fixed spark 2.4 UT failures
URL: https://github.com/apache/carbondata/pull/3629#issuecomment-590301550
 
 
   Build Failed  with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/433/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3629: [CARBONDATA-3716] Fixed spark 2.4 UT failures
URL: https://github.com/apache/carbondata/pull/3629#issuecomment-590312539
 
 
   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2133/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services