[GitHub] [carbondata] akkio-97 opened a new pull request #3629: [WIP] Fixed spark 2.4 UT failures

[GitBox]

CarbonDataQA1 commented on issue #3629: [CARBONDATA-3716] Fixed spark 2.4 UT failures
URL: https://github.com/apache/carbondata/pull/3629#issuecomment-591339708
 
 
   Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/496/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3629: [CARBONDATA-3716] Fixed spark 2.4 UT failures
URL: https://github.com/apache/carbondata/pull/3629#issuecomment-591372337
 
 
   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2194/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3629: [CARBONDATA-3716] Fixed spark 2.4 UT failures
URL: https://github.com/apache/carbondata/pull/3629#issuecomment-591479325
 
 
   Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/502/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3629: [CARBONDATA-3716] Fixed spark 2.4 UT failures
URL: https://github.com/apache/carbondata/pull/3629#issuecomment-591523001
 
 
   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2201/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3629: [CARBONDATA-3716] Fixed spark 2.4 UT failures
URL: https://github.com/apache/carbondata/pull/3629#issuecomment-592041948
 
 
   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2213/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3629: [CARBONDATA-3716] Fixed spark 2.4 UT failures
URL: https://github.com/apache/carbondata/pull/3629#issuecomment-592424620
 
 
   Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/520/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

ydvpankaj99 commented on issue #3629: [CARBONDATA-3716] Fixed spark 2.4 UT failures
URL: https://github.com/apache/carbondata/pull/3629#issuecomment-592424698
 
 
   retest this please
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3629: [CARBONDATA-3716] Fixed spark 2.4 UT failures
URL: https://github.com/apache/carbondata/pull/3629#issuecomment-592455137
 
 
   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2220/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3629: [CARBONDATA-3716] Fixed spark 2.4 UT failures
URL: https://github.com/apache/carbondata/pull/3629#issuecomment-592461807
 
 
   Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/525/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3629: [CARBONDATA-3716] Fixed spark 2.4 UT failures
URL: https://github.com/apache/carbondata/pull/3629#issuecomment-592464376
 
 
   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2225/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

akkio-97 commented on issue #3629: [CARBONDATA-3716] Fixed spark 2.4 UT failures
URL: https://github.com/apache/carbondata/pull/3629#issuecomment-592472242
 
 
   
   
   Please review.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

Indhumathi27 commented on a change in pull request #3629: [CARBONDATA-3716] Fixed spark 2.4 UT failures
URL: https://github.com/apache/carbondata/pull/3629#discussion_r385645406
 
 

 ##########
 File path: integration/spark/src/test/scala/org/apache/carbondata/integration/spark/testsuite/binary/TestBinaryDataType.scala
 ##########
 @@ -17,15 +17,17 @@
 package org.apache.carbondata.integration.spark.testsuite.binary
 
 import java.util.Arrays
+
 import org.apache.carbondata.common.exceptions.sql.MalformedCarbonCommandException
 import org.apache.carbondata.core.constants.CarbonCommonConstants
 import org.apache.carbondata.core.metadata.CarbonMetadata
 import org.apache.carbondata.core.metadata.schema.table.CarbonTable
 import org.apache.carbondata.core.util.CarbonProperties
 import org.apache.commons.codec.binary.{Base64, Hex}
-import org.apache.spark.SparkException
+import org.apache.spark.{SPARK_VERSION, SparkException}
 
 Review comment:
   Revert if unused

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

jackylk commented on a change in pull request #3629: [CARBONDATA-3716] Fixed spark 2.4 UT failures
URL: https://github.com/apache/carbondata/pull/3629#discussion_r386008073
 
 

 ##########
 File path: integration/spark/src/main/scala/org/apache/spark/sql/execution/command/management/RefreshCarbonTableCommand.scala
 ##########
 @@ -154,18 +156,36 @@ case class RefreshCarbonTableCommand(
       tableInfo: TableInfo,
       tablePath: String)(sparkSession: SparkSession): Any = {
     val operationContext = new OperationContext
+    var allowCreateTableNonEmptyLocation: String = null
+    val allowCreateTableNonEmptyLocationConf =
+      "spark.sql.legacy.allowCreatingManagedTableUsingNonemptyLocation"
     try {
       val refreshTablePreExecutionEvent: RefreshTablePreExecutionEvent =
         new RefreshTablePreExecutionEvent(sparkSession,
           tableInfo.getOrCreateAbsoluteTableIdentifier())
+      if (SparkUtil.isSparkVersionEqualTo("2.4")) {
 
 Review comment:
   use equalToOrAbove

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

jackylk commented on a change in pull request #3629: [CARBONDATA-3716] Fixed spark 2.4 UT failures
URL: https://github.com/apache/carbondata/pull/3629#discussion_r386008081
 
 

 ##########
 File path: integration/spark/src/main/scala/org/apache/spark/sql/execution/command/management/RefreshCarbonTableCommand.scala
 ##########
 @@ -154,18 +156,36 @@ case class RefreshCarbonTableCommand(
       tableInfo: TableInfo,
       tablePath: String)(sparkSession: SparkSession): Any = {
     val operationContext = new OperationContext
+    var allowCreateTableNonEmptyLocation: String = null
+    val allowCreateTableNonEmptyLocationConf =
+      "spark.sql.legacy.allowCreatingManagedTableUsingNonemptyLocation"
     try {
       val refreshTablePreExecutionEvent: RefreshTablePreExecutionEvent =
         new RefreshTablePreExecutionEvent(sparkSession,
           tableInfo.getOrCreateAbsoluteTableIdentifier())
+      if (SparkUtil.isSparkVersionEqualTo("2.4")) {
+        // During refresh table, when this option is set to true, creating managed tables with
+        // nonempty location is allowed. Otherwise, an analysis exception is thrown.
+        // https://kb.databricks.com/jobs/spark-overwrite-cancel.html
+        allowCreateTableNonEmptyLocation = sparkSession.sessionState
+          .conf.getConfString(allowCreateTableNonEmptyLocationConf)
+        sparkSession.sessionState.conf.setConfString(allowCreateTableNonEmptyLocationConf, "true")
+      }
       OperationListenerBus.getInstance.fireEvent(refreshTablePreExecutionEvent, operationContext)
       CarbonCreateTableCommand(tableInfo, ifNotExistsSet = false, tableLocation = Some(tablePath))
         .run(sparkSession)
     } catch {
       case e: AnalysisException => throw e
       case e: Exception =>
         throw e
+    } finally {
+      if (SparkUtil.isSparkVersionEqualTo("2.4")) {
 
 Review comment:
   use equalToOrAbove

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

jackylk commented on a change in pull request #3629: [CARBONDATA-3716] Fixed spark 2.4 UT failures
URL: https://github.com/apache/carbondata/pull/3629#discussion_r386008094
 
 

 ##########
 File path: integration/spark/src/main/scala/org/apache/spark/sql/execution/command/management/RefreshCarbonTableCommand.scala
 ##########
 @@ -154,18 +156,36 @@ case class RefreshCarbonTableCommand(
       tableInfo: TableInfo,
       tablePath: String)(sparkSession: SparkSession): Any = {
     val operationContext = new OperationContext
+    var allowCreateTableNonEmptyLocation: String = null
+    val allowCreateTableNonEmptyLocationConf =
+      "spark.sql.legacy.allowCreatingManagedTableUsingNonemptyLocation"
     try {
       val refreshTablePreExecutionEvent: RefreshTablePreExecutionEvent =
         new RefreshTablePreExecutionEvent(sparkSession,
           tableInfo.getOrCreateAbsoluteTableIdentifier())
+      if (SparkUtil.isSparkVersionEqualTo("2.4")) {
+        // During refresh table, when this option is set to true, creating managed tables with
+        // nonempty location is allowed. Otherwise, an analysis exception is thrown.
+        // https://kb.databricks.com/jobs/spark-overwrite-cancel.html
+        allowCreateTableNonEmptyLocation = sparkSession.sessionState
+          .conf.getConfString(allowCreateTableNonEmptyLocationConf)
+        sparkSession.sessionState.conf.setConfString(allowCreateTableNonEmptyLocationConf, "true")
+      }
       OperationListenerBus.getInstance.fireEvent(refreshTablePreExecutionEvent, operationContext)
       CarbonCreateTableCommand(tableInfo, ifNotExistsSet = false, tableLocation = Some(tablePath))
         .run(sparkSession)
     } catch {
       case e: AnalysisException => throw e
       case e: Exception =>
         throw e
 
 Review comment:
   move to previous line

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

jackylk commented on a change in pull request #3629: [CARBONDATA-3716] Fixed spark 2.4 UT failures
URL: https://github.com/apache/carbondata/pull/3629#discussion_r386008122
 
 

 ##########
 File path: integration/spark/src/main/scala/org/apache/spark/sql/secondaryindex/Jobs/SparkBlockletDataMapLoaderJob.scala
 ##########
 @@ -167,8 +167,10 @@ class DataMapLoaderRDD(
     val reader = dataMapFormat.createRecordReader(inputSplit, attemptContext)
     val iter = new Iterator[(TableBlockIndexUniqueIdentifier, BlockletDataMapDetailsWithSchema)] {
       // in case of success, failure or cancelation clear memory and stop execution
-      context.addTaskCompletionListener { _ =>
-        reader.close()
+      context.addTaskCompletionListener {
+        new TaskCompletionListener {
+          override def onTaskCompletion(context: TaskContext): Unit = reader.close()
 
 Review comment:
   seems not required to change

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

jackylk commented on a change in pull request #3629: [CARBONDATA-3716] Fixed spark 2.4 UT failures
URL: https://github.com/apache/carbondata/pull/3629#discussion_r386008268
 
 

 ##########
 File path: integration/spark/src/main/spark2.4/org/apache/spark/sql/CarbonToSparkAdapter.scala
 ##########
 @@ -161,6 +161,12 @@ object CarbonToSparkAdapter {
     storageFormat.copy(properties = map, locationUri = Some(new URI(tablePath)))
   }
 
+  def getOutput(subQueryAlias: SubqueryAlias): Seq[Attribute] = {
+    var newAlias: Seq[String] = Seq.empty
+    newAlias = newAlias :+ subQueryAlias.name.identifier
 
 Review comment:
   ```suggestion
        val newAlias = Seq(subQueryAlias.name.identifier)
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

akashrn5 commented on a change in pull request #3629: [CARBONDATA-3716] Fixed spark 2.4 UT failures
URL: https://github.com/apache/carbondata/pull/3629#discussion_r386083558
 
 

 ##########
 File path: integration/spark/src/test/scala/org/apache/carbondata/integration/spark/testsuite/binary/TestBinaryDataType.scala
 ##########
 @@ -17,15 +17,17 @@
 package org.apache.carbondata.integration.spark.testsuite.binary
 
 import java.util.Arrays
+
 
 Review comment:
   revert

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

akashrn5 commented on a change in pull request #3629: [CARBONDATA-3716] Fixed spark 2.4 UT failures
URL: https://github.com/apache/carbondata/pull/3629#discussion_r386083590
 
 

 ##########
 File path: integration/spark/src/test/scala/org/apache/carbondata/integration/spark/testsuite/binary/TestBinaryDataType.scala
 ##########
 @@ -1640,8 +1639,25 @@ class TestBinaryDataType extends QueryTest with BeforeAndAfterAll {
                    | where CUST_ID IS NULL or DOB IS NOT NULL or BIGINT_COLUMN1 =1233720368578 or DECIMAL_COLUMN1 = 12345678901.1234000058 or Double_COLUMN1 = 1.12345674897976E10 or INTEGER_COLUMN1 IS NULL limit 10
              """.stripMargin)
         }
-        assert(e3.getMessage.contains("cannot resolve 'avg(substring(uniqdata.`CUST_NAME`, 1, 2))' due to data type mismatch: function average requires numeric types, not BinaryType"))
-
+        // Exceptions are specific to spark versions
+        val message_2_3 = "cannot resolve 'avg(substring(uniqdata.`CUST_NAME`, 1, 2))' due to data type mismatch: function average requires numeric types, not BinaryType"
+        val message_2_4 = "cannot resolve 'avg(substring(default.uniqdata.`CUST_NAME`, 1, 2))' due to data type mismatch: function average requires numeric types, not binary"
+        if(SparkUtil.isSparkVersionEqualTo("2.3")) {
 
 Review comment:
   correct the style, can bring to single line

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3629: [CARBONDATA-3716] Fixed spark 2.4 UT failures
URL: https://github.com/apache/carbondata/pull/3629#issuecomment-593243229
 
 
   Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/557/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services