[GitHub] [carbondata] maheshrajus opened a new pull request #4139: [WIP] update table for primitive column not working when complex child column name and primitive column name match

classic Classic list List threaded Threaded
34 messages Options
12
Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] maheshrajus commented on a change in pull request #4139: [CARBONDATA-4191] update table for primitive column not working when complex child column name and primitive column name match

GitBox

maheshrajus commented on a change in pull request #4139:
URL: https://github.com/apache/carbondata/pull/4139#discussion_r638867906



##########
File path: integration/spark/src/main/scala/org/apache/spark/sql/parser/CarbonSpark2SqlParser.scala
##########
@@ -257,6 +258,23 @@ class CarbonSpark2SqlParser extends CarbonDDLSqlParser {
     (SET ~> "(" ~> repsep(element, ",") <~ ")") ~
     ("=" ~> restInput) <~ opt(";") ^^ {
       case tab ~ columns ~ rest =>
+        // If update is received for complex data types then throw exception
+        var finalColumns = List.empty[String]
+        var updateColumns = new ListBuffer[String]()
+        columns.foreach { column =>
+          if (column.contains('.')) {
+            val columnFullName = column.split('.')
+            if ((tab._3.isDefined && tab._3.get.equals(columnFullName(0)))
+                || tab._4.table.equals(columnFullName(0))) {
+              updateColumns += columnFullName(1)

Review comment:
       hi ajantha , i tried with below example. Carbondata not receiving command parse as spark itself throwing exception.
   Spark is allowing two level update but not three levels. Please help me if anything I am missing over here.
   
   ```
   CREATE TABLE test_rename (str1 struct<a:int>, str2 struct<a:struct<b:int>>, str3 struct<a:struct<b:struct<c:int>>>, intfield int) STORED AS carbondata
   update test_rename set(str3.a.b)=(4)
   
   ```
   ![image](https://user-images.githubusercontent.com/17046058/119519572-2d245800-bd97-11eb-90f1-700a91113999.png)
   




--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] kunal642 commented on a change in pull request #4139: [CARBONDATA-4191] update table for primitive column not working when complex child column name and primitive column name match

GitBox
In reply to this post by GitBox

kunal642 commented on a change in pull request #4139:
URL: https://github.com/apache/carbondata/pull/4139#discussion_r642830546



##########
File path: integration/spark/src/test/scala/org/apache/carbondata/integration/spark/testsuite/complexType/TestComplexDataType.scala
##########
@@ -757,14 +757,99 @@ class TestComplexDataType extends QueryTest with BeforeAndAfterAll {
     sql("DROP TABLE IF EXISTS test")
     sql("create table test(id int,a struct<b:int,c:int>,d array<int>) STORED AS carbondata")
     sql("insert into test values(1, named_struct('b', 2, 'c', 3), array(4))")
-    val structException = intercept[UnsupportedOperationException](
+    val structException = intercept[AnalysisException](
     sql("update test set(a.b)=(4) where id=1").collect())
-    assertResult("Unsupported operation on Complex data type")(structException.getMessage)
+    assert(structException.getMessage().contains("Unsupported operation on Complex data type"))
     val arrayException = intercept[UnsupportedOperationException](
     sql("update test set(a)=(4) where id=1").collect())
     assertResult("Unsupported operation on Complex data type")(arrayException.getMessage)
   }
 
+  test("check update operation on primitive data types when complex type present in table which " +
+       "has child name equal to primitive data types") {
+    sql("drop table if exists update_complex")
+    sql("create table update_complex (a int, b string, struct1 STRUCT<a:int, c:string>) " +
+        "stored as carbondata")
+    sql("insert into update_complex select 1,'c', named_struct('a',4,'b','d')")
+    sql("update update_complex set (a)=(4)")
+    checkAnswer(sql("select a from update_complex"),
+      Seq(Row(4)))
+    sql("update update_complex set (b)=('y')")

Review comment:
       can you combine these update and select commands into one.
   
   update.. set(a)
   update.. set(b) can be update.. set(a,b)
   
   same with select after doing multiple updates, fire 1 select command to verify the changes




--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] kunal642 commented on a change in pull request #4139: [CARBONDATA-4191] update table for primitive column not working when complex child column name and primitive column name match

GitBox
In reply to this post by GitBox

kunal642 commented on a change in pull request #4139:
URL: https://github.com/apache/carbondata/pull/4139#discussion_r642830546



##########
File path: integration/spark/src/test/scala/org/apache/carbondata/integration/spark/testsuite/complexType/TestComplexDataType.scala
##########
@@ -757,14 +757,99 @@ class TestComplexDataType extends QueryTest with BeforeAndAfterAll {
     sql("DROP TABLE IF EXISTS test")
     sql("create table test(id int,a struct<b:int,c:int>,d array<int>) STORED AS carbondata")
     sql("insert into test values(1, named_struct('b', 2, 'c', 3), array(4))")
-    val structException = intercept[UnsupportedOperationException](
+    val structException = intercept[AnalysisException](
     sql("update test set(a.b)=(4) where id=1").collect())
-    assertResult("Unsupported operation on Complex data type")(structException.getMessage)
+    assert(structException.getMessage().contains("Unsupported operation on Complex data type"))
     val arrayException = intercept[UnsupportedOperationException](
     sql("update test set(a)=(4) where id=1").collect())
     assertResult("Unsupported operation on Complex data type")(arrayException.getMessage)
   }
 
+  test("check update operation on primitive data types when complex type present in table which " +
+       "has child name equal to primitive data types") {
+    sql("drop table if exists update_complex")
+    sql("create table update_complex (a int, b string, struct1 STRUCT<a:int, c:string>) " +
+        "stored as carbondata")
+    sql("insert into update_complex select 1,'c', named_struct('a',4,'b','d')")
+    sql("update update_complex set (a)=(4)")
+    checkAnswer(sql("select a from update_complex"),
+      Seq(Row(4)))
+    sql("update update_complex set (b)=('y')")

Review comment:
       can you combine these update and select commands into one.
   
   update.. set(a)
   update.. set(b) can be update.. set(a,b)
   
   same with select after doing multiple updates, fire 1 select command to verify the changes
   
   
   check the same for other test cases added byyou

##########
File path: integration/spark/src/test/scala/org/apache/carbondata/integration/spark/testsuite/complexType/TestComplexDataType.scala
##########
@@ -757,14 +757,99 @@ class TestComplexDataType extends QueryTest with BeforeAndAfterAll {
     sql("DROP TABLE IF EXISTS test")
     sql("create table test(id int,a struct<b:int,c:int>,d array<int>) STORED AS carbondata")
     sql("insert into test values(1, named_struct('b', 2, 'c', 3), array(4))")
-    val structException = intercept[UnsupportedOperationException](
+    val structException = intercept[AnalysisException](
     sql("update test set(a.b)=(4) where id=1").collect())
-    assertResult("Unsupported operation on Complex data type")(structException.getMessage)
+    assert(structException.getMessage().contains("Unsupported operation on Complex data type"))
     val arrayException = intercept[UnsupportedOperationException](
     sql("update test set(a)=(4) where id=1").collect())
     assertResult("Unsupported operation on Complex data type")(arrayException.getMessage)
   }
 
+  test("check update operation on primitive data types when complex type present in table which " +
+       "has child name equal to primitive data types") {
+    sql("drop table if exists update_complex")
+    sql("create table update_complex (a int, b string, struct1 STRUCT<a:int, c:string>) " +
+        "stored as carbondata")
+    sql("insert into update_complex select 1,'c', named_struct('a',4,'b','d')")
+    sql("update update_complex set (a)=(4)")
+    checkAnswer(sql("select a from update_complex"),
+      Seq(Row(4)))
+    sql("update update_complex set (b)=('y')")

Review comment:
       can you combine these update and select commands into one.
   
   update.. set(a)
   update.. set(b) can be update.. set(a,b)
   
   same with select after doing multiple updates, fire 1 select command to verify the changes
   
   
   check the same for other test cases added by you




--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] maheshrajus commented on a change in pull request #4139: [CARBONDATA-4191] update table for primitive column not working when complex child column name and primitive column name match

GitBox
In reply to this post by GitBox

maheshrajus commented on a change in pull request #4139:
URL: https://github.com/apache/carbondata/pull/4139#discussion_r642947612



##########
File path: integration/spark/src/test/scala/org/apache/carbondata/integration/spark/testsuite/complexType/TestComplexDataType.scala
##########
@@ -757,14 +757,99 @@ class TestComplexDataType extends QueryTest with BeforeAndAfterAll {
     sql("DROP TABLE IF EXISTS test")
     sql("create table test(id int,a struct<b:int,c:int>,d array<int>) STORED AS carbondata")
     sql("insert into test values(1, named_struct('b', 2, 'c', 3), array(4))")
-    val structException = intercept[UnsupportedOperationException](
+    val structException = intercept[AnalysisException](
     sql("update test set(a.b)=(4) where id=1").collect())
-    assertResult("Unsupported operation on Complex data type")(structException.getMessage)
+    assert(structException.getMessage().contains("Unsupported operation on Complex data type"))
     val arrayException = intercept[UnsupportedOperationException](
     sql("update test set(a)=(4) where id=1").collect())
     assertResult("Unsupported operation on Complex data type")(arrayException.getMessage)
   }
 
+  test("check update operation on primitive data types when complex type present in table which " +
+       "has child name equal to primitive data types") {
+    sql("drop table if exists update_complex")
+    sql("create table update_complex (a int, b string, struct1 STRUCT<a:int, c:string>) " +
+        "stored as carbondata")
+    sql("insert into update_complex select 1,'c', named_struct('a',4,'b','d')")
+    sql("update update_complex set (a)=(4)")
+    checkAnswer(sql("select a from update_complex"),
+      Seq(Row(4)))
+    sql("update update_complex set (b)=('y')")

Review comment:
       done




--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] maheshrajus commented on a change in pull request #4139: [CARBONDATA-4191] update table for primitive column not working when complex child column name and primitive column name match

GitBox
In reply to this post by GitBox

maheshrajus commented on a change in pull request #4139:
URL: https://github.com/apache/carbondata/pull/4139#discussion_r642950598



##########
File path: integration/spark/src/main/scala/org/apache/spark/sql/parser/CarbonSpark2SqlParser.scala
##########
@@ -257,6 +258,23 @@ class CarbonSpark2SqlParser extends CarbonDDLSqlParser {
     (SET ~> "(" ~> repsep(element, ",") <~ ")") ~
     ("=" ~> restInput) <~ opt(";") ^^ {
       case tab ~ columns ~ rest =>
+        // If update is received for complex data types then throw exception
+        var finalColumns = List.empty[String]
+        var updateColumns = new ListBuffer[String]()
+        columns.foreach { column =>
+          if (column.contains('.')) {
+            val columnFullName = column.split('.')
+            if ((tab._3.isDefined && tab._3.get.equals(columnFullName(0)))
+                || tab._4.table.equals(columnFullName(0))) {
+              updateColumns += columnFullName(1)

Review comment:
       @ajantha-bhat Look like the update command has some problem when we give nested complex columns[a.b.c] from the carbon parsing side. I will check and raise the new JIRA as it is the base issue. I have to check all other impacted test cases also regarding this.




--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4139: [CARBONDATA-4191] update table for primitive column not working when complex child column name and primitive column name match

GitBox
In reply to this post by GitBox

CarbonDataQA2 commented on pull request #4139:
URL: https://github.com/apache/carbondata/pull/4139#issuecomment-852048908






--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] maheshrajus commented on pull request #4139: [CARBONDATA-4191] update table for primitive column not working when complex child column name and primitive column name match

GitBox
In reply to this post by GitBox

maheshrajus commented on pull request #4139:
URL: https://github.com/apache/carbondata/pull/4139#issuecomment-852115005


   retest this please


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4139: [CARBONDATA-4191] update table for primitive column not working when complex child column name and primitive column name match

GitBox
In reply to this post by GitBox

CarbonDataQA2 commented on pull request #4139:
URL: https://github.com/apache/carbondata/pull/4139#issuecomment-852191282


   Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/3725/
   


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4139: [CARBONDATA-4191] update table for primitive column not working when complex child column name and primitive column name match

GitBox
In reply to this post by GitBox

CarbonDataQA2 commented on pull request #4139:
URL: https://github.com/apache/carbondata/pull/4139#issuecomment-852191744


   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5469/
   


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] maheshrajus commented on pull request #4139: [CARBONDATA-4191] update table for primitive column not working when complex child column name and primitive column name match

GitBox
In reply to this post by GitBox

maheshrajus commented on pull request #4139:
URL: https://github.com/apache/carbondata/pull/4139#issuecomment-852281570


   retest this please


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4139: [CARBONDATA-4191] update table for primitive column not working when complex child column name and primitive column name match

GitBox
In reply to this post by GitBox

CarbonDataQA2 commented on pull request #4139:
URL: https://github.com/apache/carbondata/pull/4139#issuecomment-852351028


   Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/3727/
   


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4139: [CARBONDATA-4191] update table for primitive column not working when complex child column name and primitive column name match

GitBox
In reply to this post by GitBox

CarbonDataQA2 commented on pull request #4139:
URL: https://github.com/apache/carbondata/pull/4139#issuecomment-852354126


   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5471/
   


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] kunal642 commented on pull request #4139: [CARBONDATA-4191] update table for primitive column not working when complex child column name and primitive column name match

GitBox
In reply to this post by GitBox

kunal642 commented on pull request #4139:
URL: https://github.com/apache/carbondata/pull/4139#issuecomment-852767340


   LGTM


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] asfgit closed pull request #4139: [CARBONDATA-4191] update table for primitive column not working when complex child column name and primitive column name match

GitBox
In reply to this post by GitBox

asfgit closed pull request #4139:
URL: https://github.com/apache/carbondata/pull/4139


   


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


12