[GitHub] [carbondata] marchpure opened a new pull request #3981: [CARBONDATA-4027] Incorrect query result after Update/Delete and Inse…

[GitBox]

marchpure opened a new pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981


   …rt overwrite partition
   
    ### Why is this PR needed?
   If update/delete some records on one partition, then inserts overwrite another partition. Deleted records before will be queried out.
   
    ### What changes were proposed in this PR?
   When insert overwrite (or drop) partition, the updatestatusfilename in tablestatus will be tampered, which is fixed in this PR.
       
    ### Does this PR introduce any user interface change?
    - No
   
    ### Is any new testcase added?
    - Yes
   
       
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-707816761


   Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2654/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-707816884


   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4408/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

marchpure commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-707818001


   retest this please


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-707889628


   Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2656/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-707894635


   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4410/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-707955314


   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4412/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-707956430


   Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2658/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-708097990


   Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2661/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-708100450


   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4415/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

marchpure commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-708102326


   retest this please


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

Klaus-xjp commented on a change in pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#discussion_r504369446



##########
File path: integration/spark/src/test/resources/IUD/updateinpartition.csv
##########
@@ -0,0 +1,21 @@
+id,sales,dtm
+001,0,20200907
+002,0,20200907
+003,0,20200907
+004,0,20200907
+005,0,20200907
+006,0,20200907
+007,0,20200907
+008,0,20200907
+009,0,20200907
+010,0,20200907
+011,0,20200908
+012,0,20200908
+013,0,20200908
+014,0,20200908
+015,0,20200908
+016,0,20200908
+017,0,20200908
+018,0,20200908
+019,0,20200908
+020,0,20200908

Review comment:
       new line at the end of file




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-708143674


   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4418/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-708143909


   Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2664/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

marchpure commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-708148791


   retest this please


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-708186433


   Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2668/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-708201912


   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4422/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

Kejian-Li commented on a change in pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#discussion_r504458957



##########
File path: integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/iud/UpdateCarbonTableTestCase.scala
##########
@@ -69,6 +69,60 @@ class UpdateCarbonTableTestCase extends QueryTest with BeforeAndAfterAll {
     sql("""drop table iud.zerorows""")
   }
 
+  test("update and insert overwrite partition") {
+    sql("""drop table if exists iud.updateinpartition""")
+    sql(
+      """CREATE TABLE iud.updateinpartition (id STRING, sales INT)
+        | PARTITIONED BY (dtm STRING)
+        | STORED AS carbondata""".stripMargin)
+    sql(
+      s"""load data local
+         | inpath '$resourcesPath/IUD/updateinpartition.csv' into table updateinpartition""".stripMargin)
+    sql(
+      """update iud.updateinpartition u set (u.sales) = (u.sales + 1) where id='001'""".stripMargin)
+    sql(
+      """update iud.updateinpartition u set (u.sales) = (u.sales + 2) where id='011'""".stripMargin)
+
+    // delete data from a partition, make sure the update executed before still works.
+    sql("""delete from updateinpartition where dtm=20200908 and id='012'""".stripMargin)
+    checkAnswer(
+      sql("""select sales from iud.updateinpartition where id='001'""".stripMargin), Seq(Row(1))
+    )
+    checkAnswer(
+      sql("""select sales from iud.updateinpartition where id='011'""".stripMargin), Seq(Row(2))
+    )
+    checkAnswer(
+      sql("""select sales from iud.updateinpartition where id='012'""".stripMargin), Seq()
+    )
+
+    // insert overwrite a partition. make sure the update executed before still works.
+    sql(
+      """insert overwrite table iud.updateinpartition
+        | partition (dtm=20200908)
+        | select * from iud.updateinpartition where dtm = 20200907""".stripMargin)
+    checkAnswer(
+      sql(
+        """select sales from iud.updateinpartition
+          | where dtm=20200908 and id='001'""".stripMargin), Seq(Row(1))
+    )
+    checkAnswer(
+      sql(
+        """select sales from iud.updateinpartition
+          | where dtm=20200908 and id='001'""".stripMargin), Seq(Row(1))
+    )

Review comment:
       Are they duplicated?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-708340588


   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4431/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-708349031


   Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2679/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]