update bug with carbondata1.1.0 and spark1.6.0

[yangwei]

Hi,
  My test case is as follows:
 
import  cc.implicits._
var kk:Seq[(String,String,String,Int)] = Nil
for(i<-1 to 80000){
   kk = kk ++Seq((i+"", "name","city", i))
}
cc.sc.makeRDD(kk).toDF("id","name","city","age").registerTempTable("t1")
cc.sql("DROP TABLE IF EXISTS testtable")
cc.sql("CREATE TABLE IF NOT EXISTS testtable(id string, name string, city string, age Int) STORED BY 'carbondata'")
cc.sql("insert INTO TABLE testtable select * from t1")
  cc.sql("""
         UPDATE testtable SET (testtable.name) = ('india') WHERE testtable.id = '10'
         """).show()
cc.sql("SELECT count(distinct id) FROM testtable").show()
|-------|
|---c0-|
|80000|

cc.sql("""
        UPDATE testtable SET (testtable.name) = ('india') WHERE testtable.id = '10'
        """).show()
 cc.sql("SELECT count(distinct id) FROM testtable").show()
|-------|
|---c0-|
|79998|
Only one record is updated, but the total number of records is missing by two

[Liang Chen-2]

Hi

Just i copied your code and tested it at my local machine , the result as
below. Because there are no any "india" value in name column, so don't do
any update.  Please check it again.
BTW, suggest you do test at branch-1.1,  because the community is testing
branch-1.1 for preparing 1.1.1 patch release, the "update and delete" is
key improvement in 1.1.1 release.

+-----+
|  _c0|
+-----+
|80000|


+-----+
|  _c0|
+-----+
|80000|
+-----+

Regards
Liang

2017-06-20 9:38 GMT+08:00 yangwei <[hidden email]>:

> Hi,
>   My test case is as follows:
>
> import  cc.implicits._
> var kk:Seq[(String,String,String,Int)] = Nil
> for(i<-1 to 80000){
>    kk = kk ++Seq((i+"", "name","city", i))
> }
> cc.sc.makeRDD(kk).toDF("id","name","city","age").registerTempTable("t1")
> cc.sql("DROP TABLE IF EXISTS testtable")
> cc.sql("CREATE TABLE IF NOT EXISTS testtable(id string, name string, city
> string, age Int) STORED BY 'carbondata'")
> cc.sql("insert INTO TABLE testtable select * from t1")
>   cc.sql("""
>          UPDATE testtable SET (testtable.name) = ('india') WHERE
> testtable.id = '10'
>          """).show()
> cc.sql("SELECT count(distinct id) FROM testtable").show()
> |-------|
> |---c0-|
> |80000|
>
> cc.sql("""
>         UPDATE testtable SET (testtable.name) = ('india') WHERE
> testtable.id
> = '10'
>         """).show()
>  cc.sql("SELECT count(distinct id) FROM testtable").show()
> |-------|
> |---c0-|
> |79998|
> Only one record is updated, but the total number of records is missing by
> two
>
>
>
> --
> View this message in context: http://apache-carbondata-dev-
> mailing-list-archive.1130556.n5.nabble.com/update-bug-with-
> carbondata1-1-0-and-spark1-6-0-tp15578.html
> Sent from the Apache CarbonData Dev Mailing List archive mailing list
> archive at Nabble.com.
>

[Erlu Chen]

Hi

I Just test in my idea in local mode, I am not sure which mode and which branch you test.

I have tested and the result seems ok.

My test steps as follows:

1. just copy your code into CarbonExample class.
2. Run CarbonExample.


Pasted code as follows:

    import  cc.implicits._
    var kk:Seq[(String,String,String,Int)] = Nil
    for(i<-1 to 80000){
      kk = kk ++Seq((i+"", "name","city", i))
    }
    cc.sc.makeRDD(kk).toDF("id","name","city","age").registerTempTable("t1")
    cc.sql("DROP TABLE IF EXISTS testtable")
    cc.sql("CREATE TABLE IF NOT EXISTS testtable(id string, name string, city string, age Int) STORED BY 'carbondata'")
    cc.sql("insert INTO TABLE testtable select * from t1")

    cc.sql("SELECT count(distinct id) FROM testtable").show()

    cc.sql("""
         UPDATE testtable SET (testtable.name) = ('india') WHERE testtable.id = '10'
         """).show()

    cc.sql("SELECT count(distinct id) FROM testtable").show()

    cc.sql("""
        UPDATE testtable SET (testtable.name) = ('india') WHERE testtable.id = '10'
        """).show()

    cc.sql("SELECT count(distinct id) FROM testtable").show()

  }



 Test result as follows:

17/06/20 10:34:12 AUDIT CarbonDataRDDFactory$: [bogon][erlu][Thread-1]Data load request has been received for table default.testtable
17/06/20 10:34:16 AUDIT CarbonDataRDDFactory$: [bogon][erlu][Thread-1]Data load is successful for default.testtable
+-----+
|  _c0|
+-----+
|80000|
+-----+

17/06/20 10:34:28 AUDIT deleteExecution$: [bogon][erlu][Thread-1]Delete data operation is successful for default.testtable
17/06/20 10:34:28 AUDIT CarbonDataRDDFactory$: [bogon][erlu][Thread-1]Data load request has been received for table default.testtable
17/06/20 10:34:29 AUDIT CarbonDataRDDFactory$: [bogon][erlu][Thread-1]Data update is successful for default.testtable
++
||
++
++

+-----+
|  _c0|
+-----+
|80000|
+-----+

17/06/20 10:34:34 AUDIT deleteExecution$: [bogon][erlu][Thread-1]Delete data operation is successful for default.testtable
17/06/20 10:34:34 AUDIT CarbonDataRDDFactory$: [bogon][erlu][Thread-1]Data load request has been received for table default.testtable
17/06/20 10:34:34 AUDIT CarbonDataRDDFactory$: [bogon][erlu][Thread-1]Data update is successful for default.testtable
++
||
++
++

+-----+
|  _c0|
+-----+
|80000|
+-----+


Process finished with exit code 0


Correct me if I missing something important.


Regards.
Chenerlu.

[yangwei]

Thank you for your valuable advice, and I will download and test branch-1.1

[Liang Chen]

Hi

Correct my info, can do update as below , it is successful.
+---+-----+----+---+
| id| name|city|age|
+---+-----+----+---+
| 10|india|city| 10|
+---+-----+----+---+

Regards
Liang