[GitHub] carbondata pull request #1337: Fix update fail when carbon.update.persist.en...

[qiuchenjian-2]

GitHub user ravipesala opened a pull request:

    https://github.com/apache/carbondata/pull/1337

    Fix update fail when carbon.update.persist.enable'='false'

    The UDF for getting segementid while loading the data is not handled so when it needs to reexecute the rdd when persist enable is false it is not getting tupleId from carbon

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/ravipesala/incubator-carbondata update-fail

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/1337.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1337
   
----
commit df368b963d38b25a49d12dbbc62b660acae31154
Author: Ravindra Pesala <[hidden email]>
Date:   2017-09-06T15:12:34Z

    Fix update fail when carbon.update.persist.enable'='false'

----


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1337
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/580/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1337
 
    retest this please


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1337
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/582/



---

[qiuchenjian-2]

Github user jackylk commented on the issue:

    https://github.com/apache/carbondata/pull/1337
 
    LGTM


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1337#discussion_r137448549
 
    --- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/iud/UpdateCarbonTableTestCase.scala ---
    @@ -448,6 +448,40 @@ class UpdateCarbonTableTestCase extends QueryTest with BeforeAndAfterAll {
         sql("DROP TABLE IF EXISTS default.carbon1")
       }
     
    +  test("""CARBONDATA-1445 carbon.update.persist.enable=false it will fail to update data""") {
    +    CarbonProperties.getInstance()
    +      .addProperty(CarbonCommonConstants.isPersistEnabled, "false")
    +    import sqlContext.implicits._
    +    val df = sqlContext.sparkContext.parallelize(0 to 50)
    +      .map(x => ("a", x.toString, (x % 2).toString, x, x.toLong, x * 2))
    +      .toDF("stringField1", "stringField2", "stringField3", "intField", "longField", "int2Field")
    +    sql("DROP TABLE IF EXISTS default.study_carbondata ")
    +    sql(s""" CREATE TABLE IF NOT EXISTS default.study_carbondata (
    +           |    stringField1          string,
    +           |    stringField2          string,
    +           |    stringField3          string,
    +           |    intField              int,
    +           |    longField             bigint,
    +           |    int2Field             int) STORED BY 'carbondata'""".stripMargin)
    +    df.write
    +      .format("carbondata")
    +      .option("tableName", "study_carbondata")
    +      .option("compress", "true")  // just valid when tempCSV is true
    +      .option("tempCSV", "false")
    +      .option("single_pass", "true")
    +      .option("sort_scope", "LOCAL_SORT")
    +      .mode(SaveMode.Append)
    +      .save()
    +    sql("""
    +      UPDATE default.study_carbondata a
    --- End diff --
   
    Please add assert to check the updated value


---

[qiuchenjian-2]

Github user zzcclp commented on the issue:

    https://github.com/apache/carbondata/pull/1337
 
    LGTM, it works correctly now.


---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1337#discussion_r137800762
 
    --- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/iud/UpdateCarbonTableTestCase.scala ---
    @@ -448,6 +448,40 @@ class UpdateCarbonTableTestCase extends QueryTest with BeforeAndAfterAll {
         sql("DROP TABLE IF EXISTS default.carbon1")
       }
     
    +  test("""CARBONDATA-1445 carbon.update.persist.enable=false it will fail to update data""") {
    +    CarbonProperties.getInstance()
    +      .addProperty(CarbonCommonConstants.isPersistEnabled, "false")
    +    import sqlContext.implicits._
    +    val df = sqlContext.sparkContext.parallelize(0 to 50)
    +      .map(x => ("a", x.toString, (x % 2).toString, x, x.toLong, x * 2))
    +      .toDF("stringField1", "stringField2", "stringField3", "intField", "longField", "int2Field")
    +    sql("DROP TABLE IF EXISTS default.study_carbondata ")
    +    sql(s""" CREATE TABLE IF NOT EXISTS default.study_carbondata (
    +           |    stringField1          string,
    +           |    stringField2          string,
    +           |    stringField3          string,
    +           |    intField              int,
    +           |    longField             bigint,
    +           |    int2Field             int) STORED BY 'carbondata'""".stripMargin)
    +    df.write
    +      .format("carbondata")
    +      .option("tableName", "study_carbondata")
    +      .option("compress", "true")  // just valid when tempCSV is true
    +      .option("tempCSV", "false")
    +      .option("single_pass", "true")
    +      .option("sort_scope", "LOCAL_SORT")
    +      .mode(SaveMode.Append)
    +      .save()
    +    sql("""
    +      UPDATE default.study_carbondata a
    --- End diff --
   
    Added


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1337
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/616/



---

[qiuchenjian-2]

Github user zzcclp commented on the issue:

    https://github.com/apache/carbondata/pull/1337
 
    retest this please


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1337
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/619/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1337
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/621/



---

[qiuchenjian-2]

Github user zzcclp commented on the issue:

    https://github.com/apache/carbondata/pull/1337
 
    retest this please


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1337
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/626/



---

[qiuchenjian-2]

Github user zzcclp commented on the issue:

    https://github.com/apache/carbondata/pull/1337
 
    retest this please


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1337
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/642/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1337
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/643/



---

[qiuchenjian-2]

Github user zzcclp commented on the issue:

    https://github.com/apache/carbondata/pull/1337
 
    retest this please


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1337
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/649/



---

[qiuchenjian-2]

Github user zzcclp commented on the issue:

    https://github.com/apache/carbondata/pull/1337
 
    retest this please


---