[GitHub] carbondata pull request #1839: [CARBONDATA-2016] Exception displays while ex...

[qiuchenjian-2]

GitHub user anubhav100 opened a pull request:

    https://github.com/apache/carbondata/pull/1839

    [CARBONDATA-2016] Exception displays while executing compaction with alter query

    **Root Cause**
    When we apply the alter table command to add column with default value it is storing it as long object,it is wrongly written in restructure util we should get the value as the same type
    as that of datatype of columnschema in restructure util earlier in master branch in restuctureutil class if our data type is long or short or int we are always returning back a long object which is wong due to same reason compaction was failing
    if it was applied after alter table add columns command with default value because in sortdatarows there was mismatch between data type and its corresponding value
   
    **Testing:**
    1.mvn clean install is passing
    2.added new test case for same

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/anubhav100/incubator-carbondata bugfix/CARBONDATA-2016

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/1839.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1839
   
----
commit eee0a2325b410c9d1f9e18b40002446b067ed581
Author: anubhav100 <anubhav.tarar@...>
Date:   2018-01-19T16:01:14Z

    When we apply the alter table command to add column with default value it is storing it as long object,it is wrongly written in restructure util we should get the value as the same type

----


---

[qiuchenjian-2]

Github user anubhav100 commented on the issue:

    https://github.com/apache/carbondata/pull/1839
 
    @jackylk  i accidently deleted my old branch i have created this new branch for same issue please merge this one


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1839
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3000/



---

[qiuchenjian-2]

Github user anubhav100 commented on the issue:

    https://github.com/apache/carbondata/pull/1839
 
    retest this please


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1839
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1770/



---

[qiuchenjian-2]

Github user anubhav100 commented on the issue:

    https://github.com/apache/carbondata/pull/1839
 
    retest this please


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1839
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1774/



---

[qiuchenjian-2]

Github user anubhav100 commented on the issue:

    https://github.com/apache/carbondata/pull/1839
 
    retest this please


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1839
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1775/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1839
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3005/



---

[qiuchenjian-2]

Github user anubhav100 commented on the issue:

    https://github.com/apache/carbondata/pull/1839
 
    @jackylk can merge this one


---

[qiuchenjian-2]

Github user anubhav100 commented on the issue:

    https://github.com/apache/carbondata/pull/1839
 
    @jackylk please review


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1839
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3121/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1839
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1889/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1839
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3100/



---

[qiuchenjian-2]

Github user manishgupta88 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1839#discussion_r164339708
 
    --- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/iud/HorizontalCompactionTestCase.scala ---
    @@ -360,6 +393,8 @@ class HorizontalCompactionTestCase extends QueryTest with BeforeAndAfterAll {
         CarbonProperties.getInstance()
           .addProperty(CarbonCommonConstants.isHorizontalCompactionEnabled , "true")
         sql("""drop table if exists t_carbn01""")
    +    sql("""drop table if exists customer1""")
    +
    --- End diff --
   
    Remove extra line


---

[qiuchenjian-2]

Github user manishgupta88 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1839#discussion_r164339675
 
    --- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/iud/HorizontalCompactionTestCase.scala ---
    @@ -350,8 +350,41 @@ class HorizontalCompactionTestCase extends QueryTest with BeforeAndAfterAll {
         )
         sql("""drop table dest2""")
       }
    +  test("test the compaction after alter command") // As per bug Carbondata-2016
    +  {
    +    try {
    +      sql(
    +        "CREATE TABLE CUSTOMER1 ( C_CUSTKEY INT , C_NAME STRING , C_ADDRESS STRING , C_NATIONKEY INT , C_PHONE STRING , C_ACCTBAL DECIMAL(15,2) , C_MKTSEGMENT STRING , C_COMMENT STRING) stored by 'carbondata'")
     
    +      sql(
    +        "insert into customer1 values(1,'vandana','noida',1,'123456789',45987.78,'hello','comment')")
     
    +      sql(
    +        "insert into customer1 values(2,'vandana','noida',2,'123456789',487.78,'hello','comment')")
    +
    +      sql(
    +        " insert into customer1 values(3,'geetika','delhi',3,'123456789',487897.78,'hello','comment')")
    +
    +      sql(
    +        "insert into customer1 values(4,'sangeeta','delhi',3,'123456789',48789.78,'hello','comment')")
    +
    +      sql(
    +        "alter table customer1 add columns (shortfield short) TBLPROPERTIES ('DEFAULT.VALUE.shortfield'='10')")
    +
    +      sql(
    +        "alter table customer1 add columns (intfield int) TBLPROPERTIES ('DEFAULT.VALUE.intfield'='10')")
    +
    +      sql(
    +        "alter table customer1 add columns (longfield bigint) TBLPROPERTIES ('DEFAULT.VALUE.longfield'='10')")
    +
    +      sql("alter table customer1 compact 'minor' ").show()
    +      assert(true)
    --- End diff --
   
    1. Remove try catch block..its not required..in case of any exception anyways the test case will fail.
    2. In the add column DDL's. give the default value as MAX value of the corresponding datatype so that the same is validated after compaction in the select query validation.
    3. After compaction match the newly added column values as select query using checkAnswer.


---

[qiuchenjian-2]

Github user anubhav100 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1839#discussion_r164344032
 
    --- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/iud/HorizontalCompactionTestCase.scala ---
    @@ -350,8 +350,41 @@ class HorizontalCompactionTestCase extends QueryTest with BeforeAndAfterAll {
         )
         sql("""drop table dest2""")
       }
    +  test("test the compaction after alter command") // As per bug Carbondata-2016
    +  {
    +    try {
    +      sql(
    +        "CREATE TABLE CUSTOMER1 ( C_CUSTKEY INT , C_NAME STRING , C_ADDRESS STRING , C_NATIONKEY INT , C_PHONE STRING , C_ACCTBAL DECIMAL(15,2) , C_MKTSEGMENT STRING , C_COMMENT STRING) stored by 'carbondata'")
     
    +      sql(
    +        "insert into customer1 values(1,'vandana','noida',1,'123456789',45987.78,'hello','comment')")
     
    +      sql(
    +        "insert into customer1 values(2,'vandana','noida',2,'123456789',487.78,'hello','comment')")
    +
    +      sql(
    +        " insert into customer1 values(3,'geetika','delhi',3,'123456789',487897.78,'hello','comment')")
    +
    +      sql(
    +        "insert into customer1 values(4,'sangeeta','delhi',3,'123456789',48789.78,'hello','comment')")
    +
    +      sql(
    +        "alter table customer1 add columns (shortfield short) TBLPROPERTIES ('DEFAULT.VALUE.shortfield'='10')")
    +
    +      sql(
    +        "alter table customer1 add columns (intfield int) TBLPROPERTIES ('DEFAULT.VALUE.intfield'='10')")
    +
    +      sql(
    +        "alter table customer1 add columns (longfield bigint) TBLPROPERTIES ('DEFAULT.VALUE.longfield'='10')")
    +
    +      sql("alter table customer1 compact 'minor' ").show()
    +      assert(true)
    --- End diff --
   
    @manishgupta88 done


---

[qiuchenjian-2]

Github user anubhav100 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1839#discussion_r164344094
 
    --- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/iud/HorizontalCompactionTestCase.scala ---
    @@ -360,6 +393,8 @@ class HorizontalCompactionTestCase extends QueryTest with BeforeAndAfterAll {
         CarbonProperties.getInstance()
           .addProperty(CarbonCommonConstants.isHorizontalCompactionEnabled , "true")
         sql("""drop table if exists t_carbn01""")
    +    sql("""drop table if exists customer1""")
    +
    --- End diff --
   
    @manishgupta88 done you can review


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1839
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1979/



---