[GitHub] incubator-carbondata pull request #630: [CARBONDATA-730]added decimal type i...

[qiuchenjian-2]

GitHub user anubhav100 opened a pull request:

    https://github.com/apache/incubator-carbondata/pull/630

    [CARBONDATA-730]added decimal type in carbon data frame writer

    Below exception is thrown while trying to save dataframe with a decimal column type.
   
    scala> df.printSchema
    – account: integer (nullable = true)
    – currency: integer (nullable = true)
    – branch: integer (nullable = true)
    – country: integer (nullable = true)
    – date: date (nullable = true)
    – fcbalance: decimal(16,3) (nullable = true)
    – lcbalance: decimal(16,3) (nullable = true)
   
    scala> df.write.format("carbondata").option("tableName", "accBal").option("compress", "true").mode(SaveMode.Overwrite).save()
   
    java.lang.RuntimeException: unsupported type: DecimalType(16,3)
    at scala.sys.package$.error(package.scala:27)
               


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/anubhav100/incubator-carbondata CARBONDATA-730

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-carbondata/pull/630.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #630
   
----
commit acd67332af60ad78dd93c1897ad31af4621c237e
Author: anubhav100 <[hidden email]>
Date:   2017-03-06T12:39:55Z

    added decimal type in carbon data frame writer

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/630
 
    Build Failed  with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1026/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/630
 
    Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1028/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user jackylk commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/630
 
    Can you add a testcase to write datafame containing a decimal column and query it? I think it needs more implementation for load and read


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/630
 
    Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1045/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user anubhav100 commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/630
 
    @jackylk i added a example in  CaseClassDataFrameAPIExample,i think its working correct
   
    case class Employee(name: String, salary: BigDecimal)
   
     val employee = List(Employee("anubhav", 20000))
        val employeeRDD = cc.sc.parallelize(employee)
        val employeeDataFrame = employeeRDD.toDF("name", "salary")
   
        employeeDataFrame.printSchema()
        employeeDataFrame.write
          .format("carbondata")
          .option("tableName", "carbon3")
          .option("compress", "true")
          .mode(SaveMode.Overwrite)
          .save()
   
     cc.sql("SELECT * FROM carbon2").show()
   
    here is result
   
    +-------+------+
    |   name|salary|
    +-------+------+
    |anubhav| 20000|
    +-------+------+


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/630
 
    Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1050/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user jackylk commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/630
 
    I think adding test case is better instead of example as it can be verified for every build
    And please verify declaring decimal with precision and scale parameter


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user anubhav100 commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/630
 
    @jackylk
   
    i tried this with decimal value
    val employee = List(Employee("anubhav", 20000.5))
    val employeeRDD = cc.sc.parallelize(employee)
    val employeeDataFrame = employeeRDD.toDF("name", "salary")
   
    employeeDataFrame.printSchema()
    employeeDataFrame.write
      .format("carbondata")
      .option("tableName", "carbon3")
      .option("compress", "true")
      .mode(SaveMode.Overwrite)
      .save()
   
   
   
    cc.sql("SELECT * FROM carbon3").show()
   
    |   name|salary|
    +-------+------+
    |anubhav| 20001|
    +-------+------+
   
    it seems like it is rounding off the values can you suggest something



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user sanoj-mg commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/630
 
    You can use the code snippet that I had pasted in the JIRA issue. As I mentioned it resolved my issue.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user chenliang613 commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/630
 
    please change the title as per the format: [CARBONDATA-issue number>] Title of the pull request (need to add a blank)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/630#discussion_r105579596
 
    --- Diff: examples/spark/src/main/scala/org/apache/carbondata/examples/CaseClassDataFrameAPIExample.scala ---
    @@ -42,8 +45,25 @@ object CaseClassDataFrameAPIExample {
           .mode(SaveMode.Overwrite)
           .save()
     
    +    val employee = List(Employee("anubhav", 20000))
    --- End diff --
   
    I tried with `    val employee = List(Employee("anubhav", BigDecimal(324234353, 4)))`
    the output is:
    ```
    +-------+------+
    |   name|salary|
    +-------+------+
    |anubhav| 32423|
    +-------+------+
    ```
    It is not handling correctly


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/630#discussion_r105579644
 
    --- Diff: examples/spark/src/main/scala/org/apache/carbondata/examples/CaseClassDataFrameAPIExample.scala ---
    @@ -42,8 +45,25 @@ object CaseClassDataFrameAPIExample {
           .mode(SaveMode.Overwrite)
           .save()
     
    +    val employee = List(Employee("anubhav", 20000))
    --- End diff --
   
    And can you add a testcase for both spark1 and spark2 integration?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user anubhav100 commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/630#discussion_r105582287
 
    --- Diff: examples/spark/src/main/scala/org/apache/carbondata/examples/CaseClassDataFrameAPIExample.scala ---
    @@ -42,8 +45,25 @@ object CaseClassDataFrameAPIExample {
           .mode(SaveMode.Overwrite)
           .save()
     
    +    val employee = List(Employee("anubhav", 20000))
    --- End diff --
   
    I change the code as suggested in code snippet,earlier I am not giving precision and scale factor,ya sure I am adding test cases


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user sanoj-mg commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/630#discussion_r105590219
 
    --- Diff: examples/spark/src/main/scala/org/apache/carbondata/examples/CaseClassDataFrameAPIExample.scala ---
    @@ -42,8 +45,25 @@ object CaseClassDataFrameAPIExample {
           .mode(SaveMode.Overwrite)
           .save()
     
    +    val employee = List(Employee("anubhav", 20000))
    --- End diff --
   
    @anubhav100 @jackylk  I am new to carbondata, but any reason for not adding the precision as below?
   
    case dt: DecimalType => s"${CarbonType.DECIMAL.getName}(${dt.precision}, ${dt.scale})


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user anubhav100 commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/630#discussion_r105590522
 
    --- Diff: examples/spark/src/main/scala/org/apache/carbondata/examples/CaseClassDataFrameAPIExample.scala ---
    @@ -42,8 +45,25 @@ object CaseClassDataFrameAPIExample {
           .mode(SaveMode.Overwrite)
           .save()
     
    +    val employee = List(Employee("anubhav", 20000))
    --- End diff --
   
    No I think it's alright I changed my code as u suggested and when I tested it ,it is working correctly, I am adding the test cases and will be pushing it


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user anubhav100 commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/630#discussion_r105830283
 
    --- Diff: examples/spark/src/main/scala/org/apache/carbondata/examples/CaseClassDataFrameAPIExample.scala ---
    @@ -42,8 +45,25 @@ object CaseClassDataFrameAPIExample {
           .mode(SaveMode.Overwrite)
           .save()
     
    +    val employee = List(Employee("anubhav", 20000))
    --- End diff --
   
    @jackylk i added test cases in integration package for both spark version can you review?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/630
 
    Build Failed  with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1118/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/630#discussion_r105966957
 
    --- Diff: integration/spark/src/test/scala/org/apache/carbondata/integration/spark/testsuite/dataframe/Decimal_DataFrameTest.scala ---
    @@ -0,0 +1,62 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.carbondata.integration.spark.testsuite.dataframe
    +
    +import java.math.BigDecimal
    +
    +import org.apache.spark.sql.common.util.DataFrameUtils
    +import org.apache.spark.sql.types.{DecimalType, DoubleType, StringType, StructField, StructType}
    +import org.apache.spark.sql.{Row, SaveMode}
    +import org.scalatest.FunSuiteLike
    +
    +class Decimal_DataFrameTest extends FunSuiteLike {
    --- End diff --
   
    Please add it in `TestLoadDataFrame` instead of creating a new class


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/630#discussion_r105967226
 
    --- Diff: integration/spark/src/test/scala/org/apache/carbondata/integration/spark/testsuite/dataframe/Decimal_DataFrameTest.scala ---
    @@ -0,0 +1,62 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.carbondata.integration.spark.testsuite.dataframe
    +
    +import java.math.BigDecimal
    +
    +import org.apache.spark.sql.common.util.DataFrameUtils
    +import org.apache.spark.sql.types.{DecimalType, DoubleType, StringType, StructField, StructType}
    +import org.apache.spark.sql.{Row, SaveMode}
    +import org.scalatest.FunSuiteLike
    +
    +class Decimal_DataFrameTest extends FunSuiteLike {
    +
    +  val carbonContext = DataFrameUtils.createCarbonContext("Decimal_DataFrameTest")
    +
    +  carbonContext.sql("DROP TABLE IF EXISTS carbon3")
    +
    +  val rdd = carbonContext.sc.parallelize(
    +    Row(52.23, BigDecimal.valueOf(1234.4440), "Warsaw") ::
    +    Row(42.30, BigDecimal.valueOf(9999.9990), "Corte") :: Nil)
    +
    +  val schema = StructType(
    +    StructField("double", DoubleType, nullable = false) ::
    +    StructField("decimal", DecimalType(9, 2), nullable = false) ::
    +    StructField("string", StringType, nullable = false) :: Nil)
    +
    +  val dataFrame = carbonContext.createDataFrame(rdd, schema)
    +
    +  dataFrame.printSchema()
    +
    +  dataFrame.write
    --- End diff --
   
    put the dataframe creation and write inside the test function


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---