[GitHub] carbondata pull request #1774: [CARBONDATA-2001] Unable to Save DataFrame As...

[qiuchenjian-2]

GitHub user anubhav100 opened a pull request:

    https://github.com/apache/carbondata/pull/1774

    [CARBONDATA-2001] Unable to Save DataFrame As Carbondata stream table

    **jira link**:https://issues.apache.org/jira/browse/CARBONDATA-2001
    1.added table property for streaming in carbondataframewriter
    2.added test case for same
           
   


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/anubhav100/incubator-carbondata CARBONDATA-2001

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/1774.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1774
   
----
commit 0c84c963a2463104ebde315743a34d95431fab7c
Author: anubhav100 <anubhav.tarar@...>
Date:   2018-01-08T07:47:36Z

    unable to save data as carbondata stream table

----


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1774
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2790/



---

[qiuchenjian-2]

Github user anubhav100 commented on the issue:

    https://github.com/apache/carbondata/pull/1774
 
    retest this please


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1774
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1396/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1774
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2792/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1774
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2631/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1774
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2793/



---

[qiuchenjian-2]

Github user anubhav100 commented on the issue:

    https://github.com/apache/carbondata/pull/1774
 
    retest this please


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1774
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2634/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1774
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1406/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1774
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1412/



---

[qiuchenjian-2]

Github user anubhav100 commented on the issue:

    https://github.com/apache/carbondata/pull/1774
 
    retest this please


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1774
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2667/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1774
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1433/



---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1774#discussion_r160845207
 
    --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/CarbonDataFrameWriter.scala ---
    @@ -167,13 +167,19 @@ class CarbonDataFrameWriter(sqlContext: SQLContext, val dataFrame: DataFrame) {
         val carbonSchema = schema.map { field =>
           s"${ field.name } ${ convertToCarbonType(field.dataType) }"
         }
    +  val isStreaming = if (options.isStreaming) Some("true") else None
    +
         val property = Map(
           "SORT_COLUMNS" -> options.sortColumns,
           "DICTIONARY_INCLUDE" -> options.dictionaryInclude,
           "DICTIONARY_EXCLUDE" -> options.dictionaryExclude,
    -      "TABLE_BLOCKSIZE" -> options.tableBlockSize
    -    ).filter(_._2.isDefined).map(p => s"'${p._1}' = '${p._2.get}'").mkString(",")
    +      "TABLE_BLOCKSIZE" -> options.tableBlockSize,
    +      "STREAMING" -> isStreaming
    +    ).filter(_._2.isDefined).
    --- End diff --
   
    move `.` to next line


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1774#discussion_r160845350
 
    --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/CarbonDataFrameWriter.scala ---
    @@ -167,13 +167,19 @@ class CarbonDataFrameWriter(sqlContext: SQLContext, val dataFrame: DataFrame) {
         val carbonSchema = schema.map { field =>
           s"${ field.name } ${ convertToCarbonType(field.dataType) }"
         }
    +  val isStreaming = if (options.isStreaming) Some("true") else None
    +
         val property = Map(
           "SORT_COLUMNS" -> options.sortColumns,
           "DICTIONARY_INCLUDE" -> options.dictionaryInclude,
           "DICTIONARY_EXCLUDE" -> options.dictionaryExclude,
    -      "TABLE_BLOCKSIZE" -> options.tableBlockSize
    -    ).filter(_._2.isDefined).map(p => s"'${p._1}' = '${p._2.get}'").mkString(",")
    +      "TABLE_BLOCKSIZE" -> options.tableBlockSize,
    +      "STREAMING" -> isStreaming
    --- End diff --
   
    why not use `options.isStreaming` directly?


---

[qiuchenjian-2]

Github user anubhav100 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1774#discussion_r160864606
 
    --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/CarbonDataFrameWriter.scala ---
    @@ -167,13 +167,19 @@ class CarbonDataFrameWriter(sqlContext: SQLContext, val dataFrame: DataFrame) {
         val carbonSchema = schema.map { field =>
           s"${ field.name } ${ convertToCarbonType(field.dataType) }"
         }
    +  val isStreaming = if (options.isStreaming) Some("true") else None
    +
         val property = Map(
           "SORT_COLUMNS" -> options.sortColumns,
           "DICTIONARY_INCLUDE" -> options.dictionaryInclude,
           "DICTIONARY_EXCLUDE" -> options.dictionaryExclude,
    -      "TABLE_BLOCKSIZE" -> options.tableBlockSize
    -    ).filter(_._2.isDefined).map(p => s"'${p._1}' = '${p._2.get}'").mkString(",")
    +      "TABLE_BLOCKSIZE" -> options.tableBlockSize,
    +      "STREAMING" -> isStreaming
    +    ).filter(_._2.isDefined).
    --- End diff --
   
    done


---

[qiuchenjian-2]

Github user anubhav100 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1774#discussion_r160864615
 
    --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/CarbonDataFrameWriter.scala ---
    @@ -167,13 +167,19 @@ class CarbonDataFrameWriter(sqlContext: SQLContext, val dataFrame: DataFrame) {
         val carbonSchema = schema.map { field =>
           s"${ field.name } ${ convertToCarbonType(field.dataType) }"
         }
    +  val isStreaming = if (options.isStreaming) Some("true") else None
    +
         val property = Map(
           "SORT_COLUMNS" -> options.sortColumns,
           "DICTIONARY_INCLUDE" -> options.dictionaryInclude,
           "DICTIONARY_EXCLUDE" -> options.dictionaryExclude,
    -      "TABLE_BLOCKSIZE" -> options.tableBlockSize
    -    ).filter(_._2.isDefined).map(p => s"'${p._1}' = '${p._2.get}'").mkString(",")
    +      "TABLE_BLOCKSIZE" -> options.tableBlockSize,
    +      "STREAMING" -> isStreaming
    --- End diff --
   
    @jackylk reason that i not used options.isStreaming is because it is giving back a boolean value so i just converted the boolean to option


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1774
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2687/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1774
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1454/



---