[GitHub] carbondata pull request #2663: [CARBONDATA-2894] Add support for complex map...

[qiuchenjian-2]

GitHub user manishgupta88 opened a pull request:

    https://github.com/apache/carbondata/pull/2663

    [CARBONDATA-2894] Add support for complex map type through spark carbon file format API

    This PR supports loading querying complex map type through spark carbon file format API.
   
    **Note: This PR is dependent on PR #2649**
   
     - [ ] Any interfaces changed?
     No
     - [ ] Any backward compatibility impacted?
     No
     - [ ] Document update required?
    No
     - [ ] Testing done
    Added test cases      
     - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
    NA


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/manishgupta88/carbondata map_spark_carbon_file_support

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/2663.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2663
   
----
commit e67bd0cb485c4bed15ce8ac3ef3be9b3a4f3798e
Author: manishgupta88 <tomanishgupta18@...>
Date:   2018-08-20T04:59:29Z

    Added support for Avro Map type using SDK

commit 6db7f2a0d7c02406e0ecc9aa7ac69e2ec2e540a6
Author: manishgupta88 <tomanishgupta18@...>
Date:   2018-08-27T13:47:21Z

    Add support for complex map type using spark carbon file format API

----


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2663
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6426/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2663
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6427/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2663
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/8094/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2663
 
    Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/31/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2663
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6463/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2663
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/8150/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2663
 
    Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/79/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2663
 
    Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/101/



---

[qiuchenjian-2]

Github user manishgupta88 commented on the issue:

    https://github.com/apache/carbondata/pull/2663
 
    retest this please


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2663
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/8172/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2663
 
    Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/103/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2663
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/8174/



---

[qiuchenjian-2]

Github user manishgupta88 commented on the issue:

    https://github.com/apache/carbondata/pull/2663
 
    retest this please


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2663
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/8179/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2663
 
    Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/108/



---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2663#discussion_r214592707
 
    --- Diff: integration/spark-datasource/src/main/scala/org/apache/spark/sql/carbondata/execution/datasources/SparkCarbonFileFormat.scala ---
    @@ -214,6 +216,30 @@ class SparkCarbonFileFormat extends FileFormat
           data
         }
     
    +    private def extractMapData(data: AnyRef, mapType: MapType): ArrayObject = {
    +      val mapData = data.asInstanceOf[MapData]
    +      val keys: ArrayData = mapData.keyArray()
    +      val values: ArrayData = mapData.valueArray()
    +      var keyValueHolder = scala.collection.mutable.ArrayBuffer[AnyRef]()
    --- End diff --
   
    I think better you can directly use `extractData(row: ArrayData)` on key and value. And after that apply zip to create array of structobject


---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2663#discussion_r214593863
 
    --- Diff: integration/spark-datasource/src/main/scala/org/apache/spark/sql/carbondata/execution/datasources/CarbonSparkDataSourceUtil.scala ---
    @@ -198,7 +249,20 @@ object CarbonSparkDataSourceUtil {
           val dataType = convertSparkToCarbonDataType(field.dataType)
           dataType match {
             case s: CarbonStructType =>
    -          new Field(field.name, s, s.getFields)
    +          val subFields = prepareSubFields(field.name, s)
    --- End diff --
   
    Why do we need to create subfields here?


---

[qiuchenjian-2]

Github user manishgupta88 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2663#discussion_r214643373
 
    --- Diff: integration/spark-datasource/src/main/scala/org/apache/spark/sql/carbondata/execution/datasources/CarbonSparkDataSourceUtil.scala ---
    @@ -198,7 +249,20 @@ object CarbonSparkDataSourceUtil {
           val dataType = convertSparkToCarbonDataType(field.dataType)
           dataType match {
             case s: CarbonStructType =>
    -          new Field(field.name, s, s.getFields)
    +          val subFields = prepareSubFields(field.name, s)
    --- End diff --
   
    moved the preparation to Field class


---

[qiuchenjian-2]

Github user manishgupta88 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2663#discussion_r214643424
 
    --- Diff: integration/spark-datasource/src/main/scala/org/apache/spark/sql/carbondata/execution/datasources/SparkCarbonFileFormat.scala ---
    @@ -214,6 +216,30 @@ class SparkCarbonFileFormat extends FileFormat
           data
         }
     
    +    private def extractMapData(data: AnyRef, mapType: MapType): ArrayObject = {
    +      val mapData = data.asInstanceOf[MapData]
    +      val keys: ArrayData = mapData.keyArray()
    +      val values: ArrayData = mapData.valueArray()
    +      var keyValueHolder = scala.collection.mutable.ArrayBuffer[AnyRef]()
    --- End diff --
   
    ok done


---