[GitHub] carbondata pull request #2485: [CARBONDATA-2722] [CARBONDATA-2721] JsonWrite...

classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata pull request #2485: [CARBONDATA-2722] [CARBONDATA-2721] JsonWrite...

qiuchenjian-2
GitHub user ajantha-bhat opened a pull request:

    https://github.com/apache/carbondata/pull/2485

    [CARBONDATA-2722] [CARBONDATA-2721] JsonWriter issue fixes

    [CARBONDATA-2722][SDK] [JsonWriter] NPE when schema and data are not of same length or Data is null.
   
    problem: Null data is not handled in the json object to carbon row conversion.
   
    solution: add a null check when object is fetched from json map.
   
    [CARBONDATA-2721][SDK] [JsonWriter] Json writer is writing only first element of an array and discarding the rest of the elements.
   
    problem: converting json object to carbon row array object is based on array children count , instead of array element count.
    Hence as array will always have one children. only one element is filled.
   
    solution: use array element count instead of array children count.Be sure to do all of the following checklist to help us incorporate
    your contribution quickly and easily:
   
     - [ ] Any interfaces changed?NA
     
     - [ ] Any backward compatibility impacted?NA
     
     - [ ] Document update required?NA
   
     - [ ] Testing done
    updated the UT      
     - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
   


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/ajantha-bhat/carbondata master

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/2485.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2485
   
----
commit 1152176762cba7c6b4686fbc2d7825c82e373e3c
Author: ajantha-bhat <ajanthabhat@...>
Date:   2018-07-09T13:15:30Z

    [CARBONDATA-2722][SDK] [JsonWriter] NPE when schema and data are not of same length or Data is null.
   
    problem: Null data is not handled in the json object to carbon row
    conversion.
   
    solution: add a null check when object is fetched from json map.
   
    [CARBONDATA-2721][SDK] [JsonWriter] Json writer is writing only first
    element of an array and discarding the rest of the elements.
   
    problem: converting json object to carbon row array object is based on
    array children count , instead of array element count.
    Hence as array will always have one children. only one element is
    filled.
   
    solution: use array element count instead of array children count.

----


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2485: [CARBONDATA-2722] [CARBONDATA-2721] JsonWriter issue...

qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2485
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5791/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2485: [CARBONDATA-2722] [CARBONDATA-2721] JsonWriter issue...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2485
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7015/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2485: [CARBONDATA-2722] [CARBONDATA-2721] JsonWriter issue...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2485
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5767/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2485: [CARBONDATA-2722] [CARBONDATA-2721] JsonWriter issue...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user ajantha-bhat commented on the issue:

    https://github.com/apache/carbondata/pull/2485
 
    retest this please


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2485: [CARBONDATA-2722] [CARBONDATA-2721] JsonWriter issue...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2485
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7051/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2485: [CARBONDATA-2722] [CARBONDATA-2721] JsonWriter issue...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2485
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5828/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2485: [CARBONDATA-2722] [CARBONDATA-2721] JsonWriter issue...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2485
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7066/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2485: [CARBONDATA-2722] [CARBONDATA-2721] JsonWriter issue...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2485
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5843/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2485: [CARBONDATA-2722] [CARBONDATA-2721] JsonWriter issue...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2485
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5809/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata pull request #2485: [CARBONDATA-2722] [CARBONDATA-2721] JsonWrite...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2485#discussion_r202256075
 
    --- Diff: processing/src/main/java/org/apache/carbondata/processing/loading/parser/impl/JsonRowParser.java ---
    @@ -56,6 +56,9 @@ public JsonRowParser(DataField[] dataFields) {
           Map<String, Object> jsonNodeMap =
               objectMapper.readValue(jsonString, new TypeReference<Map<String, Object>>() {
               });
    +      if (jsonNodeMap == null) {
    --- End diff --
   
    Here `|| jsonNodeMap.size() == 0` check is not required?


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2485: [CARBONDATA-2722] [CARBONDATA-2721] JsonWriter issue...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2485
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7087/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata pull request #2485: [CARBONDATA-2722] [CARBONDATA-2721] JsonWrite...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user ajantha-bhat commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2485#discussion_r202277326
 
    --- Diff: processing/src/main/java/org/apache/carbondata/processing/loading/parser/impl/JsonRowParser.java ---
    @@ -56,6 +56,9 @@ public JsonRowParser(DataField[] dataFields) {
           Map<String, Object> jsonNodeMap =
               objectMapper.readValue(jsonString, new TypeReference<Map<String, Object>>() {
               });
    +      if (jsonNodeMap == null) {
    --- End diff --
   
    yeah, not required. removed it.


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2485: [CARBONDATA-2722] [CARBONDATA-2721] JsonWriter issue...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2485
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5864/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2485: [CARBONDATA-2722] [CARBONDATA-2721] JsonWriter issue...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2485
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5823/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2485: [CARBONDATA-2722] [CARBONDATA-2721] JsonWriter issue...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2485
 
    LGTM


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata pull request #2485: [CARBONDATA-2722] [CARBONDATA-2721] JsonWrite...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user asfgit closed the pull request at:

    https://github.com/apache/carbondata/pull/2485


---