[GitHub] carbondata pull request #2347: [WIP] Added support for logical type

[qiuchenjian-2]

GitHub user kunal642 opened a pull request:

    https://github.com/apache/carbondata/pull/2347

    [WIP] Added support for logical type

    Be sure to do all of the following checklist to help us incorporate
    your contribution quickly and easily:
   
     - [ ] Any interfaces changed?
     
     - [ ] Any backward compatibility impacted?
     
     - [ ] Document update required?
   
     - [ ] Testing done
            Please provide details on
            - Whether new unit test cases have been added or why no new tests are required?
            - How it is tested? Please attach test report.
            - Is it a performance related change? Please attach the performance test report.
            - Any additional information to help reviewers in testing this change.
           
     - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
   


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/kunal642/carbondata avro_logical_type_support

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/2347.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2347
   
----
commit 135719aa1e3f69497d9b12a50b27d2cd3787ffce
Author: kunal642 <kunalkapoor642@...>
Date:   2018-05-28T06:11:59Z

    added support for logical type

----


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2347
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6121/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2347
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4959/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2347
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5112/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2347
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4967/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2347
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6129/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2347
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5117/



---

[qiuchenjian-2]

Github user kunal642 commented on the issue:

    https://github.com/apache/carbondata/pull/2347
 
    retest this please


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2347
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6141/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2347
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4978/



---

[qiuchenjian-2]

Github user KanakaKumar commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2347#discussion_r191310013
 
    --- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/file/AvroCarbonWriter.java ---
    @@ -88,9 +90,18 @@
       private Object avroFieldToObject(Schema.Field avroField, Object fieldValue) {
         Object out;
         Schema.Type type = avroField.schema().getType();
    +    LogicalType logicalType = avroField.schema().getLogicalType();
         switch (type) {
    -      case BOOLEAN:
           case INT:
    +        if (logicalType != null) {
    --- End diff --
   
    Consider to convert as date only if the logical type is Date


---

[qiuchenjian-2]

Github user KanakaKumar commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2347#discussion_r191310300
 
    --- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/file/AvroCarbonWriter.java ---
    @@ -177,13 +188,22 @@ private static Field prepareFields(Schema.Field avroField) {
         String FieldName = avroField.name();
         Schema childSchema = avroField.schema();
         Schema.Type type = childSchema.getType();
    +    LogicalType logicalType = childSchema.getLogicalType();
         switch (type) {
           case BOOLEAN:
             return new Field(FieldName, DataTypes.BOOLEAN);
           case INT:
    -        return new Field(FieldName, DataTypes.INT);
    +        if (logicalType == null) {
    --- End diff --
   
    Need to check the logical type is Date or something else.


---

[qiuchenjian-2]

Github user kunal642 commented on the issue:

    https://github.com/apache/carbondata/pull/2347
 
    @ravipesala Please review


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2347
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6150/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2347
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4988/



---

[qiuchenjian-2]

Github user kunal642 commented on the issue:

    https://github.com/apache/carbondata/pull/2347
 
    retest this please


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2347
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5133/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2347
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4992/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2347
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6154/



---

[qiuchenjian-2]

Github user sounakr commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2347#discussion_r191386255
 
    --- Diff: processing/src/main/java/org/apache/carbondata/processing/datatypes/PrimitiveDataType.java ---
    @@ -288,7 +288,12 @@ public int getSurrogateIndex() {
               logHolder.setReason(message);
             }
           } else {
    -        surrogateKey = dictionaryGenerator.getOrGenerateKey(parsedValue);
    +        if (dictionaryGenerator instanceof DirectDictionary
    --- End diff --
   
      @Override public void writeByteArray(Object input, DataOutputStream dataOutputStream,
          BadRecordLogHolder logHolder) throws IOException, DictionaryGenerationException {
        String parsedValue =
            input == null ? null : DataTypeUtil.parseValue(input.toString(), carbonDimension);
   
    If the input is long, is it needed to parse as toString?


---