[GitHub] carbondata pull request #2283: [CARBONDATA-2457] Add converter to get Carbon...

[qiuchenjian-2]

GitHub user ajantha-bhat opened a pull request:

    https://github.com/apache/carbondata/pull/2283

    [CARBONDATA-2457] Add converter to get Carbon SDK Schema from Avro schema directly.

    [CARBONDATA-2457] Add converter to get Carbon SDK Schema from Avro schema directly.
   
    In the current implementation, SDK users have to manually create carbon schema of fields from avro schema.
    This is time-consuming and error-prone. Also, user should not be worried about this logic.
    So, abstract the carbon schema creation from avro schema by exposing a method to user.
   
    Be sure to do all of the following checklist to help us incorporate
    your contribution quickly and easily:
   
     - [ ] Any interfaces changed? Added new interface, not modified existing
     
     - [ ] Any backward compatibility impacted? NA
     
     - [ ] Document update required? will be handled in separate PR
   
     - [ ] Testing done
           yes, updated the test case in TestNonTransactionalCarbonTable.scala
           
     - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. NA
   


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/ajantha-bhat/carbondata multi_level_complex

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/2283.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2283
   
----
commit a65a6fd0f6663061ddbc7da146e56e8367e75c9f
Author: ajantha-bhat <ajanthabhat@...>
Date:   2018-05-08T06:42:37Z

    [CARBONDATA-2457] Added converter to get Carbon SDK Schema from Avro schema directly.
   
    In the current implementation, SDK users have to manually create carbon
    schema of fields from avro schema.
    This is time consuming and error prone. Also user should not be worried
    about this logic.
    So, abstract the carbon schema creation from avro schema by exposing a
    method to user.

----


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2283
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4574/



---

[qiuchenjian-2]

Github user ajantha-bhat commented on the issue:

    https://github.com/apache/carbondata/pull/2283
 
    retest this please


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2283
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4589/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2283
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5748/



---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2283#discussion_r186935553
 
    --- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/file/AvroCarbonWriter.java ---
    @@ -118,11 +121,131 @@ private Object avroFieldToObject(Schema.Field avroField, Object fieldValue) {
             break;
     
           default:
    -        throw new UnsupportedOperationException();
    +        throw new UnsupportedOperationException(
    +            "carbon not support " + type.toString() + " avro type yet");
         }
         return out;
       }
     
    +  /**
    +   * converts avro schema to carbon schema required by carbonWriter
    +   *
    +   * @param avroSchemaString json formatted avro schema as string
    +   * @return carbon sdk schema
    +   */
    +  public static org.apache.carbondata.sdk.file.Schema getCarbonSchemaFromAvroSchema(
    +      String avroSchemaString) {
    +    if (avroSchemaString == null) {
    +      throw new UnsupportedOperationException("avro schema string cannot be null");
    +    }
    +    Schema avroSchema = new Schema.Parser().parse(avroSchemaString);
    +    Field[] carbonField = new Field[avroSchema.getFields().size()];
    +    int i = 0;
    +    for (Schema.Field avroField : avroSchema.getFields()) {
    +      carbonField[i] = prepareFields(avroField.name(), avroField.schema());
    +      i++;
    +    }
    +    return new org.apache.carbondata.sdk.file.Schema(carbonField);
    +  }
    +
    +  private static Field prepareFields(String FieldName, Schema childSchema) {
    --- End diff --
   
    Just take avro field, no need to take 2 params


---

[qiuchenjian-2]

Github user ajantha-bhat commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2283#discussion_r186939573
 
    --- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/file/AvroCarbonWriter.java ---
    @@ -118,11 +121,131 @@ private Object avroFieldToObject(Schema.Field avroField, Object fieldValue) {
             break;
     
           default:
    -        throw new UnsupportedOperationException();
    +        throw new UnsupportedOperationException(
    +            "carbon not support " + type.toString() + " avro type yet");
         }
         return out;
       }
     
    +  /**
    +   * converts avro schema to carbon schema required by carbonWriter
    +   *
    +   * @param avroSchemaString json formatted avro schema as string
    +   * @return carbon sdk schema
    +   */
    +  public static org.apache.carbondata.sdk.file.Schema getCarbonSchemaFromAvroSchema(
    +      String avroSchemaString) {
    +    if (avroSchemaString == null) {
    +      throw new UnsupportedOperationException("avro schema string cannot be null");
    +    }
    +    Schema avroSchema = new Schema.Parser().parse(avroSchemaString);
    +    Field[] carbonField = new Field[avroSchema.getFields().size()];
    +    int i = 0;
    +    for (Schema.Field avroField : avroSchema.getFields()) {
    +      carbonField[i] = prepareFields(avroField.name(), avroField.schema());
    +      i++;
    +    }
    +    return new org.apache.carbondata.sdk.file.Schema(carbonField);
    +  }
    +
    +  private static Field prepareFields(String FieldName, Schema childSchema) {
    --- End diff --
   
    OK. fixed


---

[qiuchenjian-2]

Github user ajantha-bhat commented on the issue:

    https://github.com/apache/carbondata/pull/2283
 
    retest this please


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2283
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5755/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2283
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4596/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2283
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4816/



---

[qiuchenjian-2]

Github user ajantha-bhat commented on the issue:

    https://github.com/apache/carbondata/pull/2283
 
    @ravipesala : PR is ready


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2283
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4603/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2283
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5762/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2283
 
    LGTM


---

[qiuchenjian-2]

Github user asfgit closed the pull request at:

    https://github.com/apache/carbondata/pull/2283


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2283
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4823/



---