[GitHub] carbondata pull request #2837: [CARBONDATA-3000] Provide C++ interface for w...

[qiuchenjian-2]

GitHub user xubo245 opened a pull request:

    https://github.com/apache/carbondata/pull/2837

    [CARBONDATA-3000] Provide C++ interface for writing carbon data in CSDK

    [CARBONDATA-3000] Provide C++ interface for writing carbon data in CSDK
        1.suport string, short, int, long, double, float, array<string>, boolean data type
        2.provide builder, build, write, close interface
   
    This PR base on https://github.com/apache/carbondata/pull/2792 and https://github.com/apache/carbondata/pull/2816, and cherry its commits to this PR. this PR will remove its commits after it's merged
   
    Be sure to do all of the following checklist to help us incorporate
    your contribution quickly and easily:
   
     - [ ] Any interfaces changed?
     add C++interface
     - [ ] Any backward compatibility impacted?
     No
     - [ ] Document update required?
    Yes
     - [ ] Testing done
           add some test case
     - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
    https://issues.apache.org/jira/browse/CARBONDATA-2951


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/xubo245/carbondata CARBONDATA-3000_writeData

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/2837.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2837
   
----
commit d37f2c3962918d559ef29349e1adb2585581a8c7
Author: xubo245 <xubo29@...>
Date:   2018-10-09T09:58:48Z

     [CARBONDATA-2981] Support read primitive data type in CSDK
   
                1.support readNextCarbonRow
                2.support read different primitive data type in c code from java side: int double short long string
                3.support some data type and convert: date timestamp varchar decimal array<T>
                4.remove readNextStringRow
   
    remove the file after finished run
   
    change the file
   
    remove timestamp check

commit 673d8aab13546782e71cf8a00f6039d9d687931e
Author: xubo245 <xubo29@...>
Date:   2018-10-16T03:02:07Z

    [CARBONDATA-300] Suppor read batch row in CSDK
    1. support read batch row in SDK
    2. support read batch row in CSDK
    3. add SDKReaderBenchmark IN SDK and testNextBatchRowPerformance in CSDK
    4. improve CSDK read performance
   
    support S3 in main
   
    fix error
   
    add test case
   
    optimize
   
    release object
   
    optimize

commit 5a6def5f4d698ff5c138f23773c16dbcd6585f78
Author: xubo245 <xubo29@...>
Date:   2018-10-19T07:40:04Z

    [CARBONDATA-3000] Provide C++ interface for writing carbon data in CSDK
    1.suport string, short, int, long, double, float, array<string>, boolean data type
    2.provide builder, build, write, close interface
    3.TODO: support S3

----


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2837
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/879/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2837
 
    Build Failed  with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9144/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2837
 
    Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1077/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2837
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/881/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2837
 
    Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9146/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2837
 
    Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1079/



---

[qiuchenjian-2]

Github user ajantha-bhat commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2837#discussion_r228418984
 
    --- Diff: store/CSDK/CarbonWriter.cpp ---
    @@ -0,0 +1,83 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +#include "CarbonWriter.h"
    +
    +jobject CarbonWriter::builder(JNIEnv *env) {
    +    jniEnv = env;
    --- End diff --
   
    Add validation for null and get the exception from java using jni.
    Do this for all the newly added API.


---

[qiuchenjian-2]

Github user ajantha-bhat commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2837#discussion_r228456762
 
    --- Diff: store/CSDK/CarbonWriter.cpp ---
    @@ -0,0 +1,83 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +#include "CarbonWriter.h"
    +
    +jobject CarbonWriter::builder(JNIEnv *env) {
    +    jniEnv = env;
    +    carbonWriter = env->FindClass("org/apache/carbondata/sdk/file/CarbonWriter");
    +    jmethodID carbonWriterBuilderID = env->GetStaticMethodID(carbonWriter, "builder",
    +        "()Lorg/apache/carbondata/sdk/file/CarbonWriterBuilder;");
    +    carbonWriterBuilderObject = env->CallStaticObjectMethod(carbonWriter, carbonWriterBuilderID);
    +    return carbonWriterBuilderObject;
    +}
    +
    +jobject CarbonWriter::outputPath(char *path) {
    --- End diff --
   
    return type of builder, outputPath, withCsvInput, withHadoopConf can be void


---

[qiuchenjian-2]

Github user ajantha-bhat commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2837#discussion_r228459702
 
    --- Diff: store/CSDK/CarbonWriter.cpp ---
    @@ -0,0 +1,83 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +#include "CarbonWriter.h"
    +
    +jobject CarbonWriter::builder(JNIEnv *env) {
    +    jniEnv = env;
    +    carbonWriter = env->FindClass("org/apache/carbondata/sdk/file/CarbonWriter");
    +    jmethodID carbonWriterBuilderID = env->GetStaticMethodID(carbonWriter, "builder",
    +        "()Lorg/apache/carbondata/sdk/file/CarbonWriterBuilder;");
    +    carbonWriterBuilderObject = env->CallStaticObjectMethod(carbonWriter, carbonWriterBuilderID);
    +    return carbonWriterBuilderObject;
    +}
    +
    +jobject CarbonWriter::outputPath(char *path) {
    +    jclass carbonWriterBuilderClass = jniEnv->GetObjectClass(carbonWriterBuilderObject);
    +    jmethodID carbonWriterBuilderID = jniEnv->GetMethodID(carbonWriterBuilderClass, "outputPath",
    +        "(Ljava/lang/String;)Lorg/apache/carbondata/sdk/file/CarbonWriterBuilder;");
    +    jstring jPath = jniEnv->NewStringUTF(path);
    +    jvalue args[1];
    +    args[0].l = jPath;
    +    carbonWriterBuilderObject = jniEnv->CallObjectMethodA(carbonWriterBuilderObject, carbonWriterBuilderID, args);
    +    return carbonWriterBuilderObject;
    +}
    +
    +jobject CarbonWriter::withCsvInput(char *jsonSchema) {
    +
    +    jclass carbonWriterBuilderClass = jniEnv->GetObjectClass(carbonWriterBuilderObject);
    +    jmethodID carbonWriterBuilderID = jniEnv->GetMethodID(carbonWriterBuilderClass, "withCsvInput",
    --- End diff --
   
    just rename these variables as methodID


---

[qiuchenjian-2]

Github user ajantha-bhat commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2837#discussion_r228463349
 
    --- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonWriterBuilder.java ---
    @@ -329,6 +344,19 @@ public CarbonWriterBuilder withCsvInput(Schema schema) {
         return this;
       }
     
    +  /**
    +   * to build a {@link CarbonWriter}, which accepts row in CSV format
    +   *
    +   * @param jsonSchema json Schema string
    +   * @return CarbonWriterBuilder
    +   */
    +  public CarbonWriterBuilder withCsvInput(String jsonSchema) {
    +    Objects.requireNonNull(jsonSchema, "schema should not be null");
    +    this.schema = Schema.parseJson(jsonSchema);
    --- End diff --
   
    parse error must be notified. what is some other string passed instead of jsonString ?


---

[qiuchenjian-2]

Github user ajantha-bhat commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2837#discussion_r228464096
 
    --- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonWriterBuilder.java ---
    @@ -329,6 +344,19 @@ public CarbonWriterBuilder withCsvInput(Schema schema) {
         return this;
       }
     
    +  /**
    +   * to build a {@link CarbonWriter}, which accepts row in CSV format
    +   *
    +   * @param jsonSchema json Schema string
    +   * @return CarbonWriterBuilder
    +   */
    +  public CarbonWriterBuilder withCsvInput(String jsonSchema) {
    +    Objects.requireNonNull(jsonSchema, "schema should not be null");
    +    this.schema = Schema.parseJson(jsonSchema);
    --- End diff --
   
    why this new method ? we already support withCsvInput(Schema schema), csdk cannot use it ?


---

[qiuchenjian-2]

Github user ajantha-bhat commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2837#discussion_r228468134
 
    --- Diff: store/CSDK/main.cpp ---
    @@ -484,6 +485,264 @@ bool readPerformanceFromS3(JNIEnv *env, char *argv[]) {
         reader.close();
     }
     
    +/**
    + * test write data to local disk
    + *
    + * @param env  jni env
    + * @return
    + */
    +bool writeToLocal(JNIEnv *env, char *path) {
    +
    +    char *jsonSchema = "[{stringField:string},{shortField:short},{intField:int},{longField:long},{doubleField:double},{boolField:boolean},{dateField:date},{timeField:timestamp},{floatField:float},{arrayField:array}]";
    +
    +    CarbonWriter carbonWriterClass;
    +    carbonWriterClass.builder(env);
    +    carbonWriterClass.outputPath(path);
    +    carbonWriterClass.withCsvInput(jsonSchema);
    +    carbonWriterClass.build();
    +
    +    int rowNum = 10;
    +    int size = 10;
    +    long longValue = 0;
    +    double doubleValue = 0;
    +    float floatValue = 0;
    +    jclass objClass = env->FindClass("java/lang/String");
    +    for (int i = 0; i < rowNum; ++i) {
    +        jobjectArray arr = env->NewObjectArray(size, objClass, 0);
    +        char ctrInt[10];
    +        gcvt(i, 10, ctrInt);
    +
    +        char a[15] = "robot";
    +        strcat(a, ctrInt);
    +        jobject stringField = env->NewStringUTF(a);
    +        env->SetObjectArrayElement(arr, 0, stringField);
    +
    +        char ctrShort[10];
    +        gcvt(i % 10000, 10, ctrShort);
    +        jobject shortField = env->NewStringUTF(ctrShort);
    +        env->SetObjectArrayElement(arr, 1, shortField);
    +
    +        jobject intField = env->NewStringUTF(ctrInt);
    +        env->SetObjectArrayElement(arr, 2, intField);
    +
    +
    +        char ctrLong[10];
    +        gcvt(longValue, 10, ctrLong);
    +        longValue = longValue + 2;
    +        jobject longField = env->NewStringUTF(ctrLong);
    +        env->SetObjectArrayElement(arr, 3, longField);
    +
    +        char ctrDouble[10];
    +        gcvt(doubleValue, 10, ctrDouble);
    +        doubleValue = doubleValue + 2;
    +        jobject doubleField = env->NewStringUTF(ctrDouble);
    +        env->SetObjectArrayElement(arr, 4, doubleField);
    +
    +        jobject boolField = env->NewStringUTF("true");
    +        env->SetObjectArrayElement(arr, 5, boolField);
    +
    +        jobject dateField = env->NewStringUTF(" 2019-03-02");
    +        env->SetObjectArrayElement(arr, 6, dateField);
    +
    +        jobject timeField = env->NewStringUTF("2019-02-12 03:03:34");
    +        env->SetObjectArrayElement(arr, 7, timeField);
    +
    +        char ctrFloat[10];
    +        gcvt(floatValue, 10, ctrFloat);
    +        floatValue = floatValue + 2;
    +        jobject floatField = env->NewStringUTF(ctrFloat);
    +        env->SetObjectArrayElement(arr, 8, floatField);
    +
    +        jobject arrayField = env->NewStringUTF("Hello#World#From#Carbon");
    +        env->SetObjectArrayElement(arr, 9, arrayField);
    +
    +
    +        carbonWriterClass.write(arr);
    +
    +        env->DeleteLocalRef(stringField);
    +        env->DeleteLocalRef(shortField);
    +        env->DeleteLocalRef(intField);
    +        env->DeleteLocalRef(longField);
    +        env->DeleteLocalRef(doubleField);
    +        env->DeleteLocalRef(floatField);
    +        env->DeleteLocalRef(dateField);
    +        env->DeleteLocalRef(timeField);
    +        env->DeleteLocalRef(boolField);
    +        env->DeleteLocalRef(arrayField);
    +        env->DeleteLocalRef(arr);
    +    }
    +    carbonWriterClass.close();
    +
    +    CarbonReader carbonReader;
    +    carbonReader.builder(env, path);
    +    carbonReader.build();
    +    int i = 0;
    +    CarbonRow carbonRow(env);
    --- End diff --
   
    After this we may not need readFromLocal() testcase as this scneario covers it.


---

[qiuchenjian-2]

Github user ajantha-bhat commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2837#discussion_r228468162
 
    --- Diff: store/CSDK/main.cpp ---
    @@ -484,6 +485,264 @@ bool readPerformanceFromS3(JNIEnv *env, char *argv[]) {
         reader.close();
     }
     
    +/**
    + * test write data to local disk
    + *
    + * @param env  jni env
    + * @return
    + */
    +bool writeToLocal(JNIEnv *env, char *path) {
    +
    +    char *jsonSchema = "[{stringField:string},{shortField:short},{intField:int},{longField:long},{doubleField:double},{boolField:boolean},{dateField:date},{timeField:timestamp},{floatField:float},{arrayField:array}]";
    +
    +    CarbonWriter carbonWriterClass;
    +    carbonWriterClass.builder(env);
    +    carbonWriterClass.outputPath(path);
    +    carbonWriterClass.withCsvInput(jsonSchema);
    +    carbonWriterClass.build();
    +
    +    int rowNum = 10;
    +    int size = 10;
    +    long longValue = 0;
    +    double doubleValue = 0;
    +    float floatValue = 0;
    +    jclass objClass = env->FindClass("java/lang/String");
    +    for (int i = 0; i < rowNum; ++i) {
    +        jobjectArray arr = env->NewObjectArray(size, objClass, 0);
    +        char ctrInt[10];
    +        gcvt(i, 10, ctrInt);
    +
    +        char a[15] = "robot";
    +        strcat(a, ctrInt);
    +        jobject stringField = env->NewStringUTF(a);
    +        env->SetObjectArrayElement(arr, 0, stringField);
    +
    +        char ctrShort[10];
    +        gcvt(i % 10000, 10, ctrShort);
    +        jobject shortField = env->NewStringUTF(ctrShort);
    +        env->SetObjectArrayElement(arr, 1, shortField);
    +
    +        jobject intField = env->NewStringUTF(ctrInt);
    +        env->SetObjectArrayElement(arr, 2, intField);
    +
    +
    +        char ctrLong[10];
    +        gcvt(longValue, 10, ctrLong);
    +        longValue = longValue + 2;
    +        jobject longField = env->NewStringUTF(ctrLong);
    +        env->SetObjectArrayElement(arr, 3, longField);
    +
    +        char ctrDouble[10];
    +        gcvt(doubleValue, 10, ctrDouble);
    +        doubleValue = doubleValue + 2;
    +        jobject doubleField = env->NewStringUTF(ctrDouble);
    +        env->SetObjectArrayElement(arr, 4, doubleField);
    +
    +        jobject boolField = env->NewStringUTF("true");
    +        env->SetObjectArrayElement(arr, 5, boolField);
    +
    +        jobject dateField = env->NewStringUTF(" 2019-03-02");
    +        env->SetObjectArrayElement(arr, 6, dateField);
    +
    +        jobject timeField = env->NewStringUTF("2019-02-12 03:03:34");
    +        env->SetObjectArrayElement(arr, 7, timeField);
    +
    +        char ctrFloat[10];
    +        gcvt(floatValue, 10, ctrFloat);
    +        floatValue = floatValue + 2;
    +        jobject floatField = env->NewStringUTF(ctrFloat);
    +        env->SetObjectArrayElement(arr, 8, floatField);
    +
    +        jobject arrayField = env->NewStringUTF("Hello#World#From#Carbon");
    +        env->SetObjectArrayElement(arr, 9, arrayField);
    +
    +
    +        carbonWriterClass.write(arr);
    +
    +        env->DeleteLocalRef(stringField);
    +        env->DeleteLocalRef(shortField);
    +        env->DeleteLocalRef(intField);
    +        env->DeleteLocalRef(longField);
    +        env->DeleteLocalRef(doubleField);
    +        env->DeleteLocalRef(floatField);
    +        env->DeleteLocalRef(dateField);
    +        env->DeleteLocalRef(timeField);
    +        env->DeleteLocalRef(boolField);
    +        env->DeleteLocalRef(arrayField);
    +        env->DeleteLocalRef(arr);
    +    }
    +    carbonWriterClass.close();
    +
    +    CarbonReader carbonReader;
    +    carbonReader.builder(env, path);
    +    carbonReader.build();
    +    int i = 0;
    +    CarbonRow carbonRow(env);
    +    while (carbonReader.hasNext()) {
    +        jobject row = carbonReader.readNextCarbonRow();
    +        i++;
    +        carbonRow.setCarbonRow(row);
    +        printf("%s\t%d\t%ld\t", carbonRow.getString(0), carbonRow.getInt(1), carbonRow.getLong(2));
    +        jobjectArray array1 = carbonRow.getArray(3);
    +        jsize length = env->GetArrayLength(array1);
    +        int j = 0;
    +        for (j = 0; j < length; j++) {
    +            jobject element = env->GetObjectArrayElement(array1, j);
    +            char *str = (char *) env->GetStringUTFChars((jstring) element, JNI_FALSE);
    +            printf("%s\t", str);
    +        }
    +        printf("%d\t", carbonRow.getShort(4));
    +        printf("%d\t", carbonRow.getInt(5));
    +        printf("%ld\t", carbonRow.getLong(6));
    +        printf("%lf\t", carbonRow.getDouble(7));
    +        bool bool1 = carbonRow.getBoolean(8);
    +        if (bool1) {
    +            printf("true\t");
    +        } else {
    +            printf("false\t");
    +        }
    +        printf("%f\t\n", carbonRow.getFloat(9));
    +        env->DeleteLocalRef(row);
    +    }
    +    carbonReader.close();
    +}
    +
    +/**
    + * test write data to S3
    + *
    + * @param env  jni env
    + * @return
    + */
    +bool writeToS3(JNIEnv *env, char *path, char *argv[]) {
    +    printf("test write data ro S3");
    +    char *jsonSchema = "[{stringField:string},{shortField:short},{intField:int},{longField:long},{doubleField:double},{boolField:boolean},{dateField:date},{timeField:timestamp},{floatField:float},{arrayField:array}]";
    +
    +    char *args[3];
    +    // "your access key"
    +    args[0] = argv[1];
    +    // "your secret key"
    +    args[1] = argv[2];
    +    // "your endPoint"
    +    args[2] = argv[3];
    +
    +    CarbonWriter carbonWriterClass;
    +    carbonWriterClass.builder(env);
    +    carbonWriterClass.outputPath(path);
    +    carbonWriterClass.withHadoopConf("fs.s3a.access.key", argv[1]);
    +    carbonWriterClass.withHadoopConf("fs.s3a.secret.key", argv[2]);
    +    carbonWriterClass.withHadoopConf("fs.s3a.endpoint", argv[3]);
    +    carbonWriterClass.withCsvInput(jsonSchema);
    +    carbonWriterClass.build();
    +
    +    int rowNum = 10;
    +    int size = 10;
    +    long longValue = 0;
    +    double doubleValue = 0;
    +    float floatValue = 0;
    +    jclass objClass = env->FindClass("java/lang/String");
    +    for (int i = 0; i < rowNum; ++i) {
    +        jobjectArray arr = env->NewObjectArray(size, objClass, 0);
    +        char ctrInt[10];
    +        gcvt(i, 10, ctrInt);
    +
    +        char a[15] = "robot";
    +        strcat(a, ctrInt);
    +        jobject stringField = env->NewStringUTF(a);
    +        env->SetObjectArrayElement(arr, 0, stringField);
    +
    +        char ctrShort[10];
    +        gcvt(i % 10000, 10, ctrShort);
    +        jobject shortField = env->NewStringUTF(ctrShort);
    +        env->SetObjectArrayElement(arr, 1, shortField);
    +
    +        jobject intField = env->NewStringUTF(ctrInt);
    +        env->SetObjectArrayElement(arr, 2, intField);
    +
    +
    +        char ctrLong[10];
    +        gcvt(longValue, 10, ctrLong);
    +        longValue = longValue + 2;
    +        jobject longField = env->NewStringUTF(ctrLong);
    +        env->SetObjectArrayElement(arr, 3, longField);
    +
    +        char ctrDouble[10];
    +        gcvt(doubleValue, 10, ctrDouble);
    +        doubleValue = doubleValue + 2;
    +        jobject doubleField = env->NewStringUTF(ctrDouble);
    +        env->SetObjectArrayElement(arr, 4, doubleField);
    +
    +        jobject boolField = env->NewStringUTF("true");
    +        env->SetObjectArrayElement(arr, 5, boolField);
    +
    +        jobject dateField = env->NewStringUTF(" 2019-03-02");
    +        env->SetObjectArrayElement(arr, 6, dateField);
    +
    +        jobject timeField = env->NewStringUTF("2019-02-12 03:03:34");
    +        env->SetObjectArrayElement(arr, 7, timeField);
    +
    +        char ctrFloat[10];
    +        gcvt(floatValue, 10, ctrFloat);
    +        floatValue = floatValue + 2;
    +        jobject floatField = env->NewStringUTF(ctrFloat);
    +        env->SetObjectArrayElement(arr, 8, floatField);
    +
    +        jobject arrayField = env->NewStringUTF("Hello#World#From#Carbon");
    +        env->SetObjectArrayElement(arr, 9, arrayField);
    +
    +
    +        carbonWriterClass.write(arr);
    +
    +        env->DeleteLocalRef(stringField);
    +        env->DeleteLocalRef(shortField);
    +        env->DeleteLocalRef(intField);
    +        env->DeleteLocalRef(longField);
    +        env->DeleteLocalRef(doubleField);
    +        env->DeleteLocalRef(floatField);
    +        env->DeleteLocalRef(dateField);
    +        env->DeleteLocalRef(timeField);
    +        env->DeleteLocalRef(boolField);
    +        env->DeleteLocalRef(arrayField);
    +        env->DeleteLocalRef(arr);
    +    }
    +    carbonWriterClass.close();
    +
    +    CarbonReader carbonReader;
    +    carbonReader.builder(env, path);
    +    carbonReader.withHadoopConf("fs.s3a.access.key", argv[1]);
    +    carbonReader.withHadoopConf("fs.s3a.secret.key", argv[2]);
    +    carbonReader.withHadoopConf("fs.s3a.endpoint", argv[3]);
    +    carbonReader.build();
    +    int i = 0;
    +    CarbonRow carbonRow(env);
    --- End diff --
   
    After this we may not need readFromS3 as this scneario covers it.


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2837
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1064/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2837
 
    Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1278/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2837
 
    Build Failed  with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9330/



---

[qiuchenjian-2]

Github user xubo245 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2837#discussion_r228698701
 
    --- Diff: store/CSDK/CarbonWriter.cpp ---
    @@ -0,0 +1,83 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +#include "CarbonWriter.h"
    +
    +jobject CarbonWriter::builder(JNIEnv *env) {
    +    jniEnv = env;
    --- End diff --
   
    it's based on PR2792, not in this PR


---

[qiuchenjian-2]

Github user xubo245 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2837#discussion_r228698705
 
    --- Diff: store/CSDK/CarbonWriter.cpp ---
    @@ -0,0 +1,83 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +#include "CarbonWriter.h"
    +
    +jobject CarbonWriter::builder(JNIEnv *env) {
    +    jniEnv = env;
    +    carbonWriter = env->FindClass("org/apache/carbondata/sdk/file/CarbonWriter");
    +    jmethodID carbonWriterBuilderID = env->GetStaticMethodID(carbonWriter, "builder",
    +        "()Lorg/apache/carbondata/sdk/file/CarbonWriterBuilder;");
    +    carbonWriterBuilderObject = env->CallStaticObjectMethod(carbonWriter, carbonWriterBuilderID);
    +    return carbonWriterBuilderObject;
    +}
    +
    +jobject CarbonWriter::outputPath(char *path) {
    +    jclass carbonWriterBuilderClass = jniEnv->GetObjectClass(carbonWriterBuilderObject);
    +    jmethodID carbonWriterBuilderID = jniEnv->GetMethodID(carbonWriterBuilderClass, "outputPath",
    +        "(Ljava/lang/String;)Lorg/apache/carbondata/sdk/file/CarbonWriterBuilder;");
    +    jstring jPath = jniEnv->NewStringUTF(path);
    +    jvalue args[1];
    +    args[0].l = jPath;
    +    carbonWriterBuilderObject = jniEnv->CallObjectMethodA(carbonWriterBuilderObject, carbonWriterBuilderID, args);
    +    return carbonWriterBuilderObject;
    +}
    +
    +jobject CarbonWriter::withCsvInput(char *jsonSchema) {
    +
    +    jclass carbonWriterBuilderClass = jniEnv->GetObjectClass(carbonWriterBuilderObject);
    +    jmethodID carbonWriterBuilderID = jniEnv->GetMethodID(carbonWriterBuilderClass, "withCsvInput",
    --- End diff --
   
    it's based on PR2792, not in this PR, changed to id


---

[qiuchenjian-2]

Github user xubo245 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2837#discussion_r228699746
 
    --- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonWriterBuilder.java ---
    @@ -329,6 +344,19 @@ public CarbonWriterBuilder withCsvInput(Schema schema) {
         return this;
       }
     
    +  /**
    +   * to build a {@link CarbonWriter}, which accepts row in CSV format
    +   *
    +   * @param jsonSchema json Schema string
    +   * @return CarbonWriterBuilder
    +   */
    +  public CarbonWriterBuilder withCsvInput(String jsonSchema) {
    +    Objects.requireNonNull(jsonSchema, "schema should not be null");
    +    this.schema = Schema.parseJson(jsonSchema);
    --- End diff --
   
    ok, add catch exception when sting is wrong.
   
    CSDK need build Schema object,  which is complex for users.


---