[GitHub] carbondata pull request #2738: [CARBONDATA-2952] Provide c++ interface for S...

[qiuchenjian-2]

Github user zzcclp commented on the issue:

    https://github.com/apache/carbondata/pull/2738
 
    @xubo245 Yes, another team of our department use C, now they use C to write parquet file.


---

[qiuchenjian-2]

Github user xubo245 commented on the issue:

    https://github.com/apache/carbondata/pull/2738
 
    ok, I will try to support C to write carbon file later.


---

[qiuchenjian-2]

Github user zzcclp commented on the issue:

    https://github.com/apache/carbondata/pull/2738
 
    Thanks.


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2738
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/468/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2738
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/469/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2738
 
    Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/649/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2738
 
    Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8719/



---

[qiuchenjian-2]

Github user xubo245 commented on the issue:

    https://github.com/apache/carbondata/pull/2738
 
    @KanakaKumar @kunal642 @jackylk @ravikiran23 Can you review it?


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2738#discussion_r220447008
 
    --- Diff: store/CSDK/main.cpp ---
    @@ -0,0 +1,239 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +#include <stdio.h>
    +#include <jni.h>
    +#include <stdlib.h>
    +#include <iostream>
    +#include <unistd.h>
    +#include "CarbonReader.h"
    +
    +using namespace std;
    +
    +JavaVM *jvm;
    +
    +/**
    + * init jvm
    + *
    + * @return
    + */
    +JNIEnv *initJVM() {
    +    JNIEnv *env;
    +    JavaVMInitArgs vm_args;
    +    int parNum = 3;
    +    int res;
    +    JavaVMOption options[parNum];
    +
    +    options[0].optionString = "-Djava.compiler=NONE";
    +    options[1].optionString = "-Djava.class.path=../../sdk/target/carbondata-sdk.jar";
    +    options[2].optionString = "-verbose:jni";
    +    vm_args.version = JNI_VERSION_1_8;
    +    vm_args.nOptions = parNum;
    +    vm_args.options = options;
    +    vm_args.ignoreUnrecognized = JNI_FALSE;
    +
    +    res = JNI_CreateJavaVM(&jvm, (void **) &env, &vm_args);
    +    if (res < 0) {
    +        fprintf(stderr, "\nCan't create Java VM\n");
    +        exit(1);
    +    }
    +
    +    return env;
    +}
    +
    +/**
    + * test read data from local disk, without projection
    + *
    + * @param env  jni env
    + * @return
    + */
    +bool readFromLocalWithoutProjection(JNIEnv *env) {
    +
    +    CarbonReader carbonReaderClass;
    +    carbonReaderClass.builder(env, "../resources/carbondata", "test");
    +    carbonReaderClass.build();
    +
    +    printf("\nRead data from local  without projection:\n");
    +
    +    while (carbonReaderClass.hasNext()) {
    +        jobjectArray row = carbonReaderClass.readNextRow();
    +        jsize length = env->GetArrayLength(row);
    +
    +        int j = 0;
    +        for (j = 0; j < length; j++) {
    +            jobject element = env->GetObjectArrayElement(row, j);
    +            char *str = (char *) env->GetStringUTFChars((jstring) element, JNI_FALSE);
    +            printf("%s\t", str);
    +        }
    +        printf("\n");
    +    }
    +
    +    carbonReaderClass.close();
    +}
    +
    +/**
    + * test read data from local disk
    + *
    + * @param env  jni env
    + * @return
    + */
    +bool readFromLocal(JNIEnv *env) {
    +
    +    CarbonReader carbonReaderClass;
    +    carbonReaderClass.builder(env, "../resources/carbondata", "test");
    +
    +    char *argv[11];
    +    argv[0] = "stringField";
    +    argv[1] = "shortField";
    +    argv[2] = "intField";
    +    argv[3] = "longField";
    +    argv[4] = "doubleField";
    +    argv[5] = "boolField";
    +    argv[6] = "dateField";
    +    argv[7] = "timeField";
    +    argv[8] = "decimalField";
    +    argv[9] = "varcharField";
    +    argv[10] = "arrayField";
    +    carbonReaderClass.projection(11, argv);
    +
    +    carbonReaderClass.build();
    +
    +    printf("\nRead data from local:\n");
    +
    +    while (carbonReaderClass.hasNext()) {
    +        jobjectArray row = carbonReaderClass.readNextRow();
    +        jsize length = env->GetArrayLength(row);
    +
    +        int j = 0;
    +        for (j = 0; j < length; j++) {
    +            jobject element = env->GetObjectArrayElement(row, j);
    +            char *str = (char *) env->GetStringUTFChars((jstring) element, JNI_FALSE);
    +            printf("%s\t", str);
    +        }
    +        printf("\n");
    +    }
    +
    +    carbonReaderClass.close();
    +}
    +
    +
    +/**
    + * read data from S3
    + * parameter is ak sk endpoint
    + *
    + * @param env jni env
    + * @param argv argument vector
    + * @return
    + */
    +bool readFromS3_2(JNIEnv *env, char *argv[]) {
    +    CarbonReader carbonReaderClass;
    +
    +    char *args[3];
    +    // "your access key"
    +    args[0] = argv[1];
    +    // "your secret key"
    +    args[1] = argv[2];
    +    // "your endPoint"
    +    args[2] = argv[3];
    +
    +    carbonReaderClass.builder(env, "s3a://sdk/WriterOutput", "test");
    +    carbonReaderClass.build(args[0], args[1], args[2]);
    +
    +    printf("\nRead data from S3:\n");
    +    while (carbonReaderClass.hasNext()) {
    +        jobjectArray row = carbonReaderClass.readNextRow();
    +        jsize length = env->GetArrayLength(row);
    +
    +        int j = 0;
    +        for (j = 0; j < length; j++) {
    +            jobject element = env->GetObjectArrayElement(row, j);
    +            char *str = (char *) env->GetStringUTFChars((jstring) element, JNI_FALSE);
    +            printf("%s\t", str);
    +        }
    +        printf("\n");
    +    }
    +
    +    carbonReaderClass.close();
    +}
    +
    +/**
    + * read data from S3
    + * parameter is ak sk endpoint
    + *
    + * @param env jni env
    + * @param argv argument vector
    + * @return
    + */
    +bool readFromS3(JNIEnv *env, char *argv[]) {
    +    CarbonReader carbonReaderClass;
    +
    +    char *args[3];
    +    // "your access key"
    +    args[0] = argv[1];
    +    // "your secret key"
    +    args[1] = argv[2];
    +    // "your endPoint"
    +    args[2] = argv[3];
    +
    +    carbonReaderClass.builder(env, "s3a://sdk/WriterOutput", "test");
    +    carbonReaderClass.build(3, args);
    +
    +    printf("\nRead data from S3:\n");
    +    while (carbonReaderClass.hasNext()) {
    +        jobjectArray row = carbonReaderClass.readNextRow();
    +        jsize length = env->GetArrayLength(row);
    +
    +        int j = 0;
    +        for (j = 0; j < length; j++) {
    +            jobject element = env->GetObjectArrayElement(row, j);
    +            char *str = (char *) env->GetStringUTFChars((jstring) element, JNI_FALSE);
    +            printf("%s\t", str);
    +        }
    +        printf("\n");
    +    }
    +
    +    carbonReaderClass.close();
    +}
    +
    +
    +/**
    + * This a example for C++ interface to read carbon file
    + * If you want to test read data fromS3, please input the parameter: ak sk endpoint
    + *
    + * @param argc argument counter
    + * @param argv argument vector
    + * @return
    + */
    +int main(int argc, char *argv[]) {
    +    // init jvm
    +    JNIEnv *env;
    +    env = initJVM();
    +
    +    if (argc > 3) {
    +        readFromS3(env, argv);
    +    } else {
    +        readFromLocal(env);
    +        readFromLocalWithoutProjection(env);
    +    }
    +    cout << "destory jvm\n\n";
    +    (jvm)->DestroyJavaVM();
    +
    +    cout << "\nfinish destory jvm";
    +    fprintf(stdout, "Java VM destory.\n");
    +    return 0;
    +}
    +
    --- End diff --
   
    There are some binary files added in this PR, please remove them


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2738#discussion_r220448815
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java ---
    @@ -506,6 +506,10 @@
        * File separator
        */
       public static final String FILE_SEPARATOR = "/";
    +  /**
    +   * ARRAY separator
    +   */
    +  public static final String ARRAY_SEPARATOR = "\001";
    --- End diff --
   
    Can you put this in C code as it is not used in Java side


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2738#discussion_r220450208
 
    --- Diff: store/sdk/pom.xml ---
    @@ -39,6 +39,16 @@
           <artifactId>hadoop-aws</artifactId>
           <version>${hadoop.version}</version>
         </dependency>
    +    <dependency>
    +      <groupId>org.apache.httpcomponents</groupId>
    +      <artifactId>httpclient</artifactId>
    --- End diff --
   
    Why is this required?


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2738#discussion_r220450236
 
    --- Diff: store/sdk/pom.xml ---
    @@ -39,6 +39,16 @@
           <artifactId>hadoop-aws</artifactId>
           <version>${hadoop.version}</version>
         </dependency>
    +    <dependency>
    +      <groupId>org.apache.httpcomponents</groupId>
    +      <artifactId>httpclient</artifactId>
    +      <version>4.2</version>
    +    </dependency>
    +    <dependency>
    +      <groupId>org.apache.hadoop</groupId>
    +      <artifactId>hadoop-common</artifactId>
    --- End diff --
   
    Why is this required?


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2738#discussion_r220450790
 
    --- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonReaderBuilder.java ---
    @@ -161,6 +161,20 @@ public CarbonReaderBuilder setEndPoint(String key, String value) {
         return this;
       }
     
    +
    +  public CarbonReaderBuilder withHadoopConf(String[] args) {
    --- End diff --
   
    I think it is added in master already


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2738#discussion_r220451116
 
    --- Diff: store/CSDK/main.cpp ---
    @@ -0,0 +1,239 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +#include <stdio.h>
    +#include <jni.h>
    +#include <stdlib.h>
    +#include <iostream>
    +#include <unistd.h>
    +#include "CarbonReader.h"
    +
    +using namespace std;
    +
    +JavaVM *jvm;
    +
    +/**
    + * init jvm
    + *
    + * @return
    + */
    +JNIEnv *initJVM() {
    +    JNIEnv *env;
    +    JavaVMInitArgs vm_args;
    +    int parNum = 3;
    +    int res;
    +    JavaVMOption options[parNum];
    +
    +    options[0].optionString = "-Djava.compiler=NONE";
    +    options[1].optionString = "-Djava.class.path=../../sdk/target/carbondata-sdk.jar";
    +    options[2].optionString = "-verbose:jni";
    +    vm_args.version = JNI_VERSION_1_8;
    +    vm_args.nOptions = parNum;
    +    vm_args.options = options;
    +    vm_args.ignoreUnrecognized = JNI_FALSE;
    +
    +    res = JNI_CreateJavaVM(&jvm, (void **) &env, &vm_args);
    +    if (res < 0) {
    +        fprintf(stderr, "\nCan't create Java VM\n");
    +        exit(1);
    +    }
    +
    +    return env;
    +}
    +
    +/**
    + * test read data from local disk, without projection
    + *
    + * @param env  jni env
    + * @return
    + */
    +bool readFromLocalWithoutProjection(JNIEnv *env) {
    +
    +    CarbonReader carbonReaderClass;
    +    carbonReaderClass.builder(env, "../resources/carbondata", "test");
    +    carbonReaderClass.build();
    +
    +    printf("\nRead data from local  without projection:\n");
    +
    +    while (carbonReaderClass.hasNext()) {
    +        jobjectArray row = carbonReaderClass.readNextRow();
    +        jsize length = env->GetArrayLength(row);
    +
    +        int j = 0;
    +        for (j = 0; j < length; j++) {
    +            jobject element = env->GetObjectArrayElement(row, j);
    +            char *str = (char *) env->GetStringUTFChars((jstring) element, JNI_FALSE);
    +            printf("%s\t", str);
    +        }
    +        printf("\n");
    +    }
    +
    +    carbonReaderClass.close();
    +}
    +
    +/**
    + * test read data from local disk
    + *
    + * @param env  jni env
    + * @return
    + */
    +bool readFromLocal(JNIEnv *env) {
    +
    +    CarbonReader carbonReaderClass;
    +    carbonReaderClass.builder(env, "../resources/carbondata", "test");
    +
    +    char *argv[11];
    +    argv[0] = "stringField";
    +    argv[1] = "shortField";
    +    argv[2] = "intField";
    +    argv[3] = "longField";
    +    argv[4] = "doubleField";
    +    argv[5] = "boolField";
    +    argv[6] = "dateField";
    +    argv[7] = "timeField";
    +    argv[8] = "decimalField";
    +    argv[9] = "varcharField";
    +    argv[10] = "arrayField";
    +    carbonReaderClass.projection(11, argv);
    +
    +    carbonReaderClass.build();
    +
    +    printf("\nRead data from local:\n");
    +
    +    while (carbonReaderClass.hasNext()) {
    +        jobjectArray row = carbonReaderClass.readNextRow();
    +        jsize length = env->GetArrayLength(row);
    +
    +        int j = 0;
    +        for (j = 0; j < length; j++) {
    +            jobject element = env->GetObjectArrayElement(row, j);
    +            char *str = (char *) env->GetStringUTFChars((jstring) element, JNI_FALSE);
    +            printf("%s\t", str);
    +        }
    +        printf("\n");
    +    }
    +
    +    carbonReaderClass.close();
    +}
    +
    +
    +/**
    + * read data from S3
    + * parameter is ak sk endpoint
    + *
    + * @param env jni env
    + * @param argv argument vector
    + * @return
    + */
    +bool readFromS3_2(JNIEnv *env, char *argv[]) {
    +    CarbonReader carbonReaderClass;
    +
    +    char *args[3];
    +    // "your access key"
    +    args[0] = argv[1];
    +    // "your secret key"
    +    args[1] = argv[2];
    +    // "your endPoint"
    +    args[2] = argv[3];
    +
    +    carbonReaderClass.builder(env, "s3a://sdk/WriterOutput", "test");
    +    carbonReaderClass.build(args[0], args[1], args[2]);
    +
    +    printf("\nRead data from S3:\n");
    +    while (carbonReaderClass.hasNext()) {
    +        jobjectArray row = carbonReaderClass.readNextRow();
    +        jsize length = env->GetArrayLength(row);
    +
    +        int j = 0;
    +        for (j = 0; j < length; j++) {
    +            jobject element = env->GetObjectArrayElement(row, j);
    +            char *str = (char *) env->GetStringUTFChars((jstring) element, JNI_FALSE);
    +            printf("%s\t", str);
    +        }
    +        printf("\n");
    +    }
    +
    +    carbonReaderClass.close();
    +}
    +
    +/**
    + * read data from S3
    + * parameter is ak sk endpoint
    + *
    + * @param env jni env
    + * @param argv argument vector
    + * @return
    + */
    +bool readFromS3(JNIEnv *env, char *argv[]) {
    +    CarbonReader carbonReaderClass;
    --- End diff --
   
    rename carbonReaderClass to reader


---

[qiuchenjian-2]

Github user xubo245 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2738#discussion_r220456385
 
    --- Diff: store/CSDK/main.cpp ---
    @@ -0,0 +1,239 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +#include <stdio.h>
    +#include <jni.h>
    +#include <stdlib.h>
    +#include <iostream>
    +#include <unistd.h>
    +#include "CarbonReader.h"
    +
    +using namespace std;
    +
    +JavaVM *jvm;
    +
    +/**
    + * init jvm
    + *
    + * @return
    + */
    +JNIEnv *initJVM() {
    +    JNIEnv *env;
    +    JavaVMInitArgs vm_args;
    +    int parNum = 3;
    +    int res;
    +    JavaVMOption options[parNum];
    +
    +    options[0].optionString = "-Djava.compiler=NONE";
    +    options[1].optionString = "-Djava.class.path=../../sdk/target/carbondata-sdk.jar";
    +    options[2].optionString = "-verbose:jni";
    +    vm_args.version = JNI_VERSION_1_8;
    +    vm_args.nOptions = parNum;
    +    vm_args.options = options;
    +    vm_args.ignoreUnrecognized = JNI_FALSE;
    +
    +    res = JNI_CreateJavaVM(&jvm, (void **) &env, &vm_args);
    +    if (res < 0) {
    +        fprintf(stderr, "\nCan't create Java VM\n");
    +        exit(1);
    +    }
    +
    +    return env;
    +}
    +
    +/**
    + * test read data from local disk, without projection
    + *
    + * @param env  jni env
    + * @return
    + */
    +bool readFromLocalWithoutProjection(JNIEnv *env) {
    +
    +    CarbonReader carbonReaderClass;
    +    carbonReaderClass.builder(env, "../resources/carbondata", "test");
    +    carbonReaderClass.build();
    +
    +    printf("\nRead data from local  without projection:\n");
    +
    +    while (carbonReaderClass.hasNext()) {
    +        jobjectArray row = carbonReaderClass.readNextRow();
    +        jsize length = env->GetArrayLength(row);
    +
    +        int j = 0;
    +        for (j = 0; j < length; j++) {
    +            jobject element = env->GetObjectArrayElement(row, j);
    +            char *str = (char *) env->GetStringUTFChars((jstring) element, JNI_FALSE);
    +            printf("%s\t", str);
    +        }
    +        printf("\n");
    +    }
    +
    +    carbonReaderClass.close();
    +}
    +
    +/**
    + * test read data from local disk
    + *
    + * @param env  jni env
    + * @return
    + */
    +bool readFromLocal(JNIEnv *env) {
    +
    +    CarbonReader carbonReaderClass;
    +    carbonReaderClass.builder(env, "../resources/carbondata", "test");
    +
    +    char *argv[11];
    +    argv[0] = "stringField";
    +    argv[1] = "shortField";
    +    argv[2] = "intField";
    +    argv[3] = "longField";
    +    argv[4] = "doubleField";
    +    argv[5] = "boolField";
    +    argv[6] = "dateField";
    +    argv[7] = "timeField";
    +    argv[8] = "decimalField";
    +    argv[9] = "varcharField";
    +    argv[10] = "arrayField";
    +    carbonReaderClass.projection(11, argv);
    +
    +    carbonReaderClass.build();
    +
    +    printf("\nRead data from local:\n");
    +
    +    while (carbonReaderClass.hasNext()) {
    +        jobjectArray row = carbonReaderClass.readNextRow();
    +        jsize length = env->GetArrayLength(row);
    +
    +        int j = 0;
    +        for (j = 0; j < length; j++) {
    +            jobject element = env->GetObjectArrayElement(row, j);
    +            char *str = (char *) env->GetStringUTFChars((jstring) element, JNI_FALSE);
    +            printf("%s\t", str);
    +        }
    +        printf("\n");
    +    }
    +
    +    carbonReaderClass.close();
    +}
    +
    +
    +/**
    + * read data from S3
    + * parameter is ak sk endpoint
    + *
    + * @param env jni env
    + * @param argv argument vector
    + * @return
    + */
    +bool readFromS3_2(JNIEnv *env, char *argv[]) {
    +    CarbonReader carbonReaderClass;
    +
    +    char *args[3];
    +    // "your access key"
    +    args[0] = argv[1];
    +    // "your secret key"
    +    args[1] = argv[2];
    +    // "your endPoint"
    +    args[2] = argv[3];
    +
    +    carbonReaderClass.builder(env, "s3a://sdk/WriterOutput", "test");
    +    carbonReaderClass.build(args[0], args[1], args[2]);
    +
    +    printf("\nRead data from S3:\n");
    +    while (carbonReaderClass.hasNext()) {
    +        jobjectArray row = carbonReaderClass.readNextRow();
    +        jsize length = env->GetArrayLength(row);
    +
    +        int j = 0;
    +        for (j = 0; j < length; j++) {
    +            jobject element = env->GetObjectArrayElement(row, j);
    +            char *str = (char *) env->GetStringUTFChars((jstring) element, JNI_FALSE);
    +            printf("%s\t", str);
    +        }
    +        printf("\n");
    +    }
    +
    +    carbonReaderClass.close();
    +}
    +
    +/**
    + * read data from S3
    + * parameter is ak sk endpoint
    + *
    + * @param env jni env
    + * @param argv argument vector
    + * @return
    + */
    +bool readFromS3(JNIEnv *env, char *argv[]) {
    +    CarbonReader carbonReaderClass;
    +
    +    char *args[3];
    +    // "your access key"
    +    args[0] = argv[1];
    +    // "your secret key"
    +    args[1] = argv[2];
    +    // "your endPoint"
    +    args[2] = argv[3];
    +
    +    carbonReaderClass.builder(env, "s3a://sdk/WriterOutput", "test");
    +    carbonReaderClass.build(3, args);
    +
    +    printf("\nRead data from S3:\n");
    +    while (carbonReaderClass.hasNext()) {
    +        jobjectArray row = carbonReaderClass.readNextRow();
    +        jsize length = env->GetArrayLength(row);
    +
    +        int j = 0;
    +        for (j = 0; j < length; j++) {
    +            jobject element = env->GetObjectArrayElement(row, j);
    +            char *str = (char *) env->GetStringUTFChars((jstring) element, JNI_FALSE);
    +            printf("%s\t", str);
    +        }
    +        printf("\n");
    +    }
    +
    +    carbonReaderClass.close();
    +}
    +
    +
    +/**
    + * This a example for C++ interface to read carbon file
    + * If you want to test read data fromS3, please input the parameter: ak sk endpoint
    + *
    + * @param argc argument counter
    + * @param argv argument vector
    + * @return
    + */
    +int main(int argc, char *argv[]) {
    +    // init jvm
    +    JNIEnv *env;
    +    env = initJVM();
    +
    +    if (argc > 3) {
    +        readFromS3(env, argv);
    +    } else {
    +        readFromLocal(env);
    +        readFromLocalWithoutProjection(env);
    +    }
    +    cout << "destory jvm\n\n";
    +    (jvm)->DestroyJavaVM();
    +
    +    cout << "\nfinish destory jvm";
    +    fprintf(stdout, "Java VM destory.\n");
    +    return 0;
    +}
    +
    --- End diff --
   
    ok, done


---

[qiuchenjian-2]

Github user xubo245 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2738#discussion_r220457759
 
    --- Diff: store/sdk/pom.xml ---
    @@ -39,6 +39,16 @@
           <artifactId>hadoop-aws</artifactId>
           <version>${hadoop.version}</version>
         </dependency>
    +    <dependency>
    +      <groupId>org.apache.httpcomponents</groupId>
    +      <artifactId>httpclient</artifactId>
    --- End diff --
   
    support S3


---

[qiuchenjian-2]

Github user xubo245 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2738#discussion_r220459029
 
    --- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonReaderBuilder.java ---
    @@ -161,6 +161,20 @@ public CarbonReaderBuilder setEndPoint(String key, String value) {
         return this;
       }
     
    +
    +  public CarbonReaderBuilder withHadoopConf(String[] args) {
    --- End diff --
   
    master only support withHadoopConf(Configuration conf),  Configuration can't work in C side


---

[qiuchenjian-2]

Github user xubo245 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2738#discussion_r220459945
 
    --- Diff: store/CSDK/main.cpp ---
    @@ -0,0 +1,239 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +#include <stdio.h>
    +#include <jni.h>
    +#include <stdlib.h>
    +#include <iostream>
    +#include <unistd.h>
    +#include "CarbonReader.h"
    +
    +using namespace std;
    +
    +JavaVM *jvm;
    +
    +/**
    + * init jvm
    + *
    + * @return
    + */
    +JNIEnv *initJVM() {
    +    JNIEnv *env;
    +    JavaVMInitArgs vm_args;
    +    int parNum = 3;
    +    int res;
    +    JavaVMOption options[parNum];
    +
    +    options[0].optionString = "-Djava.compiler=NONE";
    +    options[1].optionString = "-Djava.class.path=../../sdk/target/carbondata-sdk.jar";
    +    options[2].optionString = "-verbose:jni";
    +    vm_args.version = JNI_VERSION_1_8;
    +    vm_args.nOptions = parNum;
    +    vm_args.options = options;
    +    vm_args.ignoreUnrecognized = JNI_FALSE;
    +
    +    res = JNI_CreateJavaVM(&jvm, (void **) &env, &vm_args);
    +    if (res < 0) {
    +        fprintf(stderr, "\nCan't create Java VM\n");
    +        exit(1);
    +    }
    +
    +    return env;
    +}
    +
    +/**
    + * test read data from local disk, without projection
    + *
    + * @param env  jni env
    + * @return
    + */
    +bool readFromLocalWithoutProjection(JNIEnv *env) {
    +
    +    CarbonReader carbonReaderClass;
    +    carbonReaderClass.builder(env, "../resources/carbondata", "test");
    +    carbonReaderClass.build();
    +
    +    printf("\nRead data from local  without projection:\n");
    +
    +    while (carbonReaderClass.hasNext()) {
    +        jobjectArray row = carbonReaderClass.readNextRow();
    +        jsize length = env->GetArrayLength(row);
    +
    +        int j = 0;
    +        for (j = 0; j < length; j++) {
    +            jobject element = env->GetObjectArrayElement(row, j);
    +            char *str = (char *) env->GetStringUTFChars((jstring) element, JNI_FALSE);
    +            printf("%s\t", str);
    +        }
    +        printf("\n");
    +    }
    +
    +    carbonReaderClass.close();
    +}
    +
    +/**
    + * test read data from local disk
    + *
    + * @param env  jni env
    + * @return
    + */
    +bool readFromLocal(JNIEnv *env) {
    +
    +    CarbonReader carbonReaderClass;
    +    carbonReaderClass.builder(env, "../resources/carbondata", "test");
    +
    +    char *argv[11];
    +    argv[0] = "stringField";
    +    argv[1] = "shortField";
    +    argv[2] = "intField";
    +    argv[3] = "longField";
    +    argv[4] = "doubleField";
    +    argv[5] = "boolField";
    +    argv[6] = "dateField";
    +    argv[7] = "timeField";
    +    argv[8] = "decimalField";
    +    argv[9] = "varcharField";
    +    argv[10] = "arrayField";
    +    carbonReaderClass.projection(11, argv);
    +
    +    carbonReaderClass.build();
    +
    +    printf("\nRead data from local:\n");
    +
    +    while (carbonReaderClass.hasNext()) {
    +        jobjectArray row = carbonReaderClass.readNextRow();
    +        jsize length = env->GetArrayLength(row);
    +
    +        int j = 0;
    +        for (j = 0; j < length; j++) {
    +            jobject element = env->GetObjectArrayElement(row, j);
    +            char *str = (char *) env->GetStringUTFChars((jstring) element, JNI_FALSE);
    +            printf("%s\t", str);
    +        }
    +        printf("\n");
    +    }
    +
    +    carbonReaderClass.close();
    +}
    +
    +
    +/**
    + * read data from S3
    + * parameter is ak sk endpoint
    + *
    + * @param env jni env
    + * @param argv argument vector
    + * @return
    + */
    +bool readFromS3_2(JNIEnv *env, char *argv[]) {
    +    CarbonReader carbonReaderClass;
    +
    +    char *args[3];
    +    // "your access key"
    +    args[0] = argv[1];
    +    // "your secret key"
    +    args[1] = argv[2];
    +    // "your endPoint"
    +    args[2] = argv[3];
    +
    +    carbonReaderClass.builder(env, "s3a://sdk/WriterOutput", "test");
    +    carbonReaderClass.build(args[0], args[1], args[2]);
    +
    +    printf("\nRead data from S3:\n");
    +    while (carbonReaderClass.hasNext()) {
    +        jobjectArray row = carbonReaderClass.readNextRow();
    +        jsize length = env->GetArrayLength(row);
    +
    +        int j = 0;
    +        for (j = 0; j < length; j++) {
    +            jobject element = env->GetObjectArrayElement(row, j);
    +            char *str = (char *) env->GetStringUTFChars((jstring) element, JNI_FALSE);
    +            printf("%s\t", str);
    +        }
    +        printf("\n");
    +    }
    +
    +    carbonReaderClass.close();
    +}
    +
    +/**
    + * read data from S3
    + * parameter is ak sk endpoint
    + *
    + * @param env jni env
    + * @param argv argument vector
    + * @return
    + */
    +bool readFromS3(JNIEnv *env, char *argv[]) {
    +    CarbonReader carbonReaderClass;
    --- End diff --
   
    ok, done


---

[qiuchenjian-2]

Github user xubo245 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2738#discussion_r220461925
 
    --- Diff: store/sdk/pom.xml ---
    @@ -39,6 +39,16 @@
           <artifactId>hadoop-aws</artifactId>
           <version>${hadoop.version}</version>
         </dependency>
    +    <dependency>
    +      <groupId>org.apache.httpcomponents</groupId>
    +      <artifactId>httpclient</artifactId>
    +      <version>4.2</version>
    +    </dependency>
    +    <dependency>
    +      <groupId>org.apache.hadoop</groupId>
    +      <artifactId>hadoop-common</artifactId>
    --- End diff --
   
    support S3


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2738
 
    Build Failed  with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8756/



---