[GitHub] carbondata pull request #2807: [CARBONDATA-2997] Support read schema from in...

[qiuchenjian-2]

Github user xubo245 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2807#discussion_r228570364
 
    --- Diff: store/CSDK/main.cpp ---
    @@ -168,9 +220,51 @@ bool readFromS3(JNIEnv *env, char *argv[]) {
             printf("\n");
         }
     
    +    printf("\nread Schema from Index File from S3:\n");
    +    CarbonSchemaReader carbonSchemaReader(env);
    +    jobject schema = carbonSchemaReader.readSchemaInIndexFile(
    +            "s3a://sdk/WriterOutput/carbondata/510199997055746_batchno0-0-null-510199277323454.carbonindex");
    --- End diff --
   
    no binaries files in repo, only in my local


---

[qiuchenjian-2]

Github user xubo245 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2807#discussion_r228570412
 
    --- Diff: store/CSDK/main.cpp ---
    @@ -168,9 +220,51 @@ bool readFromS3(JNIEnv *env, char *argv[]) {
             printf("\n");
         }
     
    +    printf("\nread Schema from Index File from S3:\n");
    +    CarbonSchemaReader carbonSchemaReader(env);
    +    jobject schema = carbonSchemaReader.readSchemaInIndexFile(
    +            "s3a://sdk/WriterOutput/carbondata/510199997055746_batchno0-0-null-510199277323454.carbonindex");
    +    Schema carbonSchema(env, schema);
    +    int length = carbonSchema.getFieldsLength();
    +    printf("schema length is:%d\n", length);
    +    for (int i = 0; i < length; i++) {
    +        printf("%d\t", i);
    +        printf("%s\t", carbonSchema.getFieldName(i));
    +        printf("%s\n", carbonSchema.getFieldDataTypeName(i));
    +        if (strcmp(carbonSchema.getFieldDataTypeName(i), "ARRAY") == 0) {
    +            printf("%s\n", carbonSchema.getArrayElementTypeName(i));
    +        }
    +    }
    +
         reader.close();
     }
     
    +/**
    + * test read Schema from Index File from S3
    + * TODO: need support in the future
    + *
    + * @param env jni env
    + * @return whether it is success
    + */
    +bool readSchemaInIndexFileFromS3(JNIEnv *env) {
    +    printf("\nread Schema from Index File:\n");
    +    CarbonSchemaReader carbonSchemaReader(env);
    +    jobject schema = carbonSchemaReader.readSchemaInIndexFile(
    +            "s3a://sdk/WriterOutput/carbondata/510199997055746_batchno0-0-null-510199277323454.carbonindex");
    --- End diff --
   
    no binaries files in repo, only in my local


---

[qiuchenjian-2]

Github user xubo245 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2807#discussion_r228570889
 
    --- Diff: store/CSDK/main.cpp ---
    @@ -168,9 +220,51 @@ bool readFromS3(JNIEnv *env, char *argv[]) {
             printf("\n");
         }
     
    +    printf("\nread Schema from Index File from S3:\n");
    +    CarbonSchemaReader carbonSchemaReader(env);
    +    jobject schema = carbonSchemaReader.readSchemaInIndexFile(
    +            "s3a://sdk/WriterOutput/carbondata/510199997055746_batchno0-0-null-510199277323454.carbonindex");
    +    Schema carbonSchema(env, schema);
    +    int length = carbonSchema.getFieldsLength();
    +    printf("schema length is:%d\n", length);
    +    for (int i = 0; i < length; i++) {
    +        printf("%d\t", i);
    +        printf("%s\t", carbonSchema.getFieldName(i));
    +        printf("%s\n", carbonSchema.getFieldDataTypeName(i));
    +        if (strcmp(carbonSchema.getFieldDataTypeName(i), "ARRAY") == 0) {
    +            printf("%s\n", carbonSchema.getArrayElementTypeName(i));
    +        }
    +    }
    +
         reader.close();
     }
     
    +/**
    + * test read Schema from Index File from S3
    + * TODO: need support in the future
    + *
    + * @param env jni env
    + * @return whether it is success
    + */
    +bool readSchemaInIndexFileFromS3(JNIEnv *env) {
    --- End diff --
   
    I will test and support it before merge


---

[qiuchenjian-2]

Github user xubo245 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2807#discussion_r228572221
 
    --- Diff: store/sdk/src/test/java/org/apache/carbondata/sdk/file/CarbonSchemaReaderTest.java ---
    @@ -0,0 +1,233 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    --- End diff --
   
    This test case is test for new class, like Schema. java and we should move read schema test case from CarbonReaderTest to CarbonSchemaReaderTest


---

[qiuchenjian-2]

Github user xubo245 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2807#discussion_r228577510
 
    --- Diff: store/CSDK/Schema.cpp ---
    @@ -0,0 +1,58 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +#include <jni.h>
    +#include "Schema.h"
    +
    +Schema::Schema(JNIEnv *env, jobject schema) {
    --- End diff --
   
    ok, done


---

[qiuchenjian-2]

Github user xubo245 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2807#discussion_r228578342
 
    --- Diff: store/CSDK/main.cpp ---
    @@ -55,6 +57,56 @@ JNIEnv *initJVM() {
         return env;
     }
     
    +/**
    + * test read Schema from Index File
    + *
    + * @param env jni env
    + * @return whether it is success
    + */
    +bool readSchemaInIndexFile(JNIEnv *env) {
    +    printf("\nread Schema from Index File:\n");
    +    CarbonSchemaReader carbonSchemaReader(env);
    +    jobject schema = carbonSchemaReader.readSchemaInIndexFile(
    +            "../resources/carbondata/510199997055746_batchno0-0-null-510199277323454.carbonindex");
    --- End diff --
   
    no in repo


---

[qiuchenjian-2]

Github user xubo245 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2807#discussion_r228580954
 
    --- Diff: store/CSDK/main.cpp ---
    @@ -168,9 +220,51 @@ bool readFromS3(JNIEnv *env, char *argv[]) {
             printf("\n");
         }
     
    +    printf("\nread Schema from Index File from S3:\n");
    +    CarbonSchemaReader carbonSchemaReader(env);
    +    jobject schema = carbonSchemaReader.readSchemaInIndexFile(
    +            "s3a://sdk/WriterOutput/carbondata/510199997055746_batchno0-0-null-510199277323454.carbonindex");
    +    Schema carbonSchema(env, schema);
    +    int length = carbonSchema.getFieldsLength();
    +    printf("schema length is:%d\n", length);
    +    for (int i = 0; i < length; i++) {
    +        printf("%d\t", i);
    +        printf("%s\t", carbonSchema.getFieldName(i));
    +        printf("%s\n", carbonSchema.getFieldDataTypeName(i));
    +        if (strcmp(carbonSchema.getFieldDataTypeName(i), "ARRAY") == 0) {
    +            printf("%s\n", carbonSchema.getArrayElementTypeName(i));
    +        }
    +    }
    +
         reader.close();
     }
     
    +/**
    + * test read Schema from Index File from S3
    + * TODO: need support in the future
    + *
    + * @param env jni env
    + * @return whether it is success
    + */
    +bool readSchemaInIndexFileFromS3(JNIEnv *env) {
    --- End diff --
   
    ok, removed, I will implement in the future


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2807
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1061/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2807
 
    Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1275/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2807
 
    Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9327/



---

[qiuchenjian-2]

Github user xubo245 commented on the issue:

    https://github.com/apache/carbondata/pull/2807
 
    retest this please


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2807
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1085/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2807
 
    Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1297/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2807
 
    Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9349/



---

[qiuchenjian-2]

Github user ajantha-bhat commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2807#discussion_r228920786
 
    --- Diff: store/CSDK/test/main.cpp ---
    @@ -116,6 +118,66 @@ void printResult(JNIEnv *env, CarbonReader reader) {
         reader.close();
     }
     
    +/**
    + * test read Schema from Index File
    + *
    + * @param env jni env
    + * @return whether it is success
    + */
    +bool readSchemaInIndexFile(JNIEnv *env) {
    +    printf("\nread Schema from Index File:\n");
    +    CarbonSchemaReader carbonSchemaReader(env);
    +    jobject schema;
    +    try {
    +        schema = carbonSchemaReader.readSchemaInIndexFile(
    +                "../../../../resources/carbondata/510199997055746_batchno0-0-null-510199277323454.carbonindex");
    +    } catch (jthrowable e) {
    +        env->ExceptionDescribe();
    +    }
    +    Schema carbonSchema(env, schema);
    +    int length = carbonSchema.getFieldsLength();
    +    printf("schema length is:%d\n", length);
    +    for (int i = 0; i < length; i++) {
    +        printf("%d\t", i);
    +        printf("%s\t", carbonSchema.getFieldName(i));
    +        printf("%s\n", carbonSchema.getFieldDataTypeName(i));
    +        if (strcmp(carbonSchema.getFieldDataTypeName(i), "ARRAY") == 0) {
    +            printf("Array Element Type Name is:%s\n", carbonSchema.getArrayElementTypeName(i));
    +        }
    +    }
    +    return true;
    +}
    +
    +/**
    + * test read Schema from Data File
    + *
    + * @param env jni env
    + * @return whether it is success
    + */
    +bool readSchemaInDataFile(JNIEnv *env) {
    +    printf("\nread Schema from Data File:\n");
    +    CarbonSchemaReader carbonSchemaReader(env);
    +    jobject schema;
    +    try {
    +        schema = carbonSchemaReader.readSchemaInDataFile(
    +                "../../../../resources/carbondata/part-0-510199997055746_batchno0-0-null-510199277323454.carbondata");
    --- End diff --
   
    cannot keep your local test case in repo. Make a general test case without binary dependency.


---

[qiuchenjian-2]

Github user ajantha-bhat commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2807#discussion_r228920818
 
    --- Diff: store/CSDK/test/main.cpp ---
    @@ -116,6 +118,66 @@ void printResult(JNIEnv *env, CarbonReader reader) {
         reader.close();
     }
     
    +/**
    + * test read Schema from Index File
    + *
    + * @param env jni env
    + * @return whether it is success
    + */
    +bool readSchemaInIndexFile(JNIEnv *env) {
    +    printf("\nread Schema from Index File:\n");
    +    CarbonSchemaReader carbonSchemaReader(env);
    +    jobject schema;
    +    try {
    +        schema = carbonSchemaReader.readSchemaInIndexFile(
    +                "../../../../resources/carbondata/510199997055746_batchno0-0-null-510199277323454.carbonindex");
    +    } catch (jthrowable e) {
    --- End diff --
   
    cannot keep your local test case in repo. Make a general test case without binary dependency.


---

[qiuchenjian-2]

Github user xubo245 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2807#discussion_r229167245
 
    --- Diff: store/CSDK/test/main.cpp ---
    @@ -116,6 +118,66 @@ void printResult(JNIEnv *env, CarbonReader reader) {
         reader.close();
     }
     
    +/**
    + * test read Schema from Index File
    + *
    + * @param env jni env
    + * @return whether it is success
    + */
    +bool readSchemaInIndexFile(JNIEnv *env) {
    +    printf("\nread Schema from Index File:\n");
    +    CarbonSchemaReader carbonSchemaReader(env);
    +    jobject schema;
    +    try {
    +        schema = carbonSchemaReader.readSchemaInIndexFile(
    +                "../../../../resources/carbondata/510199997055746_batchno0-0-null-510199277323454.carbonindex");
    +    } catch (jthrowable e) {
    +        env->ExceptionDescribe();
    +    }
    +    Schema carbonSchema(env, schema);
    +    int length = carbonSchema.getFieldsLength();
    +    printf("schema length is:%d\n", length);
    +    for (int i = 0; i < length; i++) {
    +        printf("%d\t", i);
    +        printf("%s\t", carbonSchema.getFieldName(i));
    +        printf("%s\n", carbonSchema.getFieldDataTypeName(i));
    +        if (strcmp(carbonSchema.getFieldDataTypeName(i), "ARRAY") == 0) {
    +            printf("Array Element Type Name is:%s\n", carbonSchema.getArrayElementTypeName(i));
    +        }
    +    }
    +    return true;
    +}
    +
    +/**
    + * test read Schema from Data File
    + *
    + * @param env jni env
    + * @return whether it is success
    + */
    +bool readSchemaInDataFile(JNIEnv *env) {
    +    printf("\nread Schema from Data File:\n");
    +    CarbonSchemaReader carbonSchemaReader(env);
    +    jobject schema;
    +    try {
    +        schema = carbonSchemaReader.readSchemaInDataFile(
    +                "../../../../resources/carbondata/part-0-510199997055746_batchno0-0-null-510199277323454.carbondata");
    --- End diff --
   
    ok, removed, after write function merged, I will optimized again.


---

[qiuchenjian-2]

Github user xubo245 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2807#discussion_r229167327
 
    --- Diff: store/CSDK/test/main.cpp ---
    @@ -116,6 +118,66 @@ void printResult(JNIEnv *env, CarbonReader reader) {
         reader.close();
     }
     
    +/**
    + * test read Schema from Index File
    + *
    + * @param env jni env
    + * @return whether it is success
    + */
    +bool readSchemaInIndexFile(JNIEnv *env) {
    +    printf("\nread Schema from Index File:\n");
    +    CarbonSchemaReader carbonSchemaReader(env);
    +    jobject schema;
    +    try {
    +        schema = carbonSchemaReader.readSchemaInIndexFile(
    +                "../../../../resources/carbondata/510199997055746_batchno0-0-null-510199277323454.carbonindex");
    +    } catch (jthrowable e) {
    --- End diff --
   
    ok, removed, after write function merged, I will optimized again.


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2807
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1133/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2807
 
    Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9397/



---