Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2792#discussion_r226560678 --- Diff: docs/documentation.md --- @@ -31,7 +31,7 @@ Apache CarbonData is a new big data file format for faster interactive query usi **CarbonData SQL Language Reference:** CarbonData extends the Spark SQL language and adds several [DDL](./ddl-of-carbondata.md) and [DML](./dml-of-carbondata.md) statements to support operations on it.Refer to the [Reference Manual](./language-manual.md) to understand the supported features and functions. -**Programming Guides:** You can read our guides about [APIs supported](./sdk-guide.md) to learn how to integrate CarbonData with your applications. +**Programming Guides:** You can read our guides about [Java APIs supported](./sdk-guide.md) or [C++ APIs supported](./CSDK-guide.md) to learn how to integrate CarbonData with your applications. --- End diff -- How to optimize? --- |
In reply to this post by qiuchenjian-2
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2792#discussion_r226562463 --- Diff: README.md --- @@ -61,6 +61,7 @@ CarbonData is built using Apache Maven, to [build CarbonData](https://github.com * [CarbonData Pre-aggregate DataMap](https://github.com/apache/carbondata/blob/master/docs/preaggregate-datamap-guide.md) * [CarbonData Timeseries DataMap](https://github.com/apache/carbondata/blob/master/docs/timeseries-datamap-guide.md) * [SDK Guide](https://github.com/apache/carbondata/blob/master/docs/sdk-guide.md) +* [CSDK Guide](https://github.com/apache/carbondata/blob/master/docs/CSDK-guide.md) --- End diff -- ok --- |
In reply to this post by qiuchenjian-2
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2792#discussion_r226562555 --- Diff: docs/CSDK-guide.md --- @@ -68,20 +68,42 @@ JNIEnv *initJVM() { bool readFromLocalWithoutProjection(JNIEnv *env) { CarbonReader carbonReaderClass; - carbonReaderClass.builder(env, "../resources/carbondata", "test"); + carbonReaderClass.builder(env, "../resources/carbondata"); carbonReaderClass.build(); + printf("\nRead data from local without projection:\n"); --- End diff -- ok, done --- |
In reply to this post by qiuchenjian-2
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2792#discussion_r226562638 --- Diff: docs/documentation.md --- @@ -31,7 +31,7 @@ Apache CarbonData is a new big data file format for faster interactive query usi **CarbonData SQL Language Reference:** CarbonData extends the Spark SQL language and adds several [DDL](./ddl-of-carbondata.md) and [DML](./dml-of-carbondata.md) statements to support operations on it.Refer to the [Reference Manual](./language-manual.md) to understand the supported features and functions. -**Programming Guides:** You can read our guides about [APIs supported](./sdk-guide.md) to learn how to integrate CarbonData with your applications. +**Programming Guides:** You can read our guides about [Java APIs supported](./sdk-guide.md) or [C++ APIs supported](./CSDK-guide.md) to learn how to integrate CarbonData with your applications. --- End diff -- ok, done --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2792 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/880/ --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2792 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1078/ --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2792 Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9145/ --- |
In reply to this post by qiuchenjian-2
Github user xuchuanyin commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2792#discussion_r226821049 --- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/file/RowUtil.java --- @@ -0,0 +1,146 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.carbondata.sdk.file; + +import java.io.Serializable; +import java.math.BigDecimal; + +/** + * This row class is used to transfer the row data from one step to other step + */ +public class RowUtil implements Serializable { + + public static String getString(Object[] data, int ordinal) { + return (String) data[ordinal]; + } + + /** + * get short type data by ordinal + * + * @param data carbon row data + * @param ordinal the data index of Row + * @return short data type data + */ + public static short getShort(Object[] data, int ordinal) { + return (short) data[ordinal]; + } + + /** + * get int data type data by ordinal + * + * @param data carbon row data + * @param ordinal the data index of Row + * @return int data type data + */ + public static int getInt(Object[] data, int ordinal) { + return (Integer) data[ordinal]; + } + + /** + * get long data type data by ordinal + * + * @param data carbon row data + * @param ordinal the data index of Row + * @return long data type data + */ + public static long getLong(Object[] data, int ordinal) { + return (long) data[ordinal]; + } + + /** + * get array data type data by ordinal + * + * @param data carbon row data + * @param ordinal the data index of Row + * @return array data type data + */ + public static Object[] getArray(Object[] data, int ordinal) { + return (Object[]) data[ordinal]; + } + + /** + * get double data type data by ordinal + * + * @param data carbon row data + * @param ordinal the data index of Row + * @return double data type data + */ + public static double getDouble(Object[] data, int ordinal) { + return (double) data[ordinal]; + } + + /** + * get boolean data type data by ordinal + * + * @param data carbon row data + * @param ordinal the data index of Row + * @return boolean data type data + */ + public static boolean getBoolean(Object[] data, int ordinal) { + return (boolean) data[ordinal]; + } + + /** + * get byte data type data by ordinal + * + * @param data carbon row data + * @param ordinal the data index of Row + * @return byte data type data + */ + public static Byte getByte(Object[] data, int ordinal) { + return (Byte) data[ordinal]; + } + + /** + * get float data type data by ordinal + * + * @param data carbon row data + * @param ordinal the data index of Row + * @return float data type data + */ + public static float getFloat(Object[] data, int ordinal) { + return (float) data[ordinal]; + } + + /** + * get varchar data type data by ordinal + * This is for CSDK + * JNI don't support varchar, so carbon convert decimal to string --- End diff -- but this method is for varchar not decimal... --- |
In reply to this post by qiuchenjian-2
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2792#discussion_r226830397 --- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/file/RowUtil.java --- @@ -0,0 +1,146 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.carbondata.sdk.file; + +import java.io.Serializable; +import java.math.BigDecimal; + +/** + * This row class is used to transfer the row data from one step to other step + */ +public class RowUtil implements Serializable { + + public static String getString(Object[] data, int ordinal) { + return (String) data[ordinal]; + } + + /** + * get short type data by ordinal + * + * @param data carbon row data + * @param ordinal the data index of Row + * @return short data type data + */ + public static short getShort(Object[] data, int ordinal) { + return (short) data[ordinal]; + } + + /** + * get int data type data by ordinal + * + * @param data carbon row data + * @param ordinal the data index of Row + * @return int data type data + */ + public static int getInt(Object[] data, int ordinal) { + return (Integer) data[ordinal]; + } + + /** + * get long data type data by ordinal + * + * @param data carbon row data + * @param ordinal the data index of Row + * @return long data type data + */ + public static long getLong(Object[] data, int ordinal) { + return (long) data[ordinal]; + } + + /** + * get array data type data by ordinal + * + * @param data carbon row data + * @param ordinal the data index of Row + * @return array data type data + */ + public static Object[] getArray(Object[] data, int ordinal) { + return (Object[]) data[ordinal]; + } + + /** + * get double data type data by ordinal + * + * @param data carbon row data + * @param ordinal the data index of Row + * @return double data type data + */ + public static double getDouble(Object[] data, int ordinal) { + return (double) data[ordinal]; + } + + /** + * get boolean data type data by ordinal + * + * @param data carbon row data + * @param ordinal the data index of Row + * @return boolean data type data + */ + public static boolean getBoolean(Object[] data, int ordinal) { + return (boolean) data[ordinal]; + } + + /** + * get byte data type data by ordinal + * + * @param data carbon row data + * @param ordinal the data index of Row + * @return byte data type data + */ + public static Byte getByte(Object[] data, int ordinal) { + return (Byte) data[ordinal]; + } + + /** + * get float data type data by ordinal + * + * @param data carbon row data + * @param ordinal the data index of Row + * @return float data type data + */ + public static float getFloat(Object[] data, int ordinal) { + return (float) data[ordinal]; + } + + /** + * get varchar data type data by ordinal + * This is for CSDK + * JNI don't support varchar, so carbon convert decimal to string --- End diff -- sorry, I changed, please check. --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2792 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/883/ --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2792 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9148/ --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2792 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1081/ --- |
In reply to this post by qiuchenjian-2
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/2792 retest this please --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2792 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/884/ --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2792 Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9149/ --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2792 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1082/ --- |
In reply to this post by qiuchenjian-2
Github user KanakaKumar commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2792#discussion_r227741794 --- Diff: store/CSDK/CarbonRow.cpp --- @@ -0,0 +1,128 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include <jni.h> +#include <cstring> +#include "CarbonRow.h" + +CarbonRow::CarbonRow(JNIEnv *env) { + this->rowUtilClass = env->FindClass("org/apache/carbondata/sdk/file/RowUtil"); + this->jniEnv = env; +} + +void CarbonRow::setCarbonRow(jobject data) { + this->carbonRow = data; +} + +short CarbonRow::getShort(int ordinal) { + jmethodID buildID = jniEnv->GetStaticMethodID(rowUtilClass, "getShort", + "([Ljava/lang/Object;I)S"); + jvalue args[2]; + args[0].l = carbonRow; + args[1].i = ordinal; + return jniEnv->CallStaticShortMethodA(rowUtilClass, buildID, args); +} + +int CarbonRow::getInt(int ordinal) { + jmethodID buildID = jniEnv->GetStaticMethodID(rowUtilClass, "getInt", + "([Ljava/lang/Object;I)I"); + jvalue args[2]; + args[0].l = carbonRow; + args[1].i = ordinal; + return jniEnv->CallStaticIntMethodA(rowUtilClass, buildID, args); +} + +long CarbonRow::getLong(int ordinal) { + jmethodID buildID = jniEnv->GetStaticMethodID(rowUtilClass, "getLong", + "([Ljava/lang/Object;I)J"); + jvalue args[2]; + args[0].l = carbonRow; + args[1].i = ordinal; + return jniEnv->CallStaticLongMethodA(rowUtilClass, buildID, args); +} + +double CarbonRow::getDouble(int ordinal) { + jmethodID buildID = jniEnv->GetStaticMethodID(rowUtilClass, "getDouble", + "([Ljava/lang/Object;I)D"); + jvalue args[2]; + args[0].l = carbonRow; + args[1].i = ordinal; + return jniEnv->CallStaticDoubleMethodA(rowUtilClass, buildID, args); +} + + +float CarbonRow::getFloat(int ordinal) { + jmethodID buildID = jniEnv->GetStaticMethodID(rowUtilClass, "getFloat", + "([Ljava/lang/Object;I)F"); + jvalue args[2]; + args[0].l = carbonRow; + args[1].i = ordinal; + return jniEnv->CallStaticFloatMethodA(rowUtilClass, buildID, args); +} + +jboolean CarbonRow::getBoolean(int ordinal) { + jmethodID buildID = jniEnv->GetStaticMethodID(rowUtilClass, "getBoolean", + "([Ljava/lang/Object;I)Z"); + jvalue args[2]; + args[0].l = carbonRow; + args[1].i = ordinal; + return jniEnv->CallStaticBooleanMethodA(rowUtilClass, buildID, args); +} + +char *CarbonRow::getString(int ordinal) { + jmethodID buildID = jniEnv->GetStaticMethodID(rowUtilClass, "getString", + "([Ljava/lang/Object;I)Ljava/lang/String;"); + jvalue args[2]; + args[0].l = carbonRow; + args[1].i = ordinal; + jobject data = jniEnv->CallStaticObjectMethodA(rowUtilClass, buildID, args); + + char *str = (char *) jniEnv->GetStringUTFChars((jstring) data, JNI_FALSE); + return str; +} + +char *CarbonRow::getDecimal(int ordinal) { + jmethodID buildID = jniEnv->GetStaticMethodID(rowUtilClass, "getDecimal", --- End diff -- jmethodID buildID = jniEnv->GetStaticMethodID(rowUtilClass, "getDecimal", "([Ljava/lang/Object;I)Ljava/lang/String;"); jvalue args[2]; Accessing the static method and initializing the array is done for every row reading. Create once and reuse may improve performance. Can you please try? --- |
In reply to this post by qiuchenjian-2
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2792#discussion_r227798659 --- Diff: store/CSDK/CarbonRow.cpp --- @@ -0,0 +1,128 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include <jni.h> +#include <cstring> +#include "CarbonRow.h" + +CarbonRow::CarbonRow(JNIEnv *env) { + this->rowUtilClass = env->FindClass("org/apache/carbondata/sdk/file/RowUtil"); + this->jniEnv = env; +} + +void CarbonRow::setCarbonRow(jobject data) { + this->carbonRow = data; +} + +short CarbonRow::getShort(int ordinal) { + jmethodID buildID = jniEnv->GetStaticMethodID(rowUtilClass, "getShort", + "([Ljava/lang/Object;I)S"); + jvalue args[2]; + args[0].l = carbonRow; + args[1].i = ordinal; + return jniEnv->CallStaticShortMethodA(rowUtilClass, buildID, args); +} + +int CarbonRow::getInt(int ordinal) { + jmethodID buildID = jniEnv->GetStaticMethodID(rowUtilClass, "getInt", + "([Ljava/lang/Object;I)I"); + jvalue args[2]; + args[0].l = carbonRow; + args[1].i = ordinal; + return jniEnv->CallStaticIntMethodA(rowUtilClass, buildID, args); +} + +long CarbonRow::getLong(int ordinal) { + jmethodID buildID = jniEnv->GetStaticMethodID(rowUtilClass, "getLong", + "([Ljava/lang/Object;I)J"); + jvalue args[2]; + args[0].l = carbonRow; + args[1].i = ordinal; + return jniEnv->CallStaticLongMethodA(rowUtilClass, buildID, args); +} + +double CarbonRow::getDouble(int ordinal) { + jmethodID buildID = jniEnv->GetStaticMethodID(rowUtilClass, "getDouble", + "([Ljava/lang/Object;I)D"); + jvalue args[2]; + args[0].l = carbonRow; + args[1].i = ordinal; + return jniEnv->CallStaticDoubleMethodA(rowUtilClass, buildID, args); +} + + +float CarbonRow::getFloat(int ordinal) { + jmethodID buildID = jniEnv->GetStaticMethodID(rowUtilClass, "getFloat", + "([Ljava/lang/Object;I)F"); + jvalue args[2]; + args[0].l = carbonRow; + args[1].i = ordinal; + return jniEnv->CallStaticFloatMethodA(rowUtilClass, buildID, args); +} + +jboolean CarbonRow::getBoolean(int ordinal) { + jmethodID buildID = jniEnv->GetStaticMethodID(rowUtilClass, "getBoolean", + "([Ljava/lang/Object;I)Z"); + jvalue args[2]; + args[0].l = carbonRow; + args[1].i = ordinal; + return jniEnv->CallStaticBooleanMethodA(rowUtilClass, buildID, args); +} + +char *CarbonRow::getString(int ordinal) { + jmethodID buildID = jniEnv->GetStaticMethodID(rowUtilClass, "getString", + "([Ljava/lang/Object;I)Ljava/lang/String;"); + jvalue args[2]; + args[0].l = carbonRow; + args[1].i = ordinal; + jobject data = jniEnv->CallStaticObjectMethodA(rowUtilClass, buildID, args); + + char *str = (char *) jniEnv->GetStringUTFChars((jstring) data, JNI_FALSE); + return str; +} + +char *CarbonRow::getDecimal(int ordinal) { + jmethodID buildID = jniEnv->GetStaticMethodID(rowUtilClass, "getDecimal", --- End diff -- ok, done --- |
In reply to this post by qiuchenjian-2
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2792#discussion_r227807282 --- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonReader.java --- @@ -93,32 +92,10 @@ public T readNextRow() throws IOException, InterruptedException { } /** - * Read and return next string row object - * limitation: only single dimension Array is supported - * TODO: support didfferent data type + * Read and return next carbon row object */ - public Object[] readNextStringRow() throws IOException, InterruptedException { - validateReader(); - T t = currentReader.getCurrentValue(); - Object[] objects = (Object[]) t; - String[] strings = new String[objects.length]; - for (int i = 0; i < objects.length; i++) { - if (objects[i] instanceof Object[]) { - Object[] arrayString = (Object[]) objects[i]; - StringBuffer stringBuffer = new StringBuffer(); - stringBuffer.append(String.valueOf(arrayString[0])); - if (arrayString.length > 1) { - for (int j = 1; j < arrayString.length; j++) { - stringBuffer.append(CarbonCommonConstants.ARRAY_SEPARATOR) - .append(String.valueOf(arrayString[j])); - } - } - strings[i] = stringBuffer.toString(); - } else { - strings[i] = String.valueOf(objects[i]); - } - } - return strings; + public Object[] readNextCarbonRow() throws IOException, InterruptedException { + return (Object[]) readNextRow(); --- End diff -- ok, done --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2792 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/996/ --- |
Free forum by Nabble | Edit this page |