GitHub user xubo245 opened a pull request:
https://github.com/apache/carbondata/pull/2337 [CARBONDATA-2519] Add document for CarbonReader Add document for CarbonReader Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? No - [ ] Any backward compatibility impacted? No - [ ] Document update required? Yes - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. No need - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. No You can merge this pull request into a Git repository by running: $ git pull https://github.com/xubo245/carbondata CARBONDATA-2519-carbonreaderDoc Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2337.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2337 ---- commit 71c21c1c634dfdadab27da305241110187d35880 Author: xubo245 <xubo29@...> Date: 2018-05-23T13:45:49Z [CARBONDATA-2519] Add document for CarbonReader ---- --- |
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2337 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6075/ --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2337 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4915/ --- |
In reply to this post by qiuchenjian-2
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2337 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5076/ --- |
In reply to this post by qiuchenjian-2
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2337 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5080/ --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2337 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6078/ --- |
In reply to this post by qiuchenjian-2
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/2337 retest this please --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2337 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4918/ --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2337 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6083/ --- |
In reply to this post by qiuchenjian-2
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/2337 retest this please --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2337 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4924/ --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2337 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6085/ --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2337 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4928/ --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2337 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6089/ --- |
In reply to this post by qiuchenjian-2
Github user ajantha-bhat commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2337#discussion_r190818765 --- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonReaderBuilder.java --- @@ -136,10 +160,17 @@ public CarbonReaderBuilder setEndPoint(String key, String value) { * @return CarbonWriterBuilder */ public CarbonReaderBuilder setEndPoint(String value) { - FileFactory.getConfiguration().set(Constants.ENDPOINT, value); - return this; + return setEndPoint(Constants.ENDPOINT, value); --- End diff -- There is one more implementation of setEndPoint. Please add that also --- |
In reply to this post by qiuchenjian-2
Github user ajantha-bhat commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2337#discussion_r190819502 --- Diff: docs/sdk-guide.md --- @@ -398,3 +399,160 @@ Reference : [list of carbon properties](http://carbondata.apache.org/configurati */ public static org.apache.carbondata.sdk.file.Schema getCarbonSchemaFromAvroSchema(String avroSchemaString); ``` +# SDK Reader +This SDK reader reads CarbonData file and carbonindex file at a given path. +External client can make use of this reader to read CarbonData files without CarbonSession. +## Quick example +``` + // 1. Create carbon reader + String path = "./testWriteFiles"; + CarbonReader reader = CarbonReader + .builder(path, "_temp") + .projection(new String[]{"name", "age"}) + .build(); + + // 2. Read data + int i = 0; + while (reader.hasNext()) { + Object[] row = (Object[]) reader.readNextRow(); + System.out.println(row[0] + "\t" + row[1]); + i++; + } + + // 3. Close this reader + reader.close(); +``` + +Find example code at [CarbonReaderExample](https://github.com/apache/carbondata/blob/master/examples/spark2/src/main/java/org/apache/carbondata/examples/sdk/CarbonReaderExample.java) in the CarbonData repo. + +## API List + +### org.apache.carbondata.sdk.file.CarbonReader +``` + /** + * Return a new CarbonReaderBuilder instance + */ + public static CarbonReaderBuilder builder(String tablePath, String tableName); +``` + +``` + /** + * Read carbondata file and return the schema + */ + public static List<ColumnSchema> readSchemaInDataFile(String dataFilePath); +``` + +``` + /** + * Read schema file and return table info object + */ + public static TableInfo readSchemaFile(String schemaFilePath); +``` + +``` + /** + * Return true if has next row + */ + public boolean hasNext(); +``` + +``` + /** + * Read and return next row object + */ + public T readNextRow(); +``` + +``` + /** + * Close reader + */ + public void close(); +``` + +###org.apache.carbondata.sdk.file.CarbonReaderBuilder +``` + /** + * Construct a CarbonReaderBuilder with table path and table name + * + * @param tablePath table path + * @param tableName table name + */ + CarbonReaderBuilder(String tablePath, String tableName); +``` + +``` + /** + * Configure the projection column names of carbon reader + * + * @param projectionColumnNames projection column names + * @return CarbonReaderBuilder object + */ + public CarbonReaderBuilder projection(String[] projectionColumnNames); +``` + +``` + /** + * Configure the transactional status of table + * + * @param isTransactionalTable whether is transactional table or not + * @return CarbonReaderBuilder object + */ + public CarbonReaderBuilder isTransactionalTable(boolean isTransactionalTable); +``` + +``` + /** + * Configure the filter expression for carbon reader + * + * @param filterExpression filter expression + * @return CarbonReaderBuilder object + */ + public CarbonReaderBuilder filter(Expression filterExpression); +``` + +``` + /** + * Set the access key for S3 + * + * @param key the string of access key for different S3 type,like: fs.s3a.access.key + * @param value the value of access key + * @return CarbonWriterBuilder + */ + public CarbonReaderBuilder setAccessKey(String key, String value); +``` + +``` + /** + * Set the secret key for S3 + * + * @param key the string of secret key for different S3 type,like: fs.s3a.secret.key + * @param value the value of secret key + * @return CarbonWriterBuilder + */ + public CarbonReaderBuilder setSecretKey(String key, String value); +``` + +``` + /** + * Set the endpoint for S3 + * + * @param key the string of endpoint for different S3 type,like: fs.s3a.endpoint + * @param value the value of endpoint + * @return CarbonWriterBuilder + */ + public CarbonReaderBuilder setEndPoint(String key, String value); --- End diff -- There is one more implementation of setEndPoint. Please add that also --- |
In reply to this post by qiuchenjian-2
Github user ajantha-bhat commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2337#discussion_r190819633 --- Diff: docs/sdk-guide.md --- @@ -398,3 +399,160 @@ Reference : [list of carbon properties](http://carbondata.apache.org/configurati */ public static org.apache.carbondata.sdk.file.Schema getCarbonSchemaFromAvroSchema(String avroSchemaString); ``` +# SDK Reader +This SDK reader reads CarbonData file and carbonindex file at a given path. +External client can make use of this reader to read CarbonData files without CarbonSession. +## Quick example +``` + // 1. Create carbon reader + String path = "./testWriteFiles"; + CarbonReader reader = CarbonReader + .builder(path, "_temp") + .projection(new String[]{"name", "age"}) + .build(); + + // 2. Read data + int i = 0; + while (reader.hasNext()) { + Object[] row = (Object[]) reader.readNextRow(); + System.out.println(row[0] + "\t" + row[1]); + i++; + } + + // 3. Close this reader + reader.close(); +``` + +Find example code at [CarbonReaderExample](https://github.com/apache/carbondata/blob/master/examples/spark2/src/main/java/org/apache/carbondata/examples/sdk/CarbonReaderExample.java) in the CarbonData repo. + +## API List + +### org.apache.carbondata.sdk.file.CarbonReader +``` + /** + * Return a new CarbonReaderBuilder instance + */ + public static CarbonReaderBuilder builder(String tablePath, String tableName); +``` + +``` + /** + * Read carbondata file and return the schema + */ + public static List<ColumnSchema> readSchemaInDataFile(String dataFilePath); +``` + +``` + /** + * Read schema file and return table info object + */ + public static TableInfo readSchemaFile(String schemaFilePath); +``` + +``` + /** + * Return true if has next row + */ + public boolean hasNext(); +``` + +``` + /** + * Read and return next row object + */ + public T readNextRow(); +``` + +``` + /** + * Close reader + */ + public void close(); +``` + +###org.apache.carbondata.sdk.file.CarbonReaderBuilder --- End diff -- need to give space after ###, else formatting will not reflect --- |
In reply to this post by qiuchenjian-2
Github user ajantha-bhat commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2337#discussion_r190819817 --- Diff: docs/sdk-guide.md --- @@ -398,3 +399,160 @@ Reference : [list of carbon properties](http://carbondata.apache.org/configurati */ public static org.apache.carbondata.sdk.file.Schema getCarbonSchemaFromAvroSchema(String avroSchemaString); ``` +# SDK Reader +This SDK reader reads CarbonData file and carbonindex file at a given path. +External client can make use of this reader to read CarbonData files without CarbonSession. +## Quick example +``` + // 1. Create carbon reader + String path = "./testWriteFiles"; + CarbonReader reader = CarbonReader + .builder(path, "_temp") + .projection(new String[]{"name", "age"}) + .build(); + + // 2. Read data + int i = 0; + while (reader.hasNext()) { + Object[] row = (Object[]) reader.readNextRow(); + System.out.println(row[0] + "\t" + row[1]); + i++; + } + + // 3. Close this reader + reader.close(); +``` + +Find example code at [CarbonReaderExample](https://github.com/apache/carbondata/blob/master/examples/spark2/src/main/java/org/apache/carbondata/examples/sdk/CarbonReaderExample.java) in the CarbonData repo. + +## API List + +### org.apache.carbondata.sdk.file.CarbonReader --- End diff -- change it to below. Just to be same as above style ### org.apache.carbondata.sdk.file.CarbonReader --- |
In reply to this post by qiuchenjian-2
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2337#discussion_r190825696 --- Diff: docs/sdk-guide.md --- @@ -398,3 +399,160 @@ Reference : [list of carbon properties](http://carbondata.apache.org/configurati */ public static org.apache.carbondata.sdk.file.Schema getCarbonSchemaFromAvroSchema(String avroSchemaString); ``` +# SDK Reader +This SDK reader reads CarbonData file and carbonindex file at a given path. +External client can make use of this reader to read CarbonData files without CarbonSession. +## Quick example +``` + // 1. Create carbon reader + String path = "./testWriteFiles"; + CarbonReader reader = CarbonReader + .builder(path, "_temp") + .projection(new String[]{"name", "age"}) + .build(); + + // 2. Read data + int i = 0; + while (reader.hasNext()) { + Object[] row = (Object[]) reader.readNextRow(); + System.out.println(row[0] + "\t" + row[1]); + i++; + } + + // 3. Close this reader + reader.close(); +``` + +Find example code at [CarbonReaderExample](https://github.com/apache/carbondata/blob/master/examples/spark2/src/main/java/org/apache/carbondata/examples/sdk/CarbonReaderExample.java) in the CarbonData repo. + +## API List + +### org.apache.carbondata.sdk.file.CarbonReader +``` + /** + * Return a new CarbonReaderBuilder instance + */ + public static CarbonReaderBuilder builder(String tablePath, String tableName); +``` + +``` + /** + * Read carbondata file and return the schema + */ + public static List<ColumnSchema> readSchemaInDataFile(String dataFilePath); +``` + +``` + /** + * Read schema file and return table info object + */ + public static TableInfo readSchemaFile(String schemaFilePath); +``` + +``` + /** + * Return true if has next row + */ + public boolean hasNext(); +``` + +``` + /** + * Read and return next row object + */ + public T readNextRow(); +``` + +``` + /** + * Close reader + */ + public void close(); +``` + +###org.apache.carbondata.sdk.file.CarbonReaderBuilder +``` + /** + * Construct a CarbonReaderBuilder with table path and table name + * + * @param tablePath table path + * @param tableName table name + */ + CarbonReaderBuilder(String tablePath, String tableName); +``` + +``` + /** + * Configure the projection column names of carbon reader + * + * @param projectionColumnNames projection column names + * @return CarbonReaderBuilder object + */ + public CarbonReaderBuilder projection(String[] projectionColumnNames); +``` + +``` + /** + * Configure the transactional status of table + * + * @param isTransactionalTable whether is transactional table or not + * @return CarbonReaderBuilder object + */ + public CarbonReaderBuilder isTransactionalTable(boolean isTransactionalTable); +``` + +``` + /** + * Configure the filter expression for carbon reader + * + * @param filterExpression filter expression + * @return CarbonReaderBuilder object + */ + public CarbonReaderBuilder filter(Expression filterExpression); +``` + +``` + /** + * Set the access key for S3 + * + * @param key the string of access key for different S3 type,like: fs.s3a.access.key + * @param value the value of access key + * @return CarbonWriterBuilder + */ + public CarbonReaderBuilder setAccessKey(String key, String value); +``` + +``` + /** + * Set the secret key for S3 + * + * @param key the string of secret key for different S3 type,like: fs.s3a.secret.key + * @param value the value of secret key + * @return CarbonWriterBuilder + */ + public CarbonReaderBuilder setSecretKey(String key, String value); +``` + +``` + /** + * Set the endpoint for S3 + * + * @param key the string of endpoint for different S3 type,like: fs.s3a.endpoint + * @param value the value of endpoint + * @return CarbonWriterBuilder + */ + public CarbonReaderBuilder setEndPoint(String key, String value); --- End diff -- ok, AK and SK also have another interface --- |
In reply to this post by qiuchenjian-2
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2337#discussion_r190825861 --- Diff: docs/sdk-guide.md --- @@ -398,3 +399,160 @@ Reference : [list of carbon properties](http://carbondata.apache.org/configurati */ public static org.apache.carbondata.sdk.file.Schema getCarbonSchemaFromAvroSchema(String avroSchemaString); ``` +# SDK Reader +This SDK reader reads CarbonData file and carbonindex file at a given path. +External client can make use of this reader to read CarbonData files without CarbonSession. +## Quick example +``` + // 1. Create carbon reader + String path = "./testWriteFiles"; + CarbonReader reader = CarbonReader + .builder(path, "_temp") + .projection(new String[]{"name", "age"}) + .build(); + + // 2. Read data + int i = 0; + while (reader.hasNext()) { + Object[] row = (Object[]) reader.readNextRow(); + System.out.println(row[0] + "\t" + row[1]); + i++; + } + + // 3. Close this reader + reader.close(); +``` + +Find example code at [CarbonReaderExample](https://github.com/apache/carbondata/blob/master/examples/spark2/src/main/java/org/apache/carbondata/examples/sdk/CarbonReaderExample.java) in the CarbonData repo. + +## API List + +### org.apache.carbondata.sdk.file.CarbonReader +``` + /** + * Return a new CarbonReaderBuilder instance + */ + public static CarbonReaderBuilder builder(String tablePath, String tableName); +``` + +``` + /** + * Read carbondata file and return the schema + */ + public static List<ColumnSchema> readSchemaInDataFile(String dataFilePath); +``` + +``` + /** + * Read schema file and return table info object + */ + public static TableInfo readSchemaFile(String schemaFilePath); +``` + +``` + /** + * Return true if has next row + */ + public boolean hasNext(); +``` + +``` + /** + * Read and return next row object + */ + public T readNextRow(); +``` + +``` + /** + * Close reader + */ + public void close(); +``` + +###org.apache.carbondata.sdk.file.CarbonReaderBuilder --- End diff -- ok, done --- |
Free forum by Nabble | Edit this page |