Github user gvramana commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2198#discussion_r183793655 --- Diff: docs/sdk-writer-guide.md --- @@ -0,0 +1,172 @@ +# SDK Writer Guide +In the carbon jars package, there exist a carbondata-store-sdk-x.x.x-SNAPSHOT.jar. +This SDK writer, writes carbondata file and carbonindex file at a given path. +External client can make use of this writer to convert other format data or live data to create carbondata and index files. +These SDK writer output contains just a carbondata and carbonindex files. No metadata folder will be present. + +## Quick example + +```scala + import java.io.IOException; + + import org.apache.carbondata.common.exceptions.sql.InvalidLoadOptionException; + import org.apache.carbondata.core.metadata.datatype.DataTypes; + import org.apache.carbondata.sdk.file.CarbonWriter; + import org.apache.carbondata.sdk.file.CarbonWriterBuilder; + import org.apache.carbondata.sdk.file.Field; + import org.apache.carbondata.sdk.file.Schema; + + public class TestSdk { + + public static void main(String[] args) throws IOException, InvalidLoadOptionException { + testSdkWriter(); + } + + public static void testSdkWriter() throws IOException, InvalidLoadOptionException { + String path ="/home/root1/Documents/ab/temp"; + + Field[] fields =new Field[2]; + fields[0] = new Field("name", DataTypes.STRING); + fields[1] = new Field("age", DataTypes.INT); + + Schema schema =new Schema(fields); + + CarbonWriterBuilder builder = CarbonWriter.builder() + .withSchema(schema) + .outputPath(path); + + CarbonWriter writer = builder.buildWriterForCSVInput(); + + int rows = 5; + for (int i = 0; i < rows; i++) { + writer.write(new String[]{"robot" + (i % 10), String.valueOf(i)}); + } + writer.close(); + } + } +``` + +## Datatypes Mapping +Each of SQL data types are mapped into data types of SDK. Following are the mapping: + +| SQL DataTypes | Mapped SDK DataTypes | +|---------------|----------------------| +| BOOLEAN | DataTypes.BOOLEAN | +| SMALLINT | DataTypes.SHORT | +| INTEGER | DataTypes.INT | +| BIGINT | DataTypes.LONG | +| DOUBLE | DataTypes.DOUBLE | +| VARCHAR | DataTypes.STRING | +| DATE | DataTypes.DATE | +| TIMESTAMP | DataTypes.TIMESTAMP | +| STRING | DataTypes.STRING | +| DECIMAL | DataTypes.createDecimalType(precision, scale) | + + +## API List +``` +/** +* prepares the builder with the schema provided +* @param schema is instance of Schema +* @return updated CarbonWriterBuilder +*/ +public CarbonWriterBuilder withSchema(Schema schema); +``` + +``` +/** +* Sets the output path of the writer builder +* @param path is the absolute path where output files are written +* @return updated CarbonWriterBuilder +*/ +public CarbonWriterBuilder outputPath(String path); +``` + +``` +/** +* If set false, writes the carbondata and carbonindex files in a flat folder structure +* @param isTransactionalTable is a boolelan value if set to false then writes +* the carbondata and carbonindex files in a flat folder structure +* @return updated CarbonWriterBuilder +*/ +public CarbonWriterBuilder isTransactionalTable(boolean isTransactionalTable); +``` + +``` +/** +* to set the timestamp in the carbondata and carbonindex index files +* @param UUID is a timestamp to be used in the carbondata +* and carbonindex index files +* @return updated CarbonWriterBuilder +*/ +public CarbonWriterBuilder uniqueIdentifier(long UUID); +``` + +``` +/** +* To set the carbondata file size in MB between 1MB-2048MB +* @param blockSize is size in MB between 1MB to 2048 MB +* @return updated CarbonWriterBuilder +*/ +public CarbonWriterBuilder withBlockSize(int blockSize); +``` + +``` +/** +* To set the blocklet size of carbondata file +* @param blockletSize is blocklet size in MB +* @return updated CarbonWriterBuilder +*/ +public CarbonWriterBuilder withBlockletSize(int blockletSize); +``` + +``` +/** +* sets the list of columns that needs to be in sorted order +* @param sortColumns is a string array of columns that needs to be sorted. +* If it is null, all dimensions are selected for sorting --- End diff -- What is the default value if not set. Default value if not set should be mentioned for all APIs --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2198 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5369/ --- |
In reply to this post by qiuchenjian-2
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2198 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4518/ --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2198 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4213/ --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2198 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5380/ --- |
In reply to this post by qiuchenjian-2
|
In reply to this post by qiuchenjian-2
Github user gvramana commented on the issue:
https://github.com/apache/carbondata/pull/2198 LGTM --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2198 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4226/ --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2198 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5393/ --- |
In reply to this post by qiuchenjian-2
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2198 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4532/ --- |
Free forum by Nabble | Edit this page |