Re: Carbondata support flink feature
Posted by
Jonathan.Wei on
Nov 27, 2018; 2:14am
URL: http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/DISCUSSION-Carbondata-support-flink-feature-tp69148p69168.html
Hi,guys!
I think it should include two aspects. Write and read.
1. Write data from Flink to carbondata
2. Extract data from carbondata and write to Flink table.
If you only think about how to write carbondata, you'll just need to
implement write
Flink soure and sink need to be customized, and carbondata API
implementation needs to be called.
http://carbondata.apache.org/sdk-guide.htmlFlink customized the reference links of source and sink as follows:
https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sourceSinks.htmlType mapping of flink table and carbondata also needs to be considered.
Flink table data type:
https://ci.apache.org/projects/flink/flink-docs-master/dev/table/tableApi.html#data-typescarbondata
http://carbondata.apache.org/supported-data-types-in-carbondata.htmlTable format support
Carbondata table format: CSV, json, text, parquet, kafka, socket
http://carbondata.apache.org/streaming-guide.htmlFlink table format: CSV,json,avro
https://ci.apache.org/projects/flink/flink-docs-master/dev/table/connect.html#table-formats--
Sent from:
http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/