[DISCUSSION] Support write Flink streaming data to Carbon

Posted by niuge on
URL: http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/DISCUSSION-Support-write-Flink-streaming-data-to-Carbon-tp85670.html

The write process is:

1.Write flink streaming data to local file system of flink task node use flink StreamingFileSink and carbon SDK;
2.Copy local carbon data file to carbon data store system, such as HDFS, S3;
3.Generate and write segment file to ${tablePath}/load_details;

Run "alter table ${tableName} collect segments" command on server, to compact segment files in ${tablePath}/load_details, and then move the compacted segment file to ${tablePath}/Metadata/Segments/，update table status file finally.

Have raised a jira https://issues.apache.org/jira/browse/CARBONDATA-3557 and attached design document to it. Request you to please have a look.

Welcome you opinion and suggestions.