Re: [DISCUSS] Support transactional table in SDK
Posted by
Jacky Li on
Dec 10, 2018; 2:53pm
URL: http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/DISCUSS-Support-transactional-table-in-SDK-tp69878p70100.html
> 在 2018年12月7日，下午11:05，ravipesala <
[hidden email]> 写道：
>
> Hi Jacky,
>
> Its a good idea to support writing transactional table from SDK. But we need
> to add following limitations as well
> 1. It can work on file systems which can take append lock like HDFS.
Likun: yes, since we need to overwrite table status file, we need file locking.
> 2. Compaction, delete segment cannot be done on online segments till it is
> converted to the transactional segment.
Likun: Compaction and other data management work will still be done by CarbonSession application in standard spark cluster.
> 3. SDK writer should be responsible to add complete carbondata file to
> online segment once the writing is done, it should not add any half cooked
> data.
Likun: yes, in the design doc, I have mentioned this