Posted by
Jacky Li on
Nov 15, 2016; 9:39am
URL: http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/Feature-proposal-for-update-and-delete-support-in-Carbon-data-tp2906p2911.html
Hi Vinod,
It is great to have this feature, as there were many people asking for data update during the CarbonData meetup earlier. I believe it will be useful for many big data applications.
For the solution you proposed, I have following doubts:
1. Data update is complex as if transaction is involved, so what kind of ACID level support are you thinking about?
2. If I understand correctly, you are proposing to do data update via base + delta file approach, right? So in this case, new file format needs to be added in CarbonData project.
3. As CarbonData has builtin support for index, any idea what is the impaction to the B tree index already in driver and executor memory?
Regards,
Jacky
> 在 2016年11月15日，下午12:25，Vinod KC <
[hidden email]> 写道：
>
> Hi All
> I would like to propose following new features in Carbon data
> 1) Update statement to support modifying existing records in carbon data
> table
> 2) Delete statement to remove records from carbon data table
>
> A) Update operation: 'Update' features can be added to CarbonData using
> intermediate Delta files [delete/update delta files] support with lesser
> impact on existing code.
> Update can be considered as a ‘delete’ followed by an‘insert’ operation.
> Once an update is done on carbon data file, on select query operation,
> Carbondata store reader can make use of delete delta data cache to exclude
> deleted records in that segment and then include records from newly added
> update delta files.
>
> B) Delete operation: In the case of delete operation, a delete delta file
> will be added to each segment matching the records. During select query
> operation Carbon data reader will exclude those deleted records from the
> result set.
>
> Please share your suggestions and thoughts about design and functional
> aspects on this feature. I’ll share a detailed design document about above
> thoughts later.
>
> Regards
> Vinod