Login  Register

[Design Discussion] Transaction manager, time travel and segment interface refactoring

Posted by Ajantha Bhat on Apr 22, 2021; 6:36am
URL: http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/Design-Discussion-Transaction-manager-time-travel-and-segment-interface-refactoring-tp107770.html

Hi All,
In this thread, I am continuing the below discussion along with the
Transaction Manager and Time Travel feature design.
http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/Discussion-Taking-the-inputs-for-Segment-Interface-Refactoring-td101950.html

The goal of this requirement is as follows.

   1.

   Implement a “Transaction Manager” with optimistic concurrency to provide
   within a table transaction/versioning. (interfaces should also be
   flexible enough to support across table transactions)
   2.

   Support time travel in carbonData.
   3.

   Decouple and clean up segment interfaces. (which should also help in
   supporting segment concepts to other open formats under carbonData metadata
   service)


The design document is attached in JIRA.
JIRA link: https://issues.apache.org/jira/browse/CARBONDATA-4171
GoogleDrive link:
https://docs.google.com/document/d/1FsVsXjj5QCuFDrzrayN4Qo0LqWc0Kcijc_jL7pCzfXo/edit?usp=sharing

Please have a look. suggestions are welcome.
I have mentioned some TODO in the document, I will be updating it in the V2
version soon.
Implementation will be done by adding subtasks under the same JIRA.

Thanks,
Ajantha