Re: [Design Discussion] Transaction manager, time travel and segment interface refactoring

Posted by kumarvishal09 on May 03, 2021; 4:40am
URL: http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/Design-Discussion-Transaction-manager-time-travel-and-segment-interface-refactoring-tp107770p108011.html

+1

Small Modification: In current design only the table status file is
getting updated. In this new design tx<Id>_tablestatus[InProgress->Success]
and <db>.<tbl>tx files are getting updated.
I think we can write tx_<Id>_InProgress and tx_<id>_Success to a new file.
and After successful load we can delete the InProgress table status file.

-Regards
Kumar Vishal

On Sun, 2 May 2021 at 10:07 PM, Venkata Gollamudi <[hidden email]>
wrote:

> +1
>
> Valuable feature for data consistency across tables, mvs, indexes.
>
> On Wed, Apr 28, 2021 at 7:02 PM Ravindra Pesala <[hidden email]>
> wrote:
>
> > +1
> >
> > Much needed feature and interface refactoring. Thanks for working on it.
> >
> > Regards,
> > Ravindra.
> >
> > On Thu, 22 Apr 2021 at 2:36 PM, Ajantha Bhat <[hidden email]>
> > wrote:
> >
> > > Hi All,
> > > In this thread, I am continuing the below discussion along with the
> > > Transaction Manager and Time Travel feature design.
> > >
> > >
> >
> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/Discussion-Taking-the-inputs-for-Segment-Interface-Refactoring-td101950.html
> > >
> > > The goal of this requirement is as follows.
> > >
> > >    1.
> > >
> > >    Implement a “Transaction Manager” with optimistic concurrency to
> > provide
> > >    within a table transaction/versioning. (interfaces should also be
> > >    flexible enough to support across table transactions)
> > >    2.
> > >
> > >    Support time travel in carbonData.
> > >    3.
> > >
> > >    Decouple and clean up segment interfaces. (which should also help in
> > >    supporting segment concepts to other open formats under carbonData
> > > metadata
> > >    service)
> > >
> > >
> > > The design document is attached in JIRA.
> > > JIRA link: https://issues.apache.org/jira/browse/CARBONDATA-4171
> > > GoogleDrive link:
> > >
> > >
> >
> https://docs.google.com/document/d/1FsVsXjj5QCuFDrzrayN4Qo0LqWc0Kcijc_jL7pCzfXo/edit?usp=sharing
> > >
> > > Please have a look. suggestions are welcome.
> > > I have mentioned some TODO in the document, I will be updating it in
> the
> > V2
> > > version soon.
> > > Implementation will be done by adding subtasks under the same JIRA.
> > >
> > > Thanks,
> > > Ajantha
> > >
> > --
> > Thanks & Regards,
> > Ravi
> >
>