[Design Discussion] Transaction manager, time travel and segment interface refactoring

[Ajantha Bhat]

Hi All,
In this thread, I am continuing the below discussion along with the
Transaction Manager and Time Travel feature design.
http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/Discussion-Taking-the-inputs-for-Segment-Interface-Refactoring-td101950.html

The goal of this requirement is as follows.

   1.

   Implement a “Transaction Manager” with optimistic concurrency to provide
   within a table transaction/versioning. (interfaces should also be
   flexible enough to support across table transactions)
   2.

   Support time travel in carbonData.
   3.

   Decouple and clean up segment interfaces. (which should also help in
   supporting segment concepts to other open formats under carbonData metadata
   service)


The design document is attached in JIRA.
JIRA link: https://issues.apache.org/jira/browse/CARBONDATA-4171
GoogleDrive link:
https://docs.google.com/document/d/1FsVsXjj5QCuFDrzrayN4Qo0LqWc0Kcijc_jL7pCzfXo/edit?usp=sharing

Please have a look. suggestions are welcome.
I have mentioned some TODO in the document, I will be updating it in the V2
version soon.
Implementation will be done by adding subtasks under the same JIRA.

Thanks,
Ajantha

[ravipesala]

+1

Much needed feature and interface refactoring. Thanks for working on it.

Regards,
Ravindra.

On Thu, 22 Apr 2021 at 2:36 PM, Ajantha Bhat <[hidden email]> wrote:

> Hi All,
> In this thread, I am continuing the below discussion along with the
> Transaction Manager and Time Travel feature design.
>
> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/Discussion-Taking-the-inputs-for-Segment-Interface-Refactoring-td101950.html
>
> The goal of this requirement is as follows.
>
>    1.
>
>    Implement a “Transaction Manager” with optimistic concurrency to provide
>    within a table transaction/versioning. (interfaces should also be
>    flexible enough to support across table transactions)
>    2.
>
>    Support time travel in carbonData.
>    3.
>
>    Decouple and clean up segment interfaces. (which should also help in
>    supporting segment concepts to other open formats under carbonData
> metadata
>    service)
>
>
> The design document is attached in JIRA.
> JIRA link: https://issues.apache.org/jira/browse/CARBONDATA-4171
> GoogleDrive link:
>
> https://docs.google.com/document/d/1FsVsXjj5QCuFDrzrayN4Qo0LqWc0Kcijc_jL7pCzfXo/edit?usp=sharing
>
> Please have a look. suggestions are welcome.
> I have mentioned some TODO in the document, I will be updating it in the V2
> version soon.
> Implementation will be done by adding subtasks under the same JIRA.
>
> Thanks,
> Ajantha
>

--
Thanks & Regards,
Ravi

[Venkata Gollamudi]

+1

Valuable feature for data consistency across tables, mvs, indexes.

On Wed, Apr 28, 2021 at 7:02 PM Ravindra Pesala <[hidden email]>
wrote:

> +1
>
> Much needed feature and interface refactoring. Thanks for working on it.
>
> Regards,
> Ravindra.
>
> On Thu, 22 Apr 2021 at 2:36 PM, Ajantha Bhat <[hidden email]>
> wrote:
>
> > Hi All,
> > In this thread, I am continuing the below discussion along with the
> > Transaction Manager and Time Travel feature design.
> >
> >
> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/Discussion-Taking-the-inputs-for-Segment-Interface-Refactoring-td101950.html
> >
> > The goal of this requirement is as follows.
> >
> >    1.
> >
> >    Implement a “Transaction Manager” with optimistic concurrency to
> provide
> >    within a table transaction/versioning. (interfaces should also be
> >    flexible enough to support across table transactions)
> >    2.
> >
> >    Support time travel in carbonData.
> >    3.
> >
> >    Decouple and clean up segment interfaces. (which should also help in
> >    supporting segment concepts to other open formats under carbonData
> > metadata
> >    service)
> >
> >
> > The design document is attached in JIRA.
> > JIRA link: https://issues.apache.org/jira/browse/CARBONDATA-4171
> > GoogleDrive link:
> >
> >
> https://docs.google.com/document/d/1FsVsXjj5QCuFDrzrayN4Qo0LqWc0Kcijc_jL7pCzfXo/edit?usp=sharing
> >
> > Please have a look. suggestions are welcome.
> > I have mentioned some TODO in the document, I will be updating it in the
> V2
> > version soon.
> > Implementation will be done by adding subtasks under the same JIRA.
> >
> > Thanks,
> > Ajantha
> >
> --
> Thanks & Regards,
> Ravi
>

[kumarvishal09]

+1

Small Modification: In current design only the table status file is
getting updated. In this new design tx<Id>_tablestatus[InProgress->Success]
and <db>.<tbl>tx files are getting updated.
I think we can write tx_<Id>_InProgress and tx_<id>_Success to a new file.
and After successful load we can delete the InProgress table status file.

-Regards
Kumar Vishal

On Sun, 2 May 2021 at 10:07 PM, Venkata Gollamudi <[hidden email]>
wrote:

> +1
>
> Valuable feature for data consistency across tables, mvs, indexes.
>
> On Wed, Apr 28, 2021 at 7:02 PM Ravindra Pesala <[hidden email]>
> wrote:
>
> > +1
> >
> > Much needed feature and interface refactoring. Thanks for working on it.
> >
> > Regards,
> > Ravindra.
> >
> > On Thu, 22 Apr 2021 at 2:36 PM, Ajantha Bhat <[hidden email]>
> > wrote:
> >
> > > Hi All,
> > > In this thread, I am continuing the below discussion along with the
> > > Transaction Manager and Time Travel feature design.
> > >
> > >
> >
> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/Discussion-Taking-the-inputs-for-Segment-Interface-Refactoring-td101950.html
> > >
> > > The goal of this requirement is as follows.
> > >
> > >    1.
> > >
> > >    Implement a “Transaction Manager” with optimistic concurrency to
> > provide
> > >    within a table transaction/versioning. (interfaces should also be
> > >    flexible enough to support across table transactions)
> > >    2.
> > >
> > >    Support time travel in carbonData.
> > >    3.
> > >
> > >    Decouple and clean up segment interfaces. (which should also help in
> > >    supporting segment concepts to other open formats under carbonData
> > > metadata
> > >    service)
> > >
> > >
> > > The design document is attached in JIRA.
> > > JIRA link: https://issues.apache.org/jira/browse/CARBONDATA-4171
> > > GoogleDrive link:
> > >
> > >
> >
> https://docs.google.com/document/d/1FsVsXjj5QCuFDrzrayN4Qo0LqWc0Kcijc_jL7pCzfXo/edit?usp=sharing
> > >
> > > Please have a look. suggestions are welcome.
> > > I have mentioned some TODO in the document, I will be updating it in
> the
> > V2
> > > version soon.
> > > Implementation will be done by adding subtasks under the same JIRA.
> > >
> > > Thanks,
> > > Ajantha
> > >
> > --
> > Thanks & Regards,
> > Ravi
> >
>