[Discussion] Segment management enhance

[David CaiQiang]

[Background]
1. In some scenes, two loading/compaction jobs maybe write data to the same
segment, it will result in some data confusion and impact some features
which will not work fine again.  
2. Loading/compaction/update/delete operations need  to clean stale data
before execution. Cleaning stale data is a high-risk operation, if it has
some exception, it will delete valid data. If the system doesn't clean stale
data,   in some scenes, it will be added into a new merged index file and
can be queried.
3. Loading/compaction takes a long time and lock will keep a long time also
in some scenes.

[Motivation & Goal]
We should avoid data confusion and the risk of clean stale data. Maybe we
can use UUID as a segment id to avoid these troubles. Even if we can do
loading/compaction without the segment/compaction lock.

[Modification]
1. segment id
  Using UUID as segment id instead of the unique numeric value.

2. segment layout
 a) move segment data folder into the table folder
 b) move carbonindexmerge file into Metadata/segments folder,

 tableFolder
    UUID1
     |_xxx.carbondata
     |_xxx.carobnindex
    UUID2
    Metadata
     |_segemnts
        |_UUID1_timestamp1.segment (segment index summary)
        |_UUID1_timestamp1.carbonindexmerge (segment index detail)
     |_schema
     |_tablestatus
    LockFiles

  partitionTableFolder
    partkey=value1
     |_xxx.carbondata
     |_xxx.carobnindex
    partkey=value2
    Metadata
     |_segemnts
        |_UUID1_timestamp1.segment (segment index summary)
        |_partkey=value1
          |_UUID1_timestamp1.carbonindexmerge (segment index detail)
        |_partkey=value2
     |_schema
     |_tablestatus
    LockFiles

3. segment management
Extracting segment interface, it can support open/close, read/write, and
segment level index pruning API.
The segment should support multiple data source types: file format(carbon,
parquet, orc...), HBase...

4. clean stale data
it will become an optional operation.



-----
Best Regards
David Cai
--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/

[kunalkapoor]

Hi David,
I don't think changing the segment ID to UUID is a good idea, it will cause
usability issues.

1. Seeing a UUID named directory in the table structure would be weird, and
not informative.
2. The show segments command would also have the same problem.

Thanks
Kunal Kapoor

On Fri, Sep 4, 2020 at 8:38 AM David CaiQiang <[hidden email]> wrote:

> [Background]
> 1. In some scenes, two loading/compaction jobs maybe write data to the same
> segment, it will result in some data confusion and impact some features
> which will not work fine again.
> 2. Loading/compaction/update/delete operations need  to clean stale data
> before execution. Cleaning stale data is a high-risk operation, if it has
> some exception, it will delete valid data. If the system doesn't clean
> stale
> data,   in some scenes, it will be added into a new merged index file and
> can be queried.
> 3. Loading/compaction takes a long time and lock will keep a long time also
> in some scenes.
>
> [Motivation & Goal]
> We should avoid data confusion and the risk of clean stale data. Maybe we
> can use UUID as a segment id to avoid these troubles. Even if we can do
> loading/compaction without the segment/compaction lock.
>
> [Modification]
> 1. segment id
>   Using UUID as segment id instead of the unique numeric value.
>
> 2. segment layout
>  a) move segment data folder into the table folder
>  b) move carbonindexmerge file into Metadata/segments folder,
>
>  tableFolder
>     UUID1
>      |_xxx.carbondata
>      |_xxx.carobnindex
>     UUID2
>     Metadata
>      |_segemnts
>         |_UUID1_timestamp1.segment (segment index summary)
>         |_UUID1_timestamp1.carbonindexmerge (segment index detail)
>      |_schema
>      |_tablestatus
>     LockFiles
>
>   partitionTableFolder
>     partkey=value1
>      |_xxx.carbondata
>      |_xxx.carobnindex
>     partkey=value2
>     Metadata
>      |_segemnts
>         |_UUID1_timestamp1.segment (segment index summary)
>         |_partkey=value1
>           |_UUID1_timestamp1.carbonindexmerge (segment index detail)
>         |_partkey=value2
>      |_schema
>      |_tablestatus
>     LockFiles
>
> 3. segment management
> Extracting segment interface, it can support open/close, read/write, and
> segment level index pruning API.
> The segment should support multiple data source types: file format(carbon,
> parquet, orc...), HBase...
>
> 4. clean stale data
> it will become an optional operation.
>
>
>
> -----
> Best Regards
> David Cai
> --
> Sent from:
> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
>

[David CaiQiang]

Hi Kunal,

   1. The user uses SQL API or other interfaces. This UUID is a transaction
id, and we already stored the timestamp and other informations in the
segment metadata.
       This transaction id can be used in the loading/compaction/update
operation. We can append this id into the log if needed.
       Git commit id also uses UUID, so we can consider to use it. What
information do you want to get from the folder name?
       
   2. It is easy to fix the show segment command's issue. Maybe we can sort
segment by timestamp and UUID to generate the index id.  The user can
continue to use it in other commands.



-----
Best Regards
David Cai
--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/

[kunalkapoor]

Hi David,
Then better we keep a mapping for the segment UUID to virtual segment
number in the table status file as well,
Any API through which the user can get the segment details should return
the virtual segment id instead of the UUID.

On Fri, Sep 4, 2020 at 12:59 PM David CaiQiang <[hidden email]> wrote:

> Hi Kunal,
>
>    1. The user uses SQL API or other interfaces. This UUID is a transaction
> id, and we already stored the timestamp and other informations in the
> segment metadata.
>        This transaction id can be used in the loading/compaction/update
> operation. We can append this id into the log if needed.
>        Git commit id also uses UUID, so we can consider to use it. What
> information do you want to get from the folder name?
>
>    2. It is easy to fix the show segment command's issue. Maybe we can sort
> segment by timestamp and UUID to generate the index id.  The user can
> continue to use it in other commands.
>
>
>
> -----
> Best Regards
> David Cai
> --
> Sent from:
> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
>

[Ajantha Bhat]

Hi David,

a) Recently we tested huge concurrent load and compactions but never faced
two loads using same segment id issue (because of table status lock in
recordNewLoadMetadata), so I am not sure whether we really need to update
to UUID.

b) And about other segment interfaces, we have to refactor it. It is long
pending. Refactor such that we can support TIME TRAVEL. I have to analyze
more on this. If somebody has already done some analysis can use thread to
refactor the segment interface discussion.

Thanks,
Ajantha

On Fri, Sep 4, 2020 at 1:11 PM Kunal Kapoor <[hidden email]>
wrote:

> Hi David,
> Then better we keep a mapping for the segment UUID to virtual segment
> number in the table status file as well,
> Any API through which the user can get the segment details should return
> the virtual segment id instead of the UUID.
>
> On Fri, Sep 4, 2020 at 12:59 PM David CaiQiang <[hidden email]>
> wrote:
>
> > Hi Kunal,
> >
> >    1. The user uses SQL API or other interfaces. This UUID is a
> transaction
> > id, and we already stored the timestamp and other informations in the
> > segment metadata.
> >        This transaction id can be used in the loading/compaction/update
> > operation. We can append this id into the log if needed.
> >        Git commit id also uses UUID, so we can consider to use it. What
> > information do you want to get from the folder name?
> >
> >    2. It is easy to fix the show segment command's issue. Maybe we can
> sort
> > segment by timestamp and UUID to generate the index id.  The user can
> > continue to use it in other commands.
> >
> >
> >
> > -----
> > Best Regards
> > David Cai
> > --
> > Sent from:
> > http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
> >
>

[Venkata Gollamudi]

Hi David,

In the current design of data load operation, isolation, consistency is
achieved with following steps
1. DataLoad operation needs to check, if it can concurrently execute along
with any other operation like Update/delete( most. operations can be
allowed, even multiple parallel data loads).
2. Once operation is allowed, then lock is acquired at tablestatus file to
create a segment which it continues to load.
3. During load operation, time stamp(long) is used as transaction id for
the complete operation. This transaction id uniquely identifies the
operation.
4. When data load is complete the operation is committed to table status by
taking lock, read & update and release lock.
5. When data load fails then the operation is not committed and data can be
cleaned up during failure flow or in case of abrupt process failures, same
is cleaned up later.
6. Temporary data with the transaction id is never read or should be
discovered by any reader(Ex: never should be discovered using list files
etc and used) . All valid data should be read using only committed
transaction references.

This method of isolation and committing operation is applicable for all the
operations like dataload, insert, update or delete.
Segment ID does not have much significance in the above flow, though
sequence number is used currently, just for convenience.
Segment file locks to ensure the atomic commit is must and cannot be
avoided, even if we support complete optimistic concurrency control.

So segment ID replacing with UUID will not solve the concurrency or data
correctness or cleanup or stale data reading issues and also cannot replace
locking. There might be some other problem which you need to deep dive.
There might be code which is not following the above actions mentioned by
me (like discovering files using file listing and without filtering
required transaction id), which might be causing issues you have mentioned.

Regards,
Ramana

On Sat, Sep 5, 2020 at 8:31 AM Ajantha Bhat <[hidden email]> wrote:

> Hi David,
>
> a) Recently we tested huge concurrent load and compactions but never faced
> two loads using same segment id issue (because of table status lock in
> recordNewLoadMetadata), so I am not sure whether we really need to update
> to UUID.
>
> b) And about other segment interfaces, we have to refactor it. It is long
> pending. Refactor such that we can support TIME TRAVEL. I have to analyze
> more on this. If somebody has already done some analysis can use thread to
> refactor the segment interface discussion.
>
> Thanks,
> Ajantha
>
> On Fri, Sep 4, 2020 at 1:11 PM Kunal Kapoor <[hidden email]>
> wrote:
>
> > Hi David,
> > Then better we keep a mapping for the segment UUID to virtual segment
> > number in the table status file as well,
> > Any API through which the user can get the segment details should return
> > the virtual segment id instead of the UUID.
> >
> > On Fri, Sep 4, 2020 at 12:59 PM David CaiQiang <[hidden email]>
> > wrote:
> >
> > > Hi Kunal,
> > >
> > >    1. The user uses SQL API or other interfaces. This UUID is a
> > transaction
> > > id, and we already stored the timestamp and other informations in the
> > > segment metadata.
> > >        This transaction id can be used in the loading/compaction/update
> > > operation. We can append this id into the log if needed.
> > >        Git commit id also uses UUID, so we can consider to use it. What
> > > information do you want to get from the folder name?
> > >
> > >    2. It is easy to fix the show segment command's issue. Maybe we can
> > sort
> > > segment by timestamp and UUID to generate the index id.  The user can
> > > continue to use it in other commands.
> > >
> > >
> > >
> > > -----
> > > Best Regards
> > > David Cai
> > > --
> > > Sent from:
> > >
> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
> > >
> >
>

[David CaiQiang]

Hi Ramana,
   I agree with you.
   When  writing segment file, the system use listFiles to collect all index
files.
   In some case, it will add stale  index files into segment file.
   We will try to fix it at first.



-----
Best Regards
David Cai
--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/