What is the transaction ability of CarbonData? Does it support the transaction like this.

[李书明]

Hi community!

I found CarbonData provides some transaction capabilities, However I have some questions about the detail transaction ability which I cannot find in the document.

There is a  problem like this:
  SessionA: insert overwrite tableA partition(dt=A) select * from x;
  SessionB: select * from tableA

For a common `Spark`(parquet) table, this situation may cause `FileNotFoundException` because `overwrite` op drops partitoin `dt=A` when SessionB is scanning this file.

However I tested `insert overwrite` of CarbonData table, it still dropped the tableA partition(dt=A).

I don’t know whether `CarbonData` source can solve this problem like this, the expected result of SessionB could be the data before `overwrite` or the complete data after `overwrite` instead of `FileNotFoundException`?

[ravipesala]

Hi ,

 Yes, you are right. Carbon supports the way you expected.  It can either
give the data before overwrite or after overwrite in another session when
you run query concurrently. It never gives `FileNotFoundException`.

Regards,
Ravindra.

On Mon, 17 Feb 2020, 20:14 李书明, <[hidden email]> wrote:

> Hi community!
>
> I found CarbonData provides some transaction capabilities, However I have
> some questions about the detail transaction ability which I cannot find in
> the document.
>
> There is a  problem like this:
>   SessionA: insert overwrite tableA partition(dt=A) select * from x;
>   SessionB: select * from tableA
>
> For a common `Spark`(parquet) table, this situation may cause
> `FileNotFoundException` because `overwrite` op drops partitoin `dt=A` when
> SessionB is scanning this file.
>
> However I tested `insert overwrite` of CarbonData table, it still dropped
> the tableA partition(dt=A).
>
> I don’t know whether `CarbonData` source can solve this problem like this,
> the expected result of SessionB could be the data before `overwrite` or the
> complete data after `overwrite` instead of `FileNotFoundException`?
>
>
>
>
>
>
>

[李书明]

Thanks for your replies.

Can you explain how CanbonData can do like this even if `Overwrite` may delete the whole subDir.

How does this session to check another session maybe reading this subDir?

> 在 2020年2月17日，下午8:44，Ravindra Pesala <[hidden email]> 写道：
>
> Hi ,
>
> Yes, you are right. Carbon supports the way you expected.  It can either
> give the data before overwrite or after overwrite in another session when
> you run query concurrently. It never gives `FileNotFoundException`.
>
> Regards,
> Ravindra.