[DISCUSSION]Improve Simple updates and delete performance in carbondata

Hi Community,

Carbondata supports update and delete using spark. So basically update is
delete + Insert, and delete is just delete
But we use spark APIs or actions on collections that use spark jobs to do
them, like map, partition etc
So Spark adds overhead of task serialization cost, total job execution in
remote nodes, shuffle etc
So even just for simple updates, Carbon takes a lot of time, and the same
for delete as well due to these overheads.

Carbondata 2.1.0 supports update and delete for SDK. This is implemented at
the carbon file format level

so we can reuse the same for simple updates and deletes and avoid spark
completely and can perform simple update

and delete on transactional tables using simple java code. This helps to
avoid all the overhead of spark and make

updates and deletes faster.

I have added an initial V1 design document, please check and give
comments/inputs/suggestions.

https://docs.google.com/document/d/1-M6xPKZG8l6yAu0c9qo3jdUKhpXHWgUR-h8HeUUmk8M/edit?usp=sharing

Thanks,

Regards,
Akash R Nilugal

[David CaiQiang]

hi Akash, for the simple update case, can you do a test to confirm your
inference after a fast change?



-----
Best Regards
David Cai
--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/

[David CaiQiang]

Hi akash,

  for simple updates and delete scenario,  you can try to do it.

  During update/delete,
  1) for updated/deleted segment, no need to update segmentMetadataInfo.
  2) for new inserted segment, you can summary blocklet level index to
segment level index, reading carbonindex/carbonindexmerge file and calculate
it.



-----
Best Regards
David Cai
--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/

[ravipesala]

+1
I am looking forward to this feature as most of the update/delete
operations are simple and it can simplify and improve the performance as
well.
Thank you.

On Thu, 19 Nov 2020 at 19:41, Akash Nilugal <[hidden email]> wrote:

> Hi Community,
>
> Carbondata supports update and delete using spark. So basically update is
> delete + Insert, and delete is just delete
> But we use spark APIs or actions on collections that use spark jobs to do
> them, like map, partition etc
> So Spark adds overhead of task serialization cost, total job execution in
> remote nodes, shuffle etc
> So even just for simple updates, Carbon takes a lot of time, and the same
> for delete as well due to these overheads.
>
> Carbondata 2.1.0 supports update and delete for SDK. This is implemented at
> the carbon file format level
>
> so we can reuse the same for simple updates and deletes and avoid spark
> completely and can perform simple update
>
> and delete on transactional tables using simple java code. This helps to
> avoid all the overhead of spark and make
>
> updates and deletes faster.
>
> I have added an initial V1 design document, please check and give
> comments/inputs/suggestions.
>
>
> https://docs.google.com/document/d/1-M6xPKZG8l6yAu0c9qo3jdUKhpXHWgUR-h8HeUUmk8M/edit?usp=sharing
>
> Thanks,
>
> Regards,
> Akash R Nilugal
>

--
Thanks & Regards,
Ravi

[kumarvishal09]

+1
Regards
Kumar Vishal

On Thu, 10 Dec 2020 at 11:10 PM, Ravindra Pesala <[hidden email]>
wrote:

> +1
> I am looking forward to this feature as most of the update/delete
> operations are simple and it can simplify and improve the performance as
> well.
> Thank you.
>
> On Thu, 19 Nov 2020 at 19:41, Akash Nilugal <[hidden email]>
> wrote:
>
> > Hi Community,
> >
> > Carbondata supports update and delete using spark. So basically update is
> > delete + Insert, and delete is just delete
> > But we use spark APIs or actions on collections that use spark jobs to do
> > them, like map, partition etc
> > So Spark adds overhead of task serialization cost, total job execution in
> > remote nodes, shuffle etc
> > So even just for simple updates, Carbon takes a lot of time, and the same
> > for delete as well due to these overheads.
> >
> > Carbondata 2.1.0 supports update and delete for SDK. This is implemented
> at
> > the carbon file format level
> >
> > so we can reuse the same for simple updates and deletes and avoid spark
> > completely and can perform simple update
> >
> > and delete on transactional tables using simple java code. This helps to
> > avoid all the overhead of spark and make
> >
> > updates and deletes faster.
> >
> > I have added an initial V1 design document, please check and give
> > comments/inputs/suggestions.
> >
> >
> >
> https://docs.google.com/document/d/1-M6xPKZG8l6yAu0c9qo3jdUKhpXHWgUR-h8HeUUmk8M/edit?usp=sharing
> >
> > Thanks,
> >
> > Regards,
> > Akash R Nilugal
> >
>
>
> --
> Thanks & Regards,
> Ravi
>