Re: Improve carbondata CDC performance

Posted by akashrn5 on Mar 31, 2021; 7:32am
URL: http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/Improve-carbondata-CDC-performance-tp106093p107298.html

Hi Ajantha,

Thanks for your points.

Now actually we cache the splits, actual join will be faster and, even
though the pruning doesn't happen it wont affect the performance much. This
is learned from the test we did during POC and it doesn't make much
difference in performance, basically no degrade as such.

And having said that, in actual use cases or I can say customer scenario,
when you have huge target table in data warehouse or data lake for
analytics, the CDC will be on small portion of data itself. Changing the
whole table is rarest scenario, so it should be fine.

Thanks

Regards,
Akash R



--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/