RE: Re:[DISCUSSION] Support Incremental load in datamap and other MV datamap enhancement

Posted by ravipesala on Feb 19, 2019; 8:07am
URL: http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/DISCUSSION-Support-Incremental-load-in-datamap-and-other-MV-datamap-enhancement-tp75160p75231.html

Hi Akash,

There is a  difference between index datamap (like bloom) and olap datamaps
(like MV). Index datamaps used only for pruning the data while olap datamaps
will be used as pre-computed data which can be fetched directly as per
query.

In OLAP datamap case lazy build or deferred build makes sense as data needs
to be always synchronized with master data otherwise we will get stale data.
So any difference in synchronization will make the datamap disable. But on
the other hand Index datamap  is used only for faster pruning so
synchronization with master data is not mandatory unless we have a mechanism
to prune synchronized data using index datamap and non-synchronized data
using default datamap. This is the same @xuchuanyin mentioned.

I feel this design is about OLAP datamap incremental loading so better not
do any changes in the behaviour of index datamaps. We can consider the
improvements of Index datamap in future but it should not be part of it.
Please update the design if mentioned anything related to Index datamap.

Regards,
Ravindra.



--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/