RE: Re:[DISCUSSION] Support Incremental load in datamap and other MV datamap enhancement

Posted by akashrn5 on Feb 19, 2019; 5:59am
URL: http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/DISCUSSION-Support-Incremental-load-in-datamap-and-other-MV-datamap-enhancement-tp75160p75227.html

I agree with you that the index created for old segments will be of no use if
rebuild is not happened and these are not considered in query for pruning.

But we go for datamap pruning (index) based on the datamap status. Status
will be just enabled or disabled. You cannot maintain status for each
segment, there is only one status. So based on this we go for pruning.

Currently we have deferred build, which is blocked for index datamaps,
preagg, BUT for MV datamap it is always considered as true. Everytime u need
to make rebuild after main table laod to enable the mv datamap. So basically
if u see, deferred is blocked for index and does not matter for MV as we
always have rebuild.

So are you telling to make everything as non-lazy? including MV?, this may
hit dataload performance. If you want to implement whatever u suggested,
need to maintain status for each segment whether for that segment it is
enable or not, which i doubt we shouldn't do.



--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/