[
https://issues.apache.org/jira/browse/CARBONDATA-3296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16833601#comment-16833601 ]
Akash R Nilugal commented on CARBONDATA-3296:
---------------------------------------------
Design document is updated at
https://docs.google.com/document/d/1AACOYmBpwwNdHjJLOub0utSc6JCBMZn8VL5CvZ9hygA/edit> Support incremental dataload to datamap and other mv datamap enhancements
> -------------------------------------------------------------------------
>
> Key: CARBONDATA-3296
> URL:
https://issues.apache.org/jira/browse/CARBONDATA-3296> Project: CarbonData
> Issue Type: Bug
> Reporter: Akash R Nilugal
> Priority: Major
>
> Currently in carbondata we have datamaps like preaggregate, lucene, bloom,
> mv and we have
> lazy and non-lazy methods to load data to datamaps. But lazy load is not
> allowed for datamaps
> like preagg, lucene, bloom.but, it is allowed for mv datamap. In lazy load
> of mv datamap, for
> every rebuild(load to datamap) we load the complete data of main table and
> overwrite the existing
> segment in datamap based on datamap query.
> This is very costly in terms of performance and we also need to support the
> lazy and non-lazy load
> for all the datamaps. This can help in reduce the actual dataload time to
> main table and whenever
> user wants, he can do the lazy load for the datamaps present for that table.
> Basically we need not overwrite the existing data every time we load to
> datamap, we need to increment
> the data in new segments similar to main table. This will help to get
> better performance.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Locked
