Akash R Nilugal created CARBONDATA-3296:
-------------------------------------------
Summary: Support incremental dataload to datamap and other mv datamap enhancements
Key: CARBONDATA-3296
URL:
https://issues.apache.org/jira/browse/CARBONDATA-3296 Project: CarbonData
Issue Type: Bug
Reporter: Akash R Nilugal
Currently in carbondata we have datamaps like preaggregate, lucene, bloom,
mv and we have
lazy and non-lazy methods to load data to datamaps. But lazy load is not
allowed for datamaps
like preagg, lucene, bloom.but, it is allowed for mv datamap. In lazy load
of mv datamap, for
every rebuild(load to datamap) we load the complete data of main table and
overwrite the existing
segment in datamap based on datamap query.
This is very costly in terms of performance and we also need to support the
lazy and non-lazy load
for all the datamaps. This can help in reduce the actual dataload time to
main table and whenever
user wants, he can do the lazy load for the datamaps present for that table.
Basically we need not overwrite the existing data every time we load to
datamap, we need to increment
the data in new segments similar to main table. This will help to get
better performance.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Locked
