ZHANGSHUNYU created CARBONDATA-3770:
---------------------------------------
Summary: improve partition count star query performance by reading from valid index files directly
Key: CARBONDATA-3770
URL:
https://issues.apache.org/jira/browse/CARBONDATA-3770 Project: CarbonData
Issue Type: Improvement
Reporter: ZHANGSHUNYU
Problem:
# currently the count(*) with filter whose culumns are all partition columns will load datamaps of these partitions including block info/minmax info, but it is no need to load them ,we can just read it from valid index files directly as the rowCount stored inside, and cache these info.
# For no-sort partition table, minmax is almost no using but cost time.
Solutions:
The detail of query flow as following if it is pure partition count star
Step 1. check whether it is pure partition count star by filter
Step 2. read tablestatus to get all valid segments, remove the segment file cache of invalid segment and expired segment
Step 3. use multi-thread to read segment files which not in cache and cache index files list of each segment into memory. If its index files already is eixst in cache, not require to read again.
Step 4. use multi-thread to prune segment and partition to get pruned index file list, which can prune most index files and reduce the files num.
Step 5. read the count from pruned index file directly and cache it, get from cache if exist in the index_file <-> rowCount map.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Locked
