[
https://issues.apache.org/jira/browse/CARBONDATA-1617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jacky Li resolved CARBONDATA-1617.
----------------------------------
Resolution: Fixed
Fix Version/s: 1.3.0
> Merging carbonindex files for each segment.
> -------------------------------------------
>
> Key: CARBONDATA-1617
> URL:
https://issues.apache.org/jira/browse/CARBONDATA-1617> Project: CarbonData
> Issue Type: New Feature
> Reporter: Ravindra Pesala
> Priority: Major
> Fix For: 1.3.0
>
> Time Spent: 10h 20m
> Remaining Estimate: 0h
>
> Hi,
> Problem :
> The first-time query of carbon becomes very slow. It is because of reading many small carbonindex files and cache to the driver at the first time.
> Many carbonindex files are created in below case
> Loading data in large cluster
> For example, if the cluster size is 100 nodes then for each load 100 index files are created per segment. So after 100 loads, the number of carbonindex files becomes 10000. .
> It will be slower to read all the files from the driver since a lot of namenode calls and IO operations.
> Solution :
> Merge the carbonindex files in two levels.so that we can reduce the IO calls to namenode and improves the read performance.
> Merge within a segment.
> Merge the carbonindex files to single file immediately after load completes within the segment. It would be named as a .carbonindexmerge file. It is actually not a true data merging but a simple file merge. So that the current structure of carbonindex files does not change. While reading we just read one file instead of many carbonindex files within the segment.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
Locked
