GitHub user ravipesala opened a pull request:
https://github.com/apache/carbondata/pull/1789 [WIP] Fix avoid reading of all block information in driver for old stores.
Problem:
For old stores prior to 1.2 version there is no blocklet information stored in carbonindex file. So the new code needs to read all carbondata files footers inside the driver to get the blocklet information. That makes the first time queries become slower. As observed count(*) query was taking 2 swconds on old version and after upgrade it takes very long time.
Solution:
If there is no information blocklet available in carbonindex file then don't read carbondata files footer in driver side. Instead read carbondata files in executor to get the blocklet information.
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
- [X] Any interfaces changed?
- [X] Any backward compatibility impacted?
- [X] Document update required?
- [X] Testing done
- [X] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
You can merge this pull request into a Git repository by running:
$ git pull
https://github.com/ravipesala/incubator-carbondata datamap-pld-store
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/1789.patchTo close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1789
----
commit f0aaf4d8d0e4761227bc1adff29d33798c88bd12
Author: ravipesala <ravi.pesala@...>
Date: 2018-01-10T15:35:48Z
Fix avoid reading of all block information in driver for old stores.
----
---
Locked
