GitHub user xuchuanyin opened a pull request:
https://github.com/apache/carbondata/pull/2660 [CARBONDATA-2873][ExternalFormat] Support query external format using bloomfilter datamap
support querying external format table using bloomfilter datamap
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
- [ ] Any interfaces changed?
- [ ] Any backward compatibility impacted?
- [ ] Document update required?
- [ ] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
You can merge this pull request into a Git repository by running:
$ git pull
https://github.com/xuchuanyin/carbondata ef_index_dm_framework_query
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/2660.patchTo close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2660
----
commit 8c0e84804c266c56cada024384d9ab2eaa89e9f2
Author: xuchuanyin <xuchuanyin@...>
Date: 2018-08-20T01:38:12Z
Support build file leve index for external format table
+ support directly generate file level index
+ support create and generate file index on existing data
+ We will flatten the input files recursively and remove the duplicated
input files in one load
The folder structure of the index file looks like below:
${datamap_name}/${segment_name}/File_level_${fact_file1_path_with_base64_encoding}/${column_name}.bloomindex
../File_level_${fact_file2_path_with_base64_encoding}/${column_name}.bloomindex
Note that in this commit, the index datamap is not used during query.
commit 30a861a92ff53df8befd14ee48b7a37499ab7c96
Author: xuchuanyin <xuchuanyin@...>
Date: 2018-08-25T10:49:43Z
Support query external format using bloomfilter datamaps
support query external format using bloomfilter datamap
----
---
Locked
