GitHub user xuchuanyin opened a pull request:
https://github.com/apache/carbondata/pull/2648 [CARBONDATA-2562][ExternalFormat] Support build file leve index for external format table
+ support directly generate file level index
+ support create and generate file index on existing data
+ We will flatten the input files recursively and remove the duplicated
input files in one load
The folder structure of the index file looks like below:
${datamap_name}/${segment_name}/File_level_${fact_file1_path_with_base64_encoding}/${column_name}.bloomindex
../File_level_${fact_file2_path_with_base64_encoding}/${column_name}.bloomindex
Note that in this commit, the index datamap is not used during query.
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
- [x] Any interfaces changed?
`Only internal used interfaces has been changed`
- [x] Any backward compatibility impacted?
`NO`
- [x] Document update required?
`NO`
- [x] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
`Added tests`
- How it is tested? Please attach test report.
`Tested in local machine`
- Is it a performance related change? Please attach the performance test report.
`NO`
- Any additional information to help reviewers in testing this change.
`NA`
- [x] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
`NA`
You can merge this pull request into a Git repository by running:
$ git pull
https://github.com/xuchuanyin/carbondata ef_index_dm_framework
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/2648.patchTo close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2648
----
commit 8c0e84804c266c56cada024384d9ab2eaa89e9f2
Author: xuchuanyin <xuchuanyin@...>
Date: 2018-08-20T01:38:12Z
Support build file leve index for external format table
+ support directly generate file level index
+ support create and generate file index on existing data
+ We will flatten the input files recursively and remove the duplicated
input files in one load
The folder structure of the index file looks like below:
${datamap_name}/${segment_name}/File_level_${fact_file1_path_with_base64_encoding}/${column_name}.bloomindex
../File_level_${fact_file2_path_with_base64_encoding}/${column_name}.bloomindex
Note that in this commit, the index datamap is not used during query.
----
---
Locked
