GitHub user xuchuanyin opened a pull request:
https://github.com/apache/carbondata/pull/2416 [CARBONDATA-2660][BloomDataMap] Add test for querying on longstring bloom index column
Filtering on longstring bloom index column is already supported in PR #2403, here we only add test for it.
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
- [ ] Any interfaces changed?
- [ ] Any backward compatibility impacted?
- [ ] Document update required?
- [ ] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
You can merge this pull request into a Git repository by running:
$ git pull
https://github.com/xuchuanyin/carbondata 0627_bloom_filtering_longstring
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/2416.patchTo close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2416
----
commit 0cf11cd443cd4531180e0e674cc870dbed867fe3
Author: xuchuanyin <xuchuanyin@...>
Date: 2018-06-24T02:06:54Z
Fix bugs in bloomfilter for dictionary/sort/date column
In carbondata,
for dictionary column, carbon convert literal value to dict value, then
convert dict value to mdk value, at last it stores the mdk value as
internal value in carbonfile.
for other columns, carbon convert literal value to internal value using
field-converter.
Since bloomfilter datamap stores the internal value, during query we
should convert the literal value in filter to internal value in order to
match the value stored in bloomfilter datamap.
Changes are made:
1. FieldConverters were refactored to extract common value convert methods.
2. BloomQueryModel was optimized to support converting literal value to
internal value.
3. fix bugs for int/float/date/timestamp as bloom index column
4. fix bugs in dictionary/sort column as bloom index column
5. add tests
6. block (deferred) rebuild for bloom datamap (contains bugs that does
not fix in this commit)
commit 29640866150ccdd66f145259dfc19fd0d7672254
Author: xuchuanyin <xuchuanyin@...>
Date: 2018-06-25T06:02:40Z
fix bugs
commit ee9359f3ac336b18658beef961ca9e47e7b0d56c
Author: xuchuanyin <xuchuanyin@...>
Date: 2018-06-25T06:17:49Z
reduce data size for test
commit b910c94d924ac015bbcdae82e60808e2f41d0b46
Author: xuchuanyin <xuchuanyin@...>
Date: 2018-06-26T06:11:17Z
Fix bugs in negative number of skipped blocklets
Currently in carbondata, default blocklet datamap will be used to prune
blocklets. Then other indexdatamap will be used.
But the other index datamap works for segment scope, which in some
scenarios, the size of pruned result will be bigger than that of default
datamap, thus causing negative number of skipped blocklets in explain
query output.
Here we add intersection after pruning. If the pruned result size is
zero, we will finish the pruning.
commit 1b09402af8af95227f9af59f3c39eec4b3c5d493
Author: xuchuanyin <xuchuanyin@...>
Date: 2018-06-26T06:13:59Z
Optimize output for explaining querying with datamap
Currently if we have multiple datamaps and the query hits all the
datamaps, carbondata explain command will only show the first datamap
and all the datamaps are not shown. In this commit, we show all the
datamaps that are hitted in this query.
commit 46f03851c28975c5366853de9f86d9cad8ff3a4f
Author: xuchuanyin <xuchuanyin@...>
Date: 2018-06-26T11:53:15Z
Fix bugs in loading and querying on bloom column
Fix bugs in loading and querying with empty values on bloom index
columns. Convert null values to corresponding values.
commit 61168b5aa6864608070cc748c6cc3623df2f4c5e
Author: xuchuanyin <xuchuanyin@...>
Date: 2018-06-27T02:20:25Z
Add tests for querying on longstring bloom index column
add test to verify that bloom filter datamap support longstring column
----
---
Locked
