GitHub user manishgupta88 opened a pull request:
https://github.com/apache/carbondata/pull/2060 [CARBONDATA-2252] Query performance slows down as the number of columns increases in like query with OR expression
Problem: In case of OR condition with like query contains and ends with, the filter is getting pushed down to carbon layer because of which the query is slow as compared to spark applying the same filter on the results returned from carbon
Analysis: This is because in case of like query the execution is done by RowLevelFilterExecutorImpl which will compute the data row by row. As the number of columns will increase the computation time will increase thereby increasing the query time.
Fix: If there is any OR condition with like query, it is better to return back all the results to spark and let spark do the computation.
- [ ] Any interfaces changed?
No
- [ ] Any backward compatibility impacted?
No
- [ ] Document update required?
No
- [ ] Testing done
Added test cases
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
NA
You can merge this pull request into a Git repository by running:
$ git pull
https://github.com/manishgupta88/carbondata like_or_disable_pushdown
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/2060.patchTo close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2060
----
commit c2b1bbc2f01700eab37a65df9ab7bd995973efc6
Author: manishgupta88 <tomanishgupta18@...>
Date: 2018-03-13T11:15:48Z
Problem: In case of OR condition with like query contains and ends with, the filter is getting pushed down to carbon layer because of which the
query is slow as compared to spark applying the same filter on the results returned from carbon
Analysis: This is because in case of like query the execution is done by RowLevelFilterExecutorImpl which will compute the data row by row. As the
number of columns will increase the computation time will increase thereby increassing the query time.
Fix: If there is any OR condition with like query, it is better to return back all the results to spark and let spark do the computation.
----
---
Locked
