GitHub user xuchuanyin opened a pull request:
https://github.com/apache/carbondata/pull/2581 [CARBONDATA-2800][Doc] Add useful tips about bloomfilter datamap add useful tips about bloomfilter datamap Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [x] Any interfaces changed? `NO` - [x] Any backward compatibility impacted? `NO` - [x] Document update required? `Yes, updated the document` - [x] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? `NA` - How it is tested? Please attach test report. `NA` - Is it a performance related change? Please attach the performance test report. `NA` - Any additional information to help reviewers in testing this change. `NA` - [x] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/xuchuanyin/carbondata 0730_useful_tips Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2581.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2581 ---- commit f8f15369ac7ae5420e5e05d18214ecd0978eb0ad Author: xuchuanyin <xuchuanyin@...> Date: 2018-07-30T12:32:05Z add useful tips about bloomfilter datamap add useful tips about bloomfilter datamap ---- --- |
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2581 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7620/ --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2581 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6355/ --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2581 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7649/ --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2581 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6375/ --- |
In reply to this post by qiuchenjian-2
Github user chenliang613 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2581#discussion_r206433538 --- Diff: docs/useful-tips-on-carbondata.md --- @@ -125,6 +125,10 @@ TBLPROPERTIES ('SORT_COLUMNS'='Dime_1, HOST, MSISDN') ``` + **NOTE:** + + BloomFilter can be created to enhance performance for queries with precise equal/in conditions. You can find more information about it in BloomFilter datamap document. + --- End diff -- please add one link to bloomfilter datamap document. --- |
In reply to this post by qiuchenjian-2
Github user chenliang613 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2581#discussion_r206434059 --- Diff: docs/datamap/bloomfilter-datamap-guide.md --- @@ -103,3 +104,24 @@ If the datamap does not prune blocklets well, you can try to increase the value ## Data Management With BloomFilter DataMap Data management with BloomFilter datamap has no difference with that on Lucene datamap. You can refer to the corresponding section in `CarbonData BloomFilter DataMap`. + +## Useful Tips ++ BloomFilter DataMap is suggested to create on the high cardinality columns. ++ BloomFilter datamap requires that the query conditions on index columns are always simple `equal` or `in`, + such as 'col1=XX', 'col1 in (XX, YY)'. Otherwise the queries cannot benefit from BloomFilter datamap. ++ We can create multiple BloomFilter datamaps on one table, + also we can create one BloomFilter datamap that contains multiple index columns. + We do recommend the later behavior since the data loading and query performance will be better. ++ `BLOOM_FPP` is only the expected number from user, the actually FPP may be worse. + If the BloomFilter datamap does not work well, + you can try to increase `BLOOM_SIZE` and decrease `BLOOM_FPP` at the same time. + Notice that bigger `BLOOM_SIZE` will increase the size of index file + and smaller `BLOOM_FPP` will increase runtime calculation while performing query. ++ '0' skipped blocklets of BloomFilter datamap in explain output indicates that + BloomFilter datamap does not prune better than Main datamap. --- End diff -- BloomFilter datamap does not prune better than Main datamap. -- can you provide more detail about this point. --- |
In reply to this post by qiuchenjian-2
Github user xuchuanyin commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2581#discussion_r206452901 --- Diff: docs/useful-tips-on-carbondata.md --- @@ -125,6 +125,10 @@ TBLPROPERTIES ('SORT_COLUMNS'='Dime_1, HOST, MSISDN') ``` + **NOTE:** + + BloomFilter can be created to enhance performance for queries with precise equal/in conditions. You can find more information about it in BloomFilter datamap document. + --- End diff -- OK --- |
In reply to this post by qiuchenjian-2
Github user xuchuanyin commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2581#discussion_r206454747 --- Diff: docs/datamap/bloomfilter-datamap-guide.md --- @@ -103,3 +104,24 @@ If the datamap does not prune blocklets well, you can try to increase the value ## Data Management With BloomFilter DataMap Data management with BloomFilter datamap has no difference with that on Lucene datamap. You can refer to the corresponding section in `CarbonData BloomFilter DataMap`. + +## Useful Tips ++ BloomFilter DataMap is suggested to create on the high cardinality columns. ++ BloomFilter datamap requires that the query conditions on index columns are always simple `equal` or `in`, + such as 'col1=XX', 'col1 in (XX, YY)'. Otherwise the queries cannot benefit from BloomFilter datamap. ++ We can create multiple BloomFilter datamaps on one table, + also we can create one BloomFilter datamap that contains multiple index columns. + We do recommend the later behavior since the data loading and query performance will be better. ++ `BLOOM_FPP` is only the expected number from user, the actually FPP may be worse. + If the BloomFilter datamap does not work well, + you can try to increase `BLOOM_SIZE` and decrease `BLOOM_FPP` at the same time. + Notice that bigger `BLOOM_SIZE` will increase the size of index file + and smaller `BLOOM_FPP` will increase runtime calculation while performing query. ++ '0' skipped blocklets of BloomFilter datamap in explain output indicates that + BloomFilter datamap does not prune better than Main datamap. --- End diff -- Added an example scenario --- |
In reply to this post by qiuchenjian-2
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2581 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6075/ --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2581 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7668/ --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2581 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6400/ --- |
In reply to this post by qiuchenjian-2
Github user brijoobopanna commented on the issue:
https://github.com/apache/carbondata/pull/2581 retest sdv please --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2581 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7701/ --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2581 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6427/ --- |
In reply to this post by qiuchenjian-2
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2581 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6094/ --- |
In reply to this post by qiuchenjian-2
|
In reply to this post by qiuchenjian-2
|
Free forum by Nabble | Edit this page |