GitHub user sounakr opened a pull request:
https://github.com/apache/carbondata/pull/1751 [WIP]Measure null Recognise Null values in blocklets so that it can help in filter prunning. - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/sounakr/incubator-carbondata measure_null Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/1751.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1751 ---- commit 7d0f9709578651ea29fda335663d1d0075091b62 Author: sounakr <sounakr@...> Date: 2017-12-29T13:44:03Z Measure Null Filter commit 82df440e98234661b8027f14c464da30e407f762 Author: sounakr <sounakr@...> Date: 2018-01-03T02:43:06Z Measure Null Value commit 67ea82d752605902fb94c73aa9b9bf31d2e9dafb Author: sounakr <sounakr@...> Date: 2018-01-03T03:06:39Z Measure Null Value ---- --- |
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2496/ --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751 Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1271/ --- |
In reply to this post by qiuchenjian-2
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1751 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2658/ --- |
In reply to this post by qiuchenjian-2
Github user sounakr commented on the issue:
https://github.com/apache/carbondata/pull/1751 Retest this please. --- |
In reply to this post by qiuchenjian-2
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r159368019 --- Diff: core/src/main/java/org/apache/carbondata/core/datastore/chunk/AbstractRawColumnChunk.java --- @@ -29,6 +29,8 @@ private byte[][] maxValues; + private byte[] nullValues; --- End diff -- please add comment --- |
In reply to this post by qiuchenjian-2
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r159368023 --- Diff: core/src/main/java/org/apache/carbondata/core/datastore/DataRefNode.java --- @@ -71,6 +71,8 @@ */ byte[][] getColumnsMinValue(); + byte[] getColumnsNullValue(); --- End diff -- please add comment --- |
In reply to this post by qiuchenjian-2
Github user sounakr commented on the issue:
https://github.com/apache/carbondata/pull/1751 Retest this please. --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2503/ --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751 Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1279/ --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2507/ --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751 Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1282/ --- |
In reply to this post by qiuchenjian-2
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1751 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2671/ --- |
In reply to this post by qiuchenjian-2
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1751 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2672/ --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1322/ --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2560/ --- |
In reply to this post by qiuchenjian-2
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1751 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2723/ --- |
In reply to this post by qiuchenjian-2
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r159829085 --- Diff: format/src/main/thrift/carbondata.thrift --- @@ -45,6 +45,7 @@ struct BlockletBTreeIndex{ struct BlockletMinMaxIndex{ 1: required list<binary> min_values; //Min value of all columns of one blocklet Bit-Packed 2: required list<binary> max_values; //Max value of all columns of one blocklet Bit-Packed + 3: optional list<byte> null_value; // Null Bool turn on if null value present. --- End diff -- Better use binary and try to use BitSet for null values --- |
In reply to this post by qiuchenjian-2
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r159829185 --- Diff: examples/spark2/src/main/scala/org/apache/carbondata/examples/ExampleUtils.scala --- @@ -39,8 +39,8 @@ object ExampleUtils { val warehouse = s"$rootPath/examples/spark2/target/warehouse" CarbonProperties.getInstance() - .addProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT, "yyyy/MM/dd HH:mm:ss") - .addProperty(CarbonCommonConstants.CARBON_DATE_FORMAT, "yyyy/MM/dd") + .addProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT, "yyyy-MM-dd HH:mm:ss") --- End diff -- Don't change the example class --- |
In reply to this post by qiuchenjian-2
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r159829285 --- Diff: examples/spark2/src/main/scala/org/apache/carbondata/examples/CarbonSessionExample.scala --- @@ -62,9 +62,9 @@ object CarbonSessionExample { spark.sql( s""" - | SELECT * - | FROM carbon_table - | WHERE stringfield = 'spark' AND decimalField > 40 + | SELECT * --- End diff -- Don't change this example. --- |
Free forum by Nabble | Edit this page |