[GitHub] carbondata pull request #1065: Add short int type support
Locked
123
123
[GitHub] carbondata pull request #1065: Add short int type support
 
|
GitHub user jackylk opened a pull request:
https://github.com/apache/carbondata/pull/1065 Add short int type support This PR add 3 bytes data type support in value compression. It can be used to compress int/long to 3 bytes bits value and decode to int/long. In subsequent PR, it will be used for value compression for dictionary dimension column. Two test cases are added to test boundary value of 3 bytes You can merge this pull request into a Git repository by running: $ git pull https://github.com/jackylk/incubator-carbondata 3bytes Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/1065.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1065 ---- commit 25d47ec3a7faa416f97340128536e0aac278ad49 Author: jackylk <[hidden email]> Date: 2017-06-20T06:32:38Z add short int type support ---- --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at [hidden email] or file a JIRA ticket with INFRA. --- |
[GitHub] carbondata issue #1065: Add short int type support
|
|
Github user asfgit commented on the issue:
https://github.com/apache/carbondata/pull/1065 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/carbondata-pr-spark-1.6/505/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at [hidden email] or file a JIRA ticket with INFRA. --- |
[GitHub] carbondata issue #1065: [CARBONDATA-1196] Add 3 bytes data type support in v...
|
|
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1065 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2602/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at [hidden email] or file a JIRA ticket with INFRA. --- |
[GitHub] carbondata issue #1065: [CARBONDATA-1196] Add 3 bytes data type support in v...
|
|
In reply to this post by qiuchenjian-2
Github user asfgit commented on the issue:
https://github.com/apache/carbondata/pull/1065 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/carbondata-pr-spark-1.6/506/<h2>Build result: FAILURE</span></h2>[...truncated 1013.30 KB...] at hudson.remoting.UserRequest.deserialize(UserRequest.java:217) at hudson.remoting.UserRequest.perform(UserRequest.java:131) at hudson.remoting.UserRequest.perform(UserRequest.java:50) at hudson.remoting.Request$2.run(Request.java:336) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) at ......remote call to ubuntu-2(Native Method) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1545) at hudson.remoting.UserResponse.retrieve(UserRequest.java:253) at hudson.remoting.Channel.call(Channel.java:830)Caused: java.io.IOException: Remote call on ubuntu-2 failed at hudson.remoting.Channel.call(Channel.java:838) at hudson.Launcher$RemoteLauncher.kill(Launcher.java:984) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:540) at hudson.model.Run.execute(Run.java:1728) at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:542) at hudson.model.ResourceController.execute(ResourceController.java:98) at hudson.model.Executor.run(Executor.java:405)Setting status of de7075b129d6ae52b0c9d2205fe1f5ffcfac7b4d to FAILURE with url https://builds.apache.org/job/carbondata-pr-spark-1.6/506/ and message: 'Tests Failed for Spark1.6 'Using context: Jenkins(Spark1.6): mvn clean test -Pspark-1.6 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at [hidden email] or file a JIRA ticket with INFRA. --- |
[GitHub] carbondata issue #1065: [CARBONDATA-1196] Add 3 bytes data type support in v...
|
|
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1065 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2603/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at [hidden email] or file a JIRA ticket with INFRA. --- |
[GitHub] carbondata issue #1065: [CARBONDATA-1196] Add 3 bytes data type support in v...
|
|
In reply to this post by qiuchenjian-2
Github user asfgit commented on the issue:
https://github.com/apache/carbondata/pull/1065 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/carbondata-pr-spark-1.6/510/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at [hidden email] or file a JIRA ticket with INFRA. --- |
[GitHub] carbondata issue #1065: [CARBONDATA-1196] Add 3 bytes data type support in v...
|
|
In reply to this post by qiuchenjian-2
Github user asfgit commented on the issue:
https://github.com/apache/carbondata/pull/1065 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/carbondata-pr-spark-1.6/512/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at [hidden email] or file a JIRA ticket with INFRA. --- |
[GitHub] carbondata issue #1065: [CARBONDATA-1196] Add 3 bytes data type support in v...
|
|
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1065 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2607/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at [hidden email] or file a JIRA ticket with INFRA. --- |
[GitHub] carbondata issue #1065: [CARBONDATA-1196] Add 3 bytes data type support in v...
|
|
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1065 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2609/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at [hidden email] or file a JIRA ticket with INFRA. --- |
[GitHub] carbondata issue #1065: [CARBONDATA-1196] Add 3 bytes data type support in v...
|
|
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1065 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2616/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at [hidden email] or file a JIRA ticket with INFRA. --- |
[GitHub] carbondata issue #1065: [CARBONDATA-1196] Add 3 bytes data type support in v...
[GitHub] carbondata issue #1065: [CARBONDATA-1196] Add 3 bytes data type support in v...
|
|
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1065 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2617/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at [hidden email] or file a JIRA ticket with INFRA. --- |
[GitHub] carbondata issue #1065: [CARBONDATA-1196] Add 3 bytes data type support in v...
|
|
In reply to this post by qiuchenjian-2
Github user asfgit commented on the issue:
https://github.com/apache/carbondata/pull/1065 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/carbondata-pr-spark-1.6/522/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at [hidden email] or file a JIRA ticket with INFRA. --- |
[GitHub] carbondata pull request #1065: [CARBONDATA-1196] Add 3 bytes data type suppo...
|
|
In reply to this post by qiuchenjian-2
Github user QiangCai commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1065#discussion_r123148990 --- Diff: core/src/main/java/org/apache/carbondata/core/util/ByteUtil.java --- @@ -465,6 +465,39 @@ public static short toShort(byte[] bytes, int offset, final int length) { } /** + * int => byte[3] + * supported range is [-8388607, 8388607], note that Math.pow(2, 24) == 8388608 + */ + public static byte[] to3Bytes(int val) { + assert val <= Math.pow(2, 23) && val >= (-Math.pow(2, 23) - 1); + + int value = val < 0 ? -val : val; + byte[] b = new byte[3]; + b[0] = (byte) (value & 0xFF); + b[1] = (byte) ((value >>> 8) & 0xFF); + b[2] = (byte) ((value >>> 16) & 0x7F); --- End diff -- I think we can do as following. no need to judge minus. b[0] = (byte)(x >> 16) b[1] = (byte)(x >> 8) b[2] = (byte)(x) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at [hidden email] or file a JIRA ticket with INFRA. --- |
[GitHub] carbondata pull request #1065: [CARBONDATA-1196] Add 3 bytes data type suppo...
|
|
In reply to this post by qiuchenjian-2
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1065#discussion_r123154384 --- Diff: core/src/main/java/org/apache/carbondata/core/util/ByteUtil.java --- @@ -465,6 +465,39 @@ public static short toShort(byte[] bytes, int offset, final int length) { } /** + * int => byte[3] + * supported range is [-8388607, 8388607], note that Math.pow(2, 24) == 8388608 + */ + public static byte[] to3Bytes(int val) { + assert val <= Math.pow(2, 23) && val >= (-Math.pow(2, 23) - 1); + + int value = val < 0 ? -val : val; + byte[] b = new byte[3]; + b[0] = (byte) (value & 0xFF); + b[1] = (byte) ((value >>> 8) & 0xFF); + b[2] = (byte) ((value >>> 16) & 0x7F); --- End diff -- Because in line474, value is converted to absolute value, so in line479 it needs to keep the "sign bit", otherwise the "sign bit" information will be lost --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at [hidden email] or file a JIRA ticket with INFRA. --- |
[GitHub] carbondata issue #1065: [CARBONDATA-1196] Add 3 bytes data type support in v...
|
|
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1065 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2619/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at [hidden email] or file a JIRA ticket with INFRA. --- |
[GitHub] carbondata issue #1065: [CARBONDATA-1196] Add 3 bytes data type support in v...
|
|
In reply to this post by qiuchenjian-2
Github user asfgit commented on the issue:
https://github.com/apache/carbondata/pull/1065 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/carbondata-pr-spark-1.6/525/<h2>Failed Tests: <span class='status-failure'>1</span></h2><h3><a name='carbondata-pr-spark-1.6/org.apache.carbondata:carbondata-spark-common-test' /><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/525/org.apache.carbondata$carbondata-spark-common-test/testReport'>carbondata-pr-spark-1.6/org.apache.carbondata:carbondata-spark-common-test</a>: <span class='status-failure'>1</span></h3><ul><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/525/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.allqueries/InsertIntoCarbonTableTestCase/insert_into_carbon_table_from_carbon_table_union_query/'><strong>org.apache.carbondata.spark.testsuite.allqueries.InsertIntoCarbonTableTestCase.insert into carbon table from carbon table union query</strong></a></li></ul> --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at [hidden email] or file a JIRA ticket with INFRA. --- |
[GitHub] carbondata pull request #1065: [CARBONDATA-1196] Add 3 bytes data type suppo...
|
|
In reply to this post by qiuchenjian-2
Github user QiangCai commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1065#discussion_r123169106 --- Diff: core/src/main/java/org/apache/carbondata/core/util/ByteUtil.java --- @@ -465,6 +465,39 @@ public static short toShort(byte[] bytes, int offset, final int length) { } /** + * int => byte[3] + * supported range is [-8388607, 8388607], note that Math.pow(2, 24) == 8388608 + */ + public static byte[] to3Bytes(int val) { + assert val <= Math.pow(2, 23) && val >= (-Math.pow(2, 23) - 1); + + int value = val < 0 ? -val : val; + byte[] b = new byte[3]; + b[0] = (byte) (value & 0xFF); + b[1] = (byte) ((value >>> 8) & 0xFF); + b[2] = (byte) ((value >>> 16) & 0x7F); --- End diff -- because we already check the range, so it will not lost 'sign bit'. It means if the value is nimus, the 24th bit is 1, if the value is positive, the 24th bit is 0. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at [hidden email] or file a JIRA ticket with INFRA. --- |
[GitHub] carbondata pull request #1065: [CARBONDATA-1196] Add 3 bytes data type suppo...
|
|
In reply to this post by qiuchenjian-2
Github user QiangCai commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1065#discussion_r123173346 --- Diff: core/src/main/java/org/apache/carbondata/core/util/ByteUtil.java --- @@ -465,6 +465,39 @@ public static short toShort(byte[] bytes, int offset, final int length) { } /** + * int => byte[3] + * supported range is [-8388607, 8388607], note that Math.pow(2, 24) == 8388608 + */ + public static byte[] to3Bytes(int val) { + assert val <= (Math.pow(2, 23) - 1) && val >= (-Math.pow(2, 23) + 1); + + int value = val < 0 ? -val : val; + byte[] b = new byte[3]; + b[0] = (byte) (value & 0xFF); + b[1] = (byte) ((value >>> 8) & 0xFF); + b[2] = (byte) ((value >>> 16) & 0x7F); + if (val < 0) { + b[2] |= 0x80; + } + return b; + } + + /** + * convert 3 bytes to int + */ + public static int valueOf3Bytes(byte[] val, int offset) { + assert val.length >= offset + 3; + int out = (val[offset] & 0xFF); + out |= ((val[offset + 1] & 0xFF) << 8); + out |= ((val[offset + 2] & 0x7F) << 16); + if ((val[offset + 2] & 0x80) != 0) { + return -out; + } else { + return out; + } --- End diff -- if (val[offset] >0) { return (((val[offset] & 0xff) << 16) | ((val[offset + 1] & 0xff) << 8) | ((val[offset + 2] & 0xff) )); } else { return (((val[offset] & 0xffff) << 16) | ((val[offset + 1] & 0xff) << 8) | ((val[offset + 2] & 0xff) )); } --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at [hidden email] or file a JIRA ticket with INFRA. --- |
[GitHub] carbondata pull request #1065: [CARBONDATA-1196] Add 3 bytes data type suppo...
|
|
In reply to this post by qiuchenjian-2
Github user QiangCai commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1065#discussion_r123176128 --- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/encoding/DefaultEncodingStrategy.java --- @@ -29,24 +29,31 @@ private static final Compressor compressor = CompressorFactory.getInstance().getCompressor(); + private static final int THREE_BYTES_MAX = (int) Math.pow(2, 23) - 1; + private static final int THREE_BYTES_MIN = - THREE_BYTES_MAX; --- End diff -- THREE_BYTES_MIN = - (THREE_BYTES_MAX + 1) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at [hidden email] or file a JIRA ticket with INFRA. --- |
«
Return to Apache CarbonData JIRA issues
|
1 view|%1 views
| Free forum by Nabble | Edit this page |