[GitHub] incubator-carbondata pull request #576: [WIP] Improve query performance with...

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

[GitHub] incubator-carbondata pull request #576: [WIP] Improve query performance with...

qiuchenjian-2
GitHub user ravipesala opened a pull request:

    https://github.com/apache/incubator-carbondata/pull/576

    [WIP] Improve query performance with read a head blocklet and changed number compressor.

    Need to work out with backward compatability

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/ravipesala/incubator-carbondata vector_perform

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-carbondata/pull/576.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #576
   
----
commit 1c64874f18610be5de1490fe4f450ea6ca2127b0
Author: ravipesala <[hidden email]>
Date:   2017-01-12T06:37:41Z

    Added prefetch to scanner

commit d61ef31ff3867bd0a1e032e4753a99a9db54e519
Author: ravipesala <[hidden email]>
Date:   2017-01-26T09:54:24Z

    Added read a head blocklet

commit 37d7ca08c7184d01eba2eb717f9c204ed8817142
Author: ravipesala <[hidden email]>
Date:   2017-01-27T07:13:22Z

    Added read a head

commit 6b3af88bf8f19c3156d9c3e5c8208bc13cf10018
Author: ravipesala <[hidden email]>
Date:   2017-01-27T07:19:12Z

    Fixed style

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|

[GitHub] incubator-carbondata issue #576: [WIP] Improve query performance with read a...

qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/576
 
    Build Failed  with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/757/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|

[GitHub] incubator-carbondata issue #576: [WIP] Improve query performance with read a...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/576
 
    Build Failed  with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/759/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|

[GitHub] incubator-carbondata issue #576: [WIP] Improve query performance with read a...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/576
 
    Build Failed  with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/761/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|

[GitHub] incubator-carbondata pull request #576: [WIP] Improve query performance with...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/576#discussion_r98163965
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/keygenerator/mdkey/NumberCompressor.java ---
    @@ -137,12 +161,31 @@ public NumberCompressor(int cardinality) {
         return words;
       }
     
    +  //  public int[] unCompress(byte[] key, int offset, int length) {
    +  //    int ls = length;
    +  //    int arrayLength = (ls * BYTE_LENGTH) / bitsLength;
    +  //    long[] words = new long[getWordsSizeFromBytesSize(ls)];
    +  //    unCompressVal(key, ls, words, offset);
    +  //    return getArray(words, arrayLength);
    +  //  }
    +
       public int[] unCompress(byte[] key, int offset, int length) {
    -    int ls = length;
    -    int arrayLength = (ls * BYTE_LENGTH) / bitsLength;
    -    long[] words = new long[getWordsSizeFromBytesSize(ls)];
    -    unCompressVal(key, ls, words, offset);
    -    return getArray(words, arrayLength);
    +    int len = length / bytesPerKey;
    +    if (length % bytesPerKey != 0) {
    +      throw new IllegalArgumentException(
    +          "Some thing wrong while decompress" + length + " " + bytesPerKey);
    --- End diff --
   
    Both length is read from footer, right? If comes here, mean the file is corrupted.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|

[GitHub] incubator-carbondata issue #576: [WIP] Improve query performance with read a...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/576
 
    Build Failed  with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/762/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|

[GitHub] incubator-carbondata issue #576: [WIP] Improve query performance with read a...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/576
 
    Build Failed  with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/790/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|

[GitHub] incubator-carbondata issue #576: [WIP] Improve query performance with read a...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/576
 
    Build Failed  with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/801/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|

[GitHub] incubator-carbondata issue #576: [WIP] Improve query performance with read a...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user ravipesala commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/576
 
    Same has been handled in https://github.com/apache/incubator-carbondata/pull/584, so closing this PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|

[GitHub] incubator-carbondata pull request #576: [WIP] Improve query performance with...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user ravipesala closed the pull request at:

    https://github.com/apache/incubator-carbondata/pull/576


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---