[GitHub] carbondata pull request #1324: [CARBONDATA-1291]:carbonData query performanc...

[qiuchenjian-2]

GitHub user kushalsaha opened a pull request:

    https://github.com/apache/carbondata/pull/1324

    [CARBONDATA-1291]:carbonData query performance improvement when numbe…

    …r of carbon blocks are high
   
    Be sure to do all of the following to help us incorporate your contribution
    quickly and easily:
   
     - [ ] Make sure the PR title is formatted like:
       `[CARBONDATA-<Jira issue #>] Description of pull request`
     - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
           Travis-CI on your fork and ensure the whole test matrix passes).
     - [ ] Replace `<Jira issue #>` in the title with the actual Jira issue
           number, if there is one.
     - [ ] If this contribution is large, please file an Apache
           [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt).
     - [ ] Testing done
     
            Please provide details on
            - Whether new unit test cases have been added or why no new tests are required?
            - What manual testing you have done?
            - Any additional information to help reviewers in testing this change.
             
     - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
                     
    ---


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/kushalsaha/carbondata CARBONDATA-1291

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/1324.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1324
   
----
commit 9b680cd01151c06c96e68be523c8c254e798b9fe
Author: kushalsaha <[hidden email]>
Date:   2017-09-05T11:40:30Z

    [CARBONDATA-1291]:carbonData query performance improvement when number of carbon blocks are high

----


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1324
 
    Can one of the admins verify this patch?


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1324
 
    Can one of the admins verify this patch?


---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1324#discussion_r137213780
 
    --- Diff: hadoop/src/main/java/org/apache/carbondata/hadoop/CarbonInputFormat.java ---
    @@ -460,21 +460,22 @@ private BitSet setMatchedPartitions(String partitionIds, CarbonTable carbonTable
           for (DataRefNode dataRefNode : dataRefNodes) {
             BlockBTreeLeafNode leafNode = (BlockBTreeLeafNode) dataRefNode;
             TableBlockInfo tableBlockInfo = leafNode.getTableBlockInfo();
    +        String[] deleteDeltaFilePath = null;
             if (isIUDTable) {
               // In case IUD is not performed in this table avoid searching for
               // invalidated blocks.
               if (CarbonUtil
    -              .isInvalidTableBlock(tableBlockInfo.getSegmentId(), tableBlockInfo.getFilePath(),
    -                  invalidBlockVOForSegmentId, updateStatusManager)) {
    +                  .isInvalidTableBlock(tableBlockInfo.getSegmentId(), tableBlockInfo.getFilePath(),
    --- End diff --
   
    `CarbonInputFormat.java` is not used now. please use `CarbonTableInputFormat`


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1324
 
    Can one of the admins verify this patch?


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1324
 
    Can one of the admins verify this patch?


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1324
 
    Can one of the admins verify this patch?


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1324
 
    Can one of the admins verify this patch?


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1324
 
    @kushalsaha Please provide some details on the PR why this change required


---

[qiuchenjian-2]

Github user kushalsaha commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1324#discussion_r138850252
 
    --- Diff: hadoop/src/main/java/org/apache/carbondata/hadoop/CarbonInputFormat.java ---
    @@ -460,21 +460,22 @@ private BitSet setMatchedPartitions(String partitionIds, CarbonTable carbonTable
           for (DataRefNode dataRefNode : dataRefNodes) {
             BlockBTreeLeafNode leafNode = (BlockBTreeLeafNode) dataRefNode;
             TableBlockInfo tableBlockInfo = leafNode.getTableBlockInfo();
    +        String[] deleteDeltaFilePath = null;
             if (isIUDTable) {
               // In case IUD is not performed in this table avoid searching for
               // invalidated blocks.
               if (CarbonUtil
    -              .isInvalidTableBlock(tableBlockInfo.getSegmentId(), tableBlockInfo.getFilePath(),
    -                  invalidBlockVOForSegmentId, updateStatusManager)) {
    +                  .isInvalidTableBlock(tableBlockInfo.getSegmentId(), tableBlockInfo.getFilePath(),
    --- End diff --
   
    Do i need to revert the changes done in CarbonInputFormat.java


---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1324#discussion_r138863711
 
    --- Diff: hadoop/src/main/java/org/apache/carbondata/hadoop/CarbonInputFormat.java ---
    @@ -460,21 +460,22 @@ private BitSet setMatchedPartitions(String partitionIds, CarbonTable carbonTable
           for (DataRefNode dataRefNode : dataRefNodes) {
             BlockBTreeLeafNode leafNode = (BlockBTreeLeafNode) dataRefNode;
             TableBlockInfo tableBlockInfo = leafNode.getTableBlockInfo();
    +        String[] deleteDeltaFilePath = null;
             if (isIUDTable) {
               // In case IUD is not performed in this table avoid searching for
               // invalidated blocks.
               if (CarbonUtil
    -              .isInvalidTableBlock(tableBlockInfo.getSegmentId(), tableBlockInfo.getFilePath(),
    -                  invalidBlockVOForSegmentId, updateStatusManager)) {
    +                  .isInvalidTableBlock(tableBlockInfo.getSegmentId(), tableBlockInfo.getFilePath(),
    --- End diff --
   
    you can keep here , not an issue since this class going to be removed soon. Please do changes in `CarbonTableInputFormat`


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1324
 
    retest this please


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1324
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/764/



---

[qiuchenjian-2]

Github user kushalsaha commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1324#discussion_r138870106
 
    --- Diff: hadoop/src/main/java/org/apache/carbondata/hadoop/CarbonInputFormat.java ---
    @@ -460,21 +460,22 @@ private BitSet setMatchedPartitions(String partitionIds, CarbonTable carbonTable
           for (DataRefNode dataRefNode : dataRefNodes) {
             BlockBTreeLeafNode leafNode = (BlockBTreeLeafNode) dataRefNode;
             TableBlockInfo tableBlockInfo = leafNode.getTableBlockInfo();
    +        String[] deleteDeltaFilePath = null;
             if (isIUDTable) {
               // In case IUD is not performed in this table avoid searching for
               // invalidated blocks.
               if (CarbonUtil
    -              .isInvalidTableBlock(tableBlockInfo.getSegmentId(), tableBlockInfo.getFilePath(),
    -                  invalidBlockVOForSegmentId, updateStatusManager)) {
    +                  .isInvalidTableBlock(tableBlockInfo.getSegmentId(), tableBlockInfo.getFilePath(),
    --- End diff --
   
    Changes done in CarbonTableInputFormat and changes is pushed


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1324
 
    Build Success with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/18/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1324
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/144/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1324
 
    retest this please


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1324
 
    LGTM


---

[qiuchenjian-2]

Github user asfgit closed the pull request at:

    https://github.com/apache/carbondata/pull/1324


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1324
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/787/



---