[GitHub] carbondata pull request #2564: [CARBONDATA-2785][ExternalFormat] Optimize ta...

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata pull request #2564: [CARBONDATA-2785][ExternalFormat] Optimize ta...

qiuchenjian-2
GitHub user xuchuanyin opened a pull request:

    https://github.com/apache/carbondata/pull/2564

    [CARBONDATA-2785][ExternalFormat] Optimize table pruning info for pruning by segment

    In previous implementation, table pruning is performed once for all
    segments. In this case, the pruning info is updated after default/CG/FG
    datamap pruning.
   
    Now we want the table pruning to be performed segment by segment, so in
    this case, the pruning info will be updated after default/CG/FG datamap
    pruning for each segment.
   
    This means that we need to accumulate the pruning info during pruning.
   
    Be sure to do all of the following checklist to help us incorporate
    your contribution quickly and easily:
   
     - [x] Any interfaces changed?
     `Only internal interfaces changed`
     - [x] Any backward compatibility impacted?
     `NO`
     - [x] Document update required?
    `NO`
     - [x] Testing done
            Please provide details on
            - Whether new unit test cases have been added or why no new tests are required?
    `NO, only optimize procedure`
            - How it is tested? Please attach test report.
    `Tested in local`
            - Is it a performance related change? Please attach the performance test report.
    `NO`
            - Any additional information to help reviewers in testing this change.
    `NO`
           
     - [x] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
    `NA`


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/xuchuanyin/carbondata 0726_getsplits_by_segment

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/2564.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2564
   
----
commit c8812c517be0bf61f61720d4c860022554fb6447
Author: xuchuanyin <xuchuanyin@...>
Date:   2018-07-26T07:19:36Z

    optimize table pruning info for pruning by segment
   
    In previous implementation, table pruning is performed once for all
    segments. In this case, the pruning info is updated after default/CG/FG
    datamap pruning.
   
    Now we want the table pruning to be performed segment by segment, so in
    this case, the pruning info will be updated after default/CG/FG datamap
    pruning for each segment.
   
    This means that we need to accumulate the pruning info during pruning.

----


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2564: [CARBONDATA-2785][ExternalFormat] Optimize table pru...

qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2564
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7524/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2564: [CARBONDATA-2785][ExternalFormat] Optimize table pru...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2564
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6278/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2564: [CARBONDATA-2785][ExternalFormat] Optimize table pru...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user brijoobopanna commented on the issue:

    https://github.com/apache/carbondata/pull/2564
 
    retest sdv please



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2564: [CARBONDATA-2785][ExternalFormat] Optimize table pru...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2564
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6051/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2564: [CARBONDATA-2785][ExternalFormat] Optimize table pru...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user brijoobopanna commented on the issue:

    https://github.com/apache/carbondata/pull/2564
 
    retest this please



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2564: [CARBONDATA-2785][ExternalFormat] Optimize table pru...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2564
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7641/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata pull request #2564: [CARBONDATA-2785][ExternalFormat] Optimize ta...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2564#discussion_r206419989
 
    --- Diff: hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonTableInputFormat.java ---
    @@ -245,9 +245,17 @@ public CarbonTable getOrCreateCarbonTable(Configuration configuration) throws IO
         FilterResolverIntf filterInterface = carbonTable.resolveFilter(filter);
     
         // do block filtering and get split
    -    List<InputSplit> splits =
    -        getSplits(job, filterInterface, filteredSegmentToAccess, matchedPartitions, partitionInfo,
    -            null, updateStatusManager);
    +    List<InputSplit> splits = new ArrayList<>();
    +    for (Segment segment : filteredSegmentToAccess) {
    +      List<InputSplit> splitsPerSegment = getSplitsForSegment(job, filterInterface, segment,
    --- End diff --
   
    There is func called getSplitsOfOneSegment, can you change both func name to make it more readable


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2564: [CARBONDATA-2785][ExternalFormat] Optimize table pru...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2564
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6383/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2564: [CARBONDATA-2785][ExternalFormat] Optimize table pru...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2564
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6198/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2564: [CARBONDATA-2785][ExternalFormat] Optimize table pru...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2564
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7825/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2564: [CARBONDATA-2785][ExternalFormat] Optimize table pru...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2564
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6550/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata pull request #2564: WIP:[CARBONDATA-2785][ExternalFormat] Optimiz...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user xuchuanyin closed the pull request at:

    https://github.com/apache/carbondata/pull/2564


---