GitHub user xuchuanyin opened a pull request:
https://github.com/apache/carbondata/pull/2564 [CARBONDATA-2785][ExternalFormat] Optimize table pruning info for pruning by segment In previous implementation, table pruning is performed once for all segments. In this case, the pruning info is updated after default/CG/FG datamap pruning. Now we want the table pruning to be performed segment by segment, so in this case, the pruning info will be updated after default/CG/FG datamap pruning for each segment. This means that we need to accumulate the pruning info during pruning. Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [x] Any interfaces changed? `Only internal interfaces changed` - [x] Any backward compatibility impacted? `NO` - [x] Document update required? `NO` - [x] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? `NO, only optimize procedure` - How it is tested? Please attach test report. `Tested in local` - Is it a performance related change? Please attach the performance test report. `NO` - Any additional information to help reviewers in testing this change. `NO` - [x] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. `NA` You can merge this pull request into a Git repository by running: $ git pull https://github.com/xuchuanyin/carbondata 0726_getsplits_by_segment Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2564.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2564 ---- commit c8812c517be0bf61f61720d4c860022554fb6447 Author: xuchuanyin <xuchuanyin@...> Date: 2018-07-26T07:19:36Z optimize table pruning info for pruning by segment In previous implementation, table pruning is performed once for all segments. In this case, the pruning info is updated after default/CG/FG datamap pruning. Now we want the table pruning to be performed segment by segment, so in this case, the pruning info will be updated after default/CG/FG datamap pruning for each segment. This means that we need to accumulate the pruning info during pruning. ---- --- |
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2564 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7524/ --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2564 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6278/ --- |
In reply to this post by qiuchenjian-2
Github user brijoobopanna commented on the issue:
https://github.com/apache/carbondata/pull/2564 retest sdv please --- |
In reply to this post by qiuchenjian-2
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2564 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6051/ --- |
In reply to this post by qiuchenjian-2
Github user brijoobopanna commented on the issue:
https://github.com/apache/carbondata/pull/2564 retest this please --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2564 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7641/ --- |
In reply to this post by qiuchenjian-2
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2564#discussion_r206419989 --- Diff: hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonTableInputFormat.java --- @@ -245,9 +245,17 @@ public CarbonTable getOrCreateCarbonTable(Configuration configuration) throws IO FilterResolverIntf filterInterface = carbonTable.resolveFilter(filter); // do block filtering and get split - List<InputSplit> splits = - getSplits(job, filterInterface, filteredSegmentToAccess, matchedPartitions, partitionInfo, - null, updateStatusManager); + List<InputSplit> splits = new ArrayList<>(); + for (Segment segment : filteredSegmentToAccess) { + List<InputSplit> splitsPerSegment = getSplitsForSegment(job, filterInterface, segment, --- End diff -- There is func called getSplitsOfOneSegment, can you change both func name to make it more readable --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2564 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6383/ --- |
In reply to this post by qiuchenjian-2
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2564 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6198/ --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2564 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7825/ --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2564 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6550/ --- |
In reply to this post by qiuchenjian-2
Github user xuchuanyin closed the pull request at:
https://github.com/apache/carbondata/pull/2564 --- |
Free forum by Nabble | Edit this page |