marchpure commented on a change in pull request #3707: [HOTFIX] Refactor code to optimize partition pruning
URL:
https://github.com/apache/carbondata/pull/3707#discussion_r409700166
##########
File path: core/src/main/java/org/apache/carbondata/core/indexstore/blockletindex/BlockletIndexFactory.java
##########
@@ -193,21 +192,18 @@ public IndexBuilder createBuilder(Segment segment, String shardName,
* get tableBlockUniqueIdentifierWrappers from segment info. If partitionsToPrune is defined,
* then get tableBlockUniqueIdentifierWrappers for the matched partitions.
*/
- private void getTableBlockUniqueIdentifierWrappers(List<PartitionSpec> partitionsToPrune,
+ private void getTableBlockUniqueIdentifierWrappers(Set<String> partitionsToPrune,
List<TableBlockIndexUniqueIdentifierWrapper> tableBlockIndexUniqueIdentifierWrappers,
Set<TableBlockIndexUniqueIdentifier> identifiers) {
for (TableBlockIndexUniqueIdentifier tableBlockIndexUniqueIdentifier : identifiers) {
- if (null != partitionsToPrune) {
+ if (!partitionsToPrune.isEmpty()) {
// add only tableBlockUniqueIdentifier that matches the partition
// get the indexFile Parent path and compare with the PartitionPath, if matches, then add
// the corresponding tableBlockIndexUniqueIdentifier for pruning
- for (PartitionSpec partitionSpec : partitionsToPrune) {
- if (partitionSpec.getLocation().toString()
- .equalsIgnoreCase(tableBlockIndexUniqueIdentifier.getIndexFilePath())) {
- tableBlockIndexUniqueIdentifierWrappers.add(
- new TableBlockIndexUniqueIdentifierWrapper(tableBlockIndexUniqueIdentifier,
- this.getCarbonTable()));
- }
+ if (partitionsToPrune.contains(tableBlockIndexUniqueIdentifier.getIndexFilePath())) {
Review comment:
I tested failed.
In this dli spark, the indexfilepath is "obs://bucktable//tablename//partitionname",
while the partitionSpec's location is "obs://bucktable/tablename/partitionname"
Leading to pruning empty blocklets when queried partition table
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[hidden email]
With regards,
Apache Git Services
Locked
