[GitHub] [carbondata] ajantha-bhat opened a new pull request #3913: [WIP] Improve partition purining perfromance in presto carbon integration

[GitBox]

ajantha-bhat commented on a change in pull request #3913:
URL: https://github.com/apache/carbondata/pull/3913#discussion_r485628731



##########
File path: integration/presto/src/main/prestodb/org/apache/carbondata/presto/CarbondataModule.java
##########
@@ -21,6 +21,8 @@
 
 import static java.util.Objects.requireNonNull;
 
+import org.apache.carbondata.core.constants.CarbonCommonConstants;
+import org.apache.carbondata.core.util.CarbonProperties;

Review comment:
       prestodb compile was broken by #3885 , They have copied code but not import statement. so, fixed it. Mentioned in PR description also




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

ajantha-bhat commented on a change in pull request #3913:
URL: https://github.com/apache/carbondata/pull/3913#discussion_r485628731



##########
File path: integration/presto/src/main/prestodb/org/apache/carbondata/presto/CarbondataModule.java
##########
@@ -21,6 +21,8 @@
 
 import static java.util.Objects.requireNonNull;
 
+import org.apache.carbondata.core.constants.CarbonCommonConstants;
+import org.apache.carbondata.core.util.CarbonProperties;

Review comment:
       prestodb compile was broken by #3885 , They have copied code from prestosql but not import statement. so, fixed it. Mentioned in PR description also




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

kunal642 commented on a change in pull request #3913:
URL: https://github.com/apache/carbondata/pull/3913#discussion_r485630147



##########
File path: integration/presto/src/main/prestodb/org/apache/carbondata/presto/CarbondataSplitManager.java
##########
@@ -117,6 +122,16 @@ public ConnectorSplitSource getSplits(ConnectorTransactionHandle transactionHand
       // file metastore case tablePath can be null, so get from location
       location = table.getStorage().getLocation();
     }
+    List<PartitionSpec> filteredPartitions = new ArrayList<>();
+    if (layout.getPartitionColumns().size() > 0 && layout.getPartitions().isPresent()) {
+      List<String> colNames =
+          layout.getPartitionColumns().stream().map(x -> ((HiveColumnHandle) x).getName())
+              .collect(Collectors.toList());
+      for (HivePartition partition : layout.getPartitions().get()) {
+        filteredPartitions.add(new PartitionSpec(colNames,

Review comment:
       according to this code all the PartitionSpec will have all the column names!!! is this correct?
   The column names in partitionSpec should be per partition right?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

kunal642 commented on a change in pull request #3913:
URL: https://github.com/apache/carbondata/pull/3913#discussion_r485634681



##########
File path: integration/presto/src/main/prestodb/org/apache/carbondata/presto/readers/ComplexTypeStreamReader.java
##########
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.presto.readers;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+
+import com.facebook.presto.spi.block.ArrayBlock;
+import com.facebook.presto.spi.block.RowBlock;
+import com.facebook.presto.spi.type.*;
+
+import org.apache.carbondata.core.metadata.datatype.DataType;
+import org.apache.carbondata.core.metadata.datatype.DataTypes;
+import org.apache.carbondata.core.metadata.datatype.StructField;
+import org.apache.carbondata.core.scan.result.vector.CarbonColumnVector;
+import org.apache.carbondata.core.scan.result.vector.impl.CarbonColumnVectorImpl;
+
+import com.facebook.presto.spi.block.Block;
+import com.facebook.presto.spi.block.BlockBuilder;
+
+import org.apache.carbondata.presto.CarbonVectorBatch;
+import org.apache.carbondata.presto.ColumnarVectorWrapperDirect;
+
+/**
+ * Class to read the complex type Stream [array/struct/map]
+ */
+
+public class ComplexTypeStreamReader extends CarbonColumnVectorImpl

Review comment:
       This change is too big and not related to partition pruning, please raise a seperate PR for this.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

ajantha-bhat commented on a change in pull request #3913:
URL: https://github.com/apache/carbondata/pull/3913#discussion_r485635431



##########
File path: integration/presto/src/main/prestodb/org/apache/carbondata/presto/CarbondataSplitManager.java
##########
@@ -117,6 +122,16 @@ public ConnectorSplitSource getSplits(ConnectorTransactionHandle transactionHand
       // file metastore case tablePath can be null, so get from location
       location = table.getStorage().getLocation();
     }
+    List<PartitionSpec> filteredPartitions = new ArrayList<>();
+    if (layout.getPartitionColumns().size() > 0 && layout.getPartitions().isPresent()) {
+      List<String> colNames =
+          layout.getPartitionColumns().stream().map(x -> ((HiveColumnHandle) x).getName())
+              .collect(Collectors.toList());
+      for (HivePartition partition : layout.getPartitions().get()) {
+        filteredPartitions.add(new PartitionSpec(colNames,

Review comment:
       you mean all the partition column right ? not the all table columns ?
   
   Ideally they use partitionPath. I have tested it. It didn't impacted. Let me check again.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

kunal642 commented on a change in pull request #3913:
URL: https://github.com/apache/carbondata/pull/3913#discussion_r485636199



##########
File path: integration/presto/src/main/prestodb/org/apache/carbondata/presto/CarbondataSplitManager.java
##########
@@ -117,6 +122,16 @@ public ConnectorSplitSource getSplits(ConnectorTransactionHandle transactionHand
       // file metastore case tablePath can be null, so get from location
       location = table.getStorage().getLocation();
     }
+    List<PartitionSpec> filteredPartitions = new ArrayList<>();
+    if (layout.getPartitionColumns().size() > 0 && layout.getPartitions().isPresent()) {
+      List<String> colNames =
+          layout.getPartitionColumns().stream().map(x -> ((HiveColumnHandle) x).getName())
+              .collect(Collectors.toList());
+      for (HivePartition partition : layout.getPartitions().get()) {
+        filteredPartitions.add(new PartitionSpec(colNames,

Review comment:
       yeah all partition column, 1 partition spec should have only the columns it is created on, not all columns.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

ajantha-bhat commented on a change in pull request #3913:
URL: https://github.com/apache/carbondata/pull/3913#discussion_r485636502



##########
File path: integration/presto/src/main/prestodb/org/apache/carbondata/presto/readers/ComplexTypeStreamReader.java
##########
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.presto.readers;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+
+import com.facebook.presto.spi.block.ArrayBlock;
+import com.facebook.presto.spi.block.RowBlock;
+import com.facebook.presto.spi.type.*;
+
+import org.apache.carbondata.core.metadata.datatype.DataType;
+import org.apache.carbondata.core.metadata.datatype.DataTypes;
+import org.apache.carbondata.core.metadata.datatype.StructField;
+import org.apache.carbondata.core.scan.result.vector.CarbonColumnVector;
+import org.apache.carbondata.core.scan.result.vector.impl.CarbonColumnVectorImpl;
+
+import com.facebook.presto.spi.block.Block;
+import com.facebook.presto.spi.block.BlockBuilder;
+
+import org.apache.carbondata.presto.CarbonVectorBatch;
+import org.apache.carbondata.presto.ColumnarVectorWrapperDirect;
+
+/**
+ * Class to read the complex type Stream [array/struct/map]
+ */
+
+public class ComplexTypeStreamReader extends CarbonColumnVectorImpl

Review comment:
       Then I have to raise partition changes only for prestosql, because prestodb profile is not compiling.  or I fix the prestodb compile first. you can merge it. I will raise separate PR for partition. what you suggest ?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

kunal642 commented on a change in pull request #3913:
URL: https://github.com/apache/carbondata/pull/3913#discussion_r485638458



##########
File path: integration/presto/src/main/prestodb/org/apache/carbondata/presto/readers/ComplexTypeStreamReader.java
##########
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.presto.readers;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+
+import com.facebook.presto.spi.block.ArrayBlock;
+import com.facebook.presto.spi.block.RowBlock;
+import com.facebook.presto.spi.type.*;
+
+import org.apache.carbondata.core.metadata.datatype.DataType;
+import org.apache.carbondata.core.metadata.datatype.DataTypes;
+import org.apache.carbondata.core.metadata.datatype.StructField;
+import org.apache.carbondata.core.scan.result.vector.CarbonColumnVector;
+import org.apache.carbondata.core.scan.result.vector.impl.CarbonColumnVectorImpl;
+
+import com.facebook.presto.spi.block.Block;
+import com.facebook.presto.spi.block.BlockBuilder;
+
+import org.apache.carbondata.presto.CarbonVectorBatch;
+import org.apache.carbondata.presto.ColumnarVectorWrapperDirect;
+
+/**
+ * Class to read the complex type Stream [array/struct/map]
+ */
+
+public class ComplexTypeStreamReader extends CarbonColumnVectorImpl

Review comment:
       is this compilation fix? if not then keep the compilation changes in this PR and ComplexTypeStreamReader looks to be a bug fix!!!




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

ajantha-bhat commented on a change in pull request #3913:
URL: https://github.com/apache/carbondata/pull/3913#discussion_r485640004



##########
File path: integration/presto/src/main/prestodb/org/apache/carbondata/presto/CarbondataSplitManager.java
##########
@@ -117,6 +122,16 @@ public ConnectorSplitSource getSplits(ConnectorTransactionHandle transactionHand
       // file metastore case tablePath can be null, so get from location
       location = table.getStorage().getLocation();
     }
+    List<PartitionSpec> filteredPartitions = new ArrayList<>();
+    if (layout.getPartitionColumns().size() > 0 && layout.getPartitions().isPresent()) {
+      List<String> colNames =
+          layout.getPartitionColumns().stream().map(x -> ((HiveColumnHandle) x).getName())
+              .collect(Collectors.toList());
+      for (HivePartition partition : layout.getPartitions().get()) {
+        filteredPartitions.add(new PartitionSpec(colNames,

Review comment:
       I think HiveTableHandle will give matching column for the partitions only , because this is per query derived.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

ajantha-bhat commented on a change in pull request #3913:
URL: https://github.com/apache/carbondata/pull/3913#discussion_r485640004



##########
File path: integration/presto/src/main/prestodb/org/apache/carbondata/presto/CarbondataSplitManager.java
##########
@@ -117,6 +122,16 @@ public ConnectorSplitSource getSplits(ConnectorTransactionHandle transactionHand
       // file metastore case tablePath can be null, so get from location
       location = table.getStorage().getLocation();
     }
+    List<PartitionSpec> filteredPartitions = new ArrayList<>();
+    if (layout.getPartitionColumns().size() > 0 && layout.getPartitions().isPresent()) {
+      List<String> colNames =
+          layout.getPartitionColumns().stream().map(x -> ((HiveColumnHandle) x).getName())
+              .collect(Collectors.toList());
+      for (HivePartition partition : layout.getPartitions().get()) {
+        filteredPartitions.add(new PartitionSpec(colNames,

Review comment:
       I think HiveTableHandle will give matching column for the partitions only , because this is per query derived.  Let me retest and get back to you




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

ajantha-bhat commented on a change in pull request #3913:
URL: https://github.com/apache/carbondata/pull/3913#discussion_r485642687



##########
File path: integration/presto/src/main/prestodb/org/apache/carbondata/presto/readers/ComplexTypeStreamReader.java
##########
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.presto.readers;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+
+import com.facebook.presto.spi.block.ArrayBlock;
+import com.facebook.presto.spi.block.RowBlock;
+import com.facebook.presto.spi.type.*;
+
+import org.apache.carbondata.core.metadata.datatype.DataType;
+import org.apache.carbondata.core.metadata.datatype.DataTypes;
+import org.apache.carbondata.core.metadata.datatype.StructField;
+import org.apache.carbondata.core.scan.result.vector.CarbonColumnVector;
+import org.apache.carbondata.core.scan.result.vector.impl.CarbonColumnVectorImpl;
+
+import com.facebook.presto.spi.block.Block;
+import com.facebook.presto.spi.block.BlockBuilder;
+
+import org.apache.carbondata.presto.CarbonVectorBatch;
+import org.apache.carbondata.presto.ColumnarVectorWrapperDirect;
+
+/**
+ * Class to read the complex type Stream [array/struct/map]
+ */
+
+public class ComplexTypeStreamReader extends CarbonColumnVectorImpl

Review comment:
       Not a bug fix, you need to know how prestodb and prestosql is divided.
   there are some common files and individual files.  This class is individual class
   
   There is a complexStreamReader in prestoSql also. same code but import packages are from prestosql. import packages are from prestodb




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

ajantha-bhat commented on a change in pull request #3913:
URL: https://github.com/apache/carbondata/pull/3913#discussion_r485642687



##########
File path: integration/presto/src/main/prestodb/org/apache/carbondata/presto/readers/ComplexTypeStreamReader.java
##########
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.presto.readers;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+
+import com.facebook.presto.spi.block.ArrayBlock;
+import com.facebook.presto.spi.block.RowBlock;
+import com.facebook.presto.spi.type.*;
+
+import org.apache.carbondata.core.metadata.datatype.DataType;
+import org.apache.carbondata.core.metadata.datatype.DataTypes;
+import org.apache.carbondata.core.metadata.datatype.StructField;
+import org.apache.carbondata.core.scan.result.vector.CarbonColumnVector;
+import org.apache.carbondata.core.scan.result.vector.impl.CarbonColumnVectorImpl;
+
+import com.facebook.presto.spi.block.Block;
+import com.facebook.presto.spi.block.BlockBuilder;
+
+import org.apache.carbondata.presto.CarbonVectorBatch;
+import org.apache.carbondata.presto.ColumnarVectorWrapperDirect;
+
+/**
+ * Class to read the complex type Stream [array/struct/map]
+ */
+
+public class ComplexTypeStreamReader extends CarbonColumnVectorImpl

Review comment:
       Not a bug fix, you need to know how prestodb and prestosql is divided.
   there are some common files and individual files.  This class is individual class
   
   There is a complexStreamReader in prestoSql also. same code but import packages are from prestosql. here, import packages are from prestodb




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

kunal642 commented on a change in pull request #3913:
URL: https://github.com/apache/carbondata/pull/3913#discussion_r485651041



##########
File path: integration/presto/src/main/prestodb/org/apache/carbondata/presto/readers/ComplexTypeStreamReader.java
##########
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.presto.readers;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+
+import com.facebook.presto.spi.block.ArrayBlock;
+import com.facebook.presto.spi.block.RowBlock;
+import com.facebook.presto.spi.type.*;
+
+import org.apache.carbondata.core.metadata.datatype.DataType;
+import org.apache.carbondata.core.metadata.datatype.DataTypes;
+import org.apache.carbondata.core.metadata.datatype.StructField;
+import org.apache.carbondata.core.scan.result.vector.CarbonColumnVector;
+import org.apache.carbondata.core.scan.result.vector.impl.CarbonColumnVectorImpl;
+
+import com.facebook.presto.spi.block.Block;
+import com.facebook.presto.spi.block.BlockBuilder;
+
+import org.apache.carbondata.presto.CarbonVectorBatch;
+import org.apache.carbondata.presto.ColumnarVectorWrapperDirect;
+
+/**
+ * Class to read the complex type Stream [array/struct/map]
+ */
+
+public class ComplexTypeStreamReader extends CarbonColumnVectorImpl

Review comment:
       ok




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

kunal642 commented on a change in pull request #3913:
URL: https://github.com/apache/carbondata/pull/3913#discussion_r485651703



##########
File path: integration/presto/src/main/prestodb/org/apache/carbondata/presto/CarbondataModule.java
##########
@@ -21,6 +21,8 @@
 
 import static java.util.Objects.requireNonNull;
 
+import org.apache.carbondata.core.constants.CarbonCommonConstants;
+import org.apache.carbondata.core.util.CarbonProperties;

Review comment:
       ok




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

ajantha-bhat commented on a change in pull request #3913:
URL: https://github.com/apache/carbondata/pull/3913#discussion_r486048290



##########
File path: integration/presto/src/main/prestodb/org/apache/carbondata/presto/CarbondataSplitManager.java
##########
@@ -117,6 +122,16 @@ public ConnectorSplitSource getSplits(ConnectorTransactionHandle transactionHand
       // file metastore case tablePath can be null, so get from location
       location = table.getStorage().getLocation();
     }
+    List<PartitionSpec> filteredPartitions = new ArrayList<>();
+    if (layout.getPartitionColumns().size() > 0 && layout.getPartitions().isPresent()) {
+      List<String> colNames =
+          layout.getPartitionColumns().stream().map(x -> ((HiveColumnHandle) x).getName())
+              .collect(Collectors.toList());
+      for (HivePartition partition : layout.getPartitions().get()) {
+        filteredPartitions.add(new PartitionSpec(colNames,

Review comment:
       I have verified. This scenario can work. I guess. No need to modify any code.
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

ajantha-bhat commented on a change in pull request #3913:
URL: https://github.com/apache/carbondata/pull/3913#discussion_r486048413



##########
File path: integration/presto/src/main/prestodb/org/apache/carbondata/presto/CarbondataSplitManager.java
##########
@@ -117,6 +122,16 @@ public ConnectorSplitSource getSplits(ConnectorTransactionHandle transactionHand
       // file metastore case tablePath can be null, so get from location
       location = table.getStorage().getLocation();
     }
+    List<PartitionSpec> filteredPartitions = new ArrayList<>();
+    if (layout.getPartitionColumns().size() > 0 && layout.getPartitions().isPresent()) {
+      List<String> colNames =
+          layout.getPartitionColumns().stream().map(x -> ((HiveColumnHandle) x).getName())
+              .collect(Collectors.toList());
+      for (HivePartition partition : layout.getPartitions().get()) {
+        filteredPartitions.add(new PartitionSpec(colNames,

Review comment:
       ```
   presto:redods> select dtm,hh from dw_log_ubt_partition_carbon_neww;
        dtm     |     hh
   -------------+------------
    part_dtm_01 | part_hh_01
    part_dtm_01 | part_hh_01
    part_dtm_01 | part_hh_02
    part_dtm_20 | part_hh_21
    part_dtm_01 | part_hh_03
    part_dtm_21 | NULL
   (6 rows)
   
   Query 20200910_035416_00017_wv9qh, FINISHED, 3 nodes
   Splits: 22 total, 22 done (100.00%)
   0:01 [6 rows, 176B] [9 rows/s, 282B/s]
   
   presto:redods> select dtm,hh from dw_log_ubt_partition_carbon_neww where (dtm = 'part_dtm_01' and hh = 'part_hh_03') or dtm='part_dtm_21';
        dtm     |     hh
   -------------+------------
    part_dtm_01 | part_hh_03
    part_dtm_21 | NULL
   (2 rows)
   
   Query 20200910_035548_00018_wv9qh, FINISHED, 3 nodes
   Splits: 18 total, 18 done (100.00%)
   0:01 [2 rows, 0B] [1 rows/s, 0B/s]
   
   
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3913:
URL: https://github.com/apache/carbondata/pull/3913#issuecomment-690292515


   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4038/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3913:
URL: https://github.com/apache/carbondata/pull/3913#issuecomment-690295508


   Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2299/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

ajantha-bhat commented on a change in pull request #3913:
URL: https://github.com/apache/carbondata/pull/3913#discussion_r486797641



##########
File path: integration/presto/src/main/prestosql/org/apache/carbondata/presto/impl/CarbonTableReader.java
##########
@@ -245,16 +242,14 @@ private CarbonTableCacheModel getValidCacheBySchemaTableName(SchemaTableName sch
    *
    * @param tableCacheModel cached table
    * @param filters carbonData filters
-   * @param constraints presto filters
+   * @param filteredPartitions matched partitionSpec for the filter
    * @param config hadoop conf
    * @return list of multiblock split
    * @throws IOException
    */
-  public List<CarbonLocalMultiBlockSplit> getInputSplits(
-      CarbonTableCacheModel tableCacheModel,
-      Expression filters,
-      TupleDomain<HiveColumnHandle> constraints,
-      Configuration config) throws IOException {
+  public List<CarbonLocalMultiBlockSplit> getInputSplits(CarbonTableCacheModel tableCacheModel,
+      Expression filters, List<PartitionSpec> filteredPartitions, Configuration config)

Review comment:
       done

##########
File path: integration/presto/src/test/prestodb/org/apache/carbondata/presto/server/PrestoTestUtil.scala
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.presto.server
+
+import com.facebook.presto.jdbc.PrestoArray
+

Review comment:
       done




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

kunal642 commented on pull request #3913:
URL: https://github.com/apache/carbondata/pull/3913#issuecomment-691443363


   retest this please


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]