[GitHub] carbondata pull request #1125: [CarbonData-1250] change default partition id...

[qiuchenjian-2]

Github user QiangCai commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1125#discussion_r125954219
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/metadata/schema/PartitionInfo.java ---
    @@ -43,13 +44,30 @@
       private List<List<String>> listInfo;
     
       /**
    -   * number of partitions
    +   * number of hash partitions
        */
    -  private int numPartitions;
    +  private int hashNumber;
    +
    +  /**
    +   * total count of partitions
    +   */
    +  private int numberOfPartitions;
    --- End diff --
   
    for hash partition,  hashNumber is same with numberOfPartitions.
    why we need two variables?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user QiangCai commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1125#discussion_r125957306
 
    --- Diff: hadoop/src/main/java/org/apache/carbondata/hadoop/CarbonInputFormat.java ---
    @@ -274,28 +275,27 @@ private static AbsoluteTableIdentifier getAbsoluteTableIdentifier(Configuration
           CarbonInputFormatUtil.processFilterExpression(filter, carbonTable);
     
           // prune partitions for filter query on partition table
    -      BitSet matchedPartitions = null;
    -      if (null != filter) {
    -        PartitionInfo partitionInfo = carbonTable.getPartitionInfo(carbonTable.getFactTableName());
    -        if (null != partitionInfo) {
    -          Partitioner partitioner = PartitionUtil.getPartitioner(partitionInfo);
    -          matchedPartitions = new FilterExpressionProcessor()
    -              .getFilteredPartitions(filter, partitionInfo, partitioner);
    +      BitSet matchedPartitions = setMatchedPartitions(null, carbonTable, filter);
    +      if (partitionInfo != null) {
    +        Partitioner partitioner = PartitionUtil.getPartitioner(partitionInfo);
    +        if (matchedPartitions != null) {
               if (matchedPartitions.cardinality() == 0) {
                 // no partition is required
                 return new ArrayList<InputSplit>();
    -          }
    -          if (matchedPartitions.cardinality() == partitioner.numPartitions()) {
    +          } else if (matchedPartitions.cardinality() == partitioner.numPartitions()) {
                 // all partitions are required, no need to prune partitions
                 matchedPartitions = null;
               }
             }
           }
     
    +
    +
    --- End diff --
   
    remove this line


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user QiangCai commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1125#discussion_r125954839
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/scan/filter/partition/PartitionFilterUtil.java ---
    @@ -229,35 +229,37 @@ public static BitSet getPartitionMapForRangeFilter(PartitionInfo partitionInfo,
           // filter value is in default partition
           if (isGreaterThan) {
             // GreaterThan(>), GreaterThanEqualTo(>=)
    -        partitionMap.set(numPartitions);
    +        partitionMap.set(0);
           } else {
             // LessThan(<), LessThanEqualTo(<=)
    -        partitionMap.set(0, partitioner.numPartitions());
    +        partitionMap.set(0, partitioner.numPartitions() - 1);
    --- End diff --
   
    why we lost a partition?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user QiangCai commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1125#discussion_r125954460
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/metadata/schema/PartitionInfo.java ---
    @@ -43,13 +44,30 @@
       private List<List<String>> listInfo;
     
       /**
    -   * number of partitions
    +   * number of hash partitions
        */
    -  private int numPartitions;
    +  private int hashNumber;
    +
    +  /**
    +   * total count of partitions
    +   */
    +  private int numberOfPartitions;
    +
    +  /**
    +   * task id of max partition
    +   */
    +  private int MAX_PARTITION;
    --- End diff --
   
    why need to save MAX_PARTITION to schema?
    please correct comment.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user QiangCai commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1125#discussion_r125954525
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/metadata/schema/PartitionInfo.java ---
    @@ -43,13 +44,30 @@
       private List<List<String>> listInfo;
     
       /**
    -   * number of partitions
    +   * number of hash partitions
        */
    -  private int numPartitions;
    +  private int hashNumber;
    +
    +  /**
    +   * total count of partitions
    +   */
    +  private int numberOfPartitions;
    +
    +  /**
    +   * task id of max partition
    +   */
    +  private int MAX_PARTITION;
    +
    +  /**
    +   * record the task id in partition order
    --- End diff --
   
    please correct comment


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user QiangCai commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1125#discussion_r125955225
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/scan/filter/partition/PartitionFilterUtil.java ---
    @@ -229,35 +229,37 @@ public static BitSet getPartitionMapForRangeFilter(PartitionInfo partitionInfo,
           // filter value is in default partition
           if (isGreaterThan) {
             // GreaterThan(>), GreaterThanEqualTo(>=)
    -        partitionMap.set(numPartitions);
    +        partitionMap.set(0);
           } else {
             // LessThan(<), LessThanEqualTo(<=)
    -        partitionMap.set(0, partitioner.numPartitions());
    +        partitionMap.set(0, partitioner.numPartitions() - 1);
           }
         } else {
           // filter value is not in default partition
           if (result == 0) {
             // if result is 0, the filter value is a bound value of range partition.
             if (isGreaterThan) {
               // GreaterThan(>), GreaterThanEqualTo(>=)
    -          partitionMap.set(partitionIndex + 1, partitioner.numPartitions());
    +          partitionMap.set(partitionIndex + 2, partitioner.numPartitions() - 1);
    --- End diff --
   
    why we lost two partition?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user QiangCai commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1125#discussion_r125958721
 
    --- Diff: hadoop/src/main/java/org/apache/carbondata/hadoop/CarbonInputFormat.java ---
    @@ -417,18 +441,26 @@ protected Expression getFilterPredicates(Configuration configuration) {
               getSegmentAbstractIndexs(job, absoluteTableIdentifier, segmentId, cacheClient,
                   updateStatusManager);
           List<DataRefNode> resultFilterredBlocks = new LinkedList<DataRefNode>();
    +      int partitionIndex = -1;
    +      List<Integer> taskIdGroup = new ArrayList<>();
    +      if (partitionInfo != null) {
    +        taskIdGroup = partitionInfo.getTaskIdInPartitionOrder();
    --- End diff --
   
    what is task id?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user QiangCai commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1125#discussion_r125955474
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/scan/filter/partition/PartitionFilterUtil.java ---
    @@ -229,35 +229,37 @@ public static BitSet getPartitionMapForRangeFilter(PartitionInfo partitionInfo,
           // filter value is in default partition
           if (isGreaterThan) {
             // GreaterThan(>), GreaterThanEqualTo(>=)
    -        partitionMap.set(numPartitions);
    +        partitionMap.set(0);
           } else {
             // LessThan(<), LessThanEqualTo(<=)
    -        partitionMap.set(0, partitioner.numPartitions());
    +        partitionMap.set(0, partitioner.numPartitions() - 1);
           }
         } else {
           // filter value is not in default partition
           if (result == 0) {
             // if result is 0, the filter value is a bound value of range partition.
             if (isGreaterThan) {
               // GreaterThan(>), GreaterThanEqualTo(>=)
    -          partitionMap.set(partitionIndex + 1, partitioner.numPartitions());
    +          partitionMap.set(partitionIndex + 2, partitioner.numPartitions() - 1);
    +          partitionMap.set(0);
             } else {
               if (isEqualTo) {
                 // LessThanEqualTo(<=)
    -            partitionMap.set(0, partitionIndex + 2);
    +            partitionMap.set(1, partitionIndex + 3);
               } else {
                 // LessThan(<)
    -            partitionMap.set(0, partitionIndex + 1);
    +            partitionMap.set(1, partitionIndex + 2);
               }
             }
           } else {
             // the filter value is not a bound value of range partition
             if (isGreaterThan) {
               // GreaterThan(>), GreaterThanEqualTo(>=)
    -          partitionMap.set(partitionIndex, partitioner.numPartitions());
    +          partitionMap.set(partitionIndex + 1, partitioner.numPartitions() - 1);
    --- End diff --
   
    why we lost 2 partitions


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user QiangCai commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1125#discussion_r125957701
 
    --- Diff: hadoop/src/main/java/org/apache/carbondata/hadoop/CarbonInputFormat.java ---
    @@ -274,28 +275,27 @@ private static AbsoluteTableIdentifier getAbsoluteTableIdentifier(Configuration
           CarbonInputFormatUtil.processFilterExpression(filter, carbonTable);
     
           // prune partitions for filter query on partition table
    -      BitSet matchedPartitions = null;
    -      if (null != filter) {
    -        PartitionInfo partitionInfo = carbonTable.getPartitionInfo(carbonTable.getFactTableName());
    -        if (null != partitionInfo) {
    -          Partitioner partitioner = PartitionUtil.getPartitioner(partitionInfo);
    -          matchedPartitions = new FilterExpressionProcessor()
    -              .getFilteredPartitions(filter, partitionInfo, partitioner);
    +      BitSet matchedPartitions = setMatchedPartitions(null, carbonTable, filter);
    +      if (partitionInfo != null) {
    +        Partitioner partitioner = PartitionUtil.getPartitioner(partitionInfo);
    --- End diff --
   
    this method was invoked twice


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user QiangCai commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1125#discussion_r125956098
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/util/comparator/Comparator.java ---
    @@ -0,0 +1,89 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.carbondata.core.util.comparator;
    +
    +import java.math.BigDecimal;
    +
    +import org.apache.carbondata.core.metadata.datatype.DataType;
    +import org.apache.carbondata.core.util.ByteUtil;
    +
    +public final class Comparator {
    +
    +  private static SerializableComparator comparator;
    --- End diff --
   
    no need


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user QiangCai commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1125#discussion_r125960197
 
    --- Diff: hadoop/src/main/java/org/apache/carbondata/hadoop/CarbonInputFormat.java ---
    @@ -327,6 +327,29 @@ private static AbsoluteTableIdentifier getAbsoluteTableIdentifier(Configuration
         return carbonSplits;
       }
     
    +  private BitSet setMatchedPartitions(String partitionIds, CarbonTable carbonTable,
    +      Expression filter) {
    +    BitSet matchedPartitions = null;
    +    if (null != partitionIds) {
    +      String[] partList = partitionIds.replace("[","").replace("]","").split(",");
    +      matchedPartitions = new BitSet(Integer.parseInt(partList[0]));
    +      for (String partitionId : partList) {
    +        matchedPartitions.set(Integer.parseInt(partitionId));
    +      }
    +    } else {
    +      if (null != filter) {
    +        PartitionInfo partitionInfo =
    +            carbonTable.getPartitionInfo(carbonTable.getFactTableName());
    +        if (null != partitionInfo) {
    +          Partitioner partitioner = PartitionUtil.getPartitioner(partitionInfo);
    +          matchedPartitions = new FilterExpressionProcessor()
    +              .getFilteredPartitions(filter, partitionInfo, partitioner);
    --- End diff --
   
    I think we need modify getFilteredPartitions and FilterExpression to use partition id instead of array index.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user QiangCai commented on the issue:

    https://github.com/apache/carbondata/pull/1125
 
    retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1125
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2935/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1125
 
    Build Failed with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/349/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user lionelcao commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1125#discussion_r126061892
 
    --- Diff: hadoop/src/main/java/org/apache/carbondata/hadoop/CarbonInputFormat.java ---
    @@ -417,18 +441,26 @@ protected Expression getFilterPredicates(Configuration configuration) {
               getSegmentAbstractIndexs(job, absoluteTableIdentifier, segmentId, cacheClient,
                   updateStatusManager);
           List<DataRefNode> resultFilterredBlocks = new LinkedList<DataRefNode>();
    +      int partitionIndex = -1;
    +      List<Integer> taskIdGroup = new ArrayList<>();
    +      if (partitionInfo != null) {
    +        taskIdGroup = partitionInfo.getTaskIdInPartitionOrder();
    --- End diff --
   
    Fixed.  it was missed when I change taskIdGroup to partitionIdList


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1125
 
    Build Failed with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/351/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1125
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2938/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user lionelcao commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1125#discussion_r126062042
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/metadata/schema/PartitionInfo.java ---
    @@ -43,13 +44,30 @@
       private List<List<String>> listInfo;
     
       /**
    -   * number of partitions
    +   * number of hash partitions
        */
    -  private int numPartitions;
    +  private int hashNumber;
    +
    +  /**
    +   * total count of partitions
    +   */
    +  private int numberOfPartitions;
    --- End diff --
   
    I think it represent different concept. Though they are equal in hash table. That would be more clear for developers.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user lionelcao commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1125#discussion_r126062119
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/metadata/schema/PartitionInfo.java ---
    @@ -43,13 +44,30 @@
       private List<List<String>> listInfo;
     
       /**
    -   * number of partitions
    +   * number of hash partitions
        */
    -  private int numPartitions;
    +  private int hashNumber;
    +
    +  /**
    +   * total count of partitions
    +   */
    +  private int numberOfPartitions;
    +
    +  /**
    +   * task id of max partition
    +   */
    +  private int MAX_PARTITION;
    --- End diff --
   
    MAX_PARTITION will be used in alter table partition. it's increased only.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1125
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2939/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---