[GitHub] carbondata pull request #1079: [WIP]Measure Filter implementation

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1079#discussion_r125152345
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/statistics/ColumnPageStatsVO.java ---
    @@ -102,9 +109,15 @@ public void update(Object value) {
             break;
           case DECIMAL:
             BigDecimal decimalValue = DataTypeUtil.byteToBigDecimal((byte[]) value);
    -        decimal = decimalValue.scale();
    -        BigDecimal val = (BigDecimal) min;
    -        nonExistValue = (val.subtract(new BigDecimal(1.0)));
    +        if (isFirst) {
    --- End diff --
   
    I think this `isFirst` is required. just check null for max or min here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1079#discussion_r125152368
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/metadata/schema/table/CarbonTable.java ---
    @@ -137,6 +137,8 @@
        */
       private int numberOfNoDictSortColumns;
     
    +  private int lastDimensionColumnOrdinal;
    --- End diff --
   
    Not used any where, please remove


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1079#discussion_r125152433
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/scan/expression/ColumnExpression.java ---
    @@ -31,12 +32,16 @@
     
       private boolean isDimension;
     
    +  private boolean isMeasure;
    --- End diff --
   
    same information is available inside `measure`, so please remove it and get it from `measure`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1079#discussion_r125152630
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/scan/filter/FilterUtil.java ---
    @@ -180,9 +185,27 @@ private static FilterExecuter createFilterExecuterTree(
        * @return
        */
       private static FilterExecuter getIncludeFilterExecuter(
    -      DimColumnResolvedFilterInfo dimColResolvedFilterInfo, SegmentProperties segmentProperties) {
    -
    -    if (dimColResolvedFilterInfo.getDimension().isColumnar()) {
    +      DimColumnResolvedFilterInfo dimColResolvedFilterInfo,
    +      MeasureColumnResolvedFilterInfo msrColResolvedFilterInfo,
    +      SegmentProperties segmentProperties) {
    +    if (null != msrColResolvedFilterInfo && msrColResolvedFilterInfo.getMeasure().isColumnar()) {
    --- End diff --
   
    I don't think `msrColResolvedFilterInfo.getMeasure().isColumnar()` is really required. it is only for dimensions.please remove


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1079#discussion_r125152663
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/scan/filter/FilterUtil.java ---
    @@ -180,9 +185,27 @@ private static FilterExecuter createFilterExecuterTree(
        * @return
        */
       private static FilterExecuter getIncludeFilterExecuter(
    -      DimColumnResolvedFilterInfo dimColResolvedFilterInfo, SegmentProperties segmentProperties) {
    -
    -    if (dimColResolvedFilterInfo.getDimension().isColumnar()) {
    +      DimColumnResolvedFilterInfo dimColResolvedFilterInfo,
    +      MeasureColumnResolvedFilterInfo msrColResolvedFilterInfo,
    +      SegmentProperties segmentProperties) {
    +    if (null != msrColResolvedFilterInfo && msrColResolvedFilterInfo.getMeasure().isColumnar()) {
    +      CarbonMeasure measuresFromCurrentBlock = segmentProperties
    +          .getMeasureFromCurrentBlock(msrColResolvedFilterInfo.getMeasure().getColumnId());
    +      if (null != measuresFromCurrentBlock) {
    +        // update dimension and column index according to the dimension position in current block
    +        MeasureColumnResolvedFilterInfo msrColResolvedFilterInfoCopyObject =
    +            msrColResolvedFilterInfo.getCopyObject();
    +        msrColResolvedFilterInfoCopyObject.setMeasure(measuresFromCurrentBlock);
    +        msrColResolvedFilterInfoCopyObject.setColumnIndex(measuresFromCurrentBlock.getOrdinal());
    +        msrColResolvedFilterInfoCopyObject.setType(measuresFromCurrentBlock.getDataType());
    +        return new IncludeFilterExecuterImpl(null, msrColResolvedFilterInfoCopyObject,
    +            segmentProperties, true);
    +      } else {
    +        return new RestructureIncludeFilterExecutorImpl(dimColResolvedFilterInfo,
    +            msrColResolvedFilterInfo, segmentProperties, true);
    +      }
    +    }
    --- End diff --
   
    isn't `else` should be here for `dimColResolvedFilterInfo` check


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1079#discussion_r125152709
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/scan/filter/FilterUtil.java ---
    @@ -209,9 +233,29 @@ private static FilterExecuter getIncludeFilterExecuter(
        * @return
        */
       private static FilterExecuter getExcludeFilterExecuter(
    -      DimColumnResolvedFilterInfo dimColResolvedFilterInfo, SegmentProperties segmentProperties) {
    +      DimColumnResolvedFilterInfo dimColResolvedFilterInfo,
    +      MeasureColumnResolvedFilterInfo msrColResolvedFilterInfo,
    +      SegmentProperties segmentProperties) {
     
    -    if (dimColResolvedFilterInfo.getDimension().isColumnar()) {
    +    if (null != msrColResolvedFilterInfo && msrColResolvedFilterInfo.getMeasure().isColumnar()) {
    --- End diff --
   
    even here `isColumnar` check not required


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1079#discussion_r125152824
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/scan/filter/FilterUtil.java ---
    @@ -395,6 +440,58 @@ public static DimColumnFilterInfo getNoDictionaryValKeyMemberForFilter(
       }
     
       /**
    +   * This method will get the no dictionary data based on filters and same
    +   * will be in ColumnFilterInfo
    +   *
    +   * @param evaluateResultListFinal
    +   * @param isIncludeFilter
    +   * @return ColumnFilterInfo
    +   */
    +  public static ColumnFilterInfo getMeasureValKeyMemberForFilter(
    +      List<String> evaluateResultListFinal, boolean isIncludeFilter, DataType dataType,
    +      CarbonMeasure carbonMeasure) throws FilterUnsupportedException {
    +    List<byte[]> filterValuesList = new ArrayList<byte[]>(20);
    +    String result = null;
    +    try {
    +      int length = evaluateResultListFinal.size();
    +      for (int i = 0; i < length; i++) {
    +        result = evaluateResultListFinal.get(i);
    +        if (CarbonCommonConstants.MEMBER_DEFAULT_VAL.equals(result)) {
    +          filterValuesList.add(new byte[0]);
    +          continue;
    +        }
    +        // TODO have to understand what method to be used for measures.
    +        // filterValuesList
    +        //  .add(DataTypeUtil.getBytesBasedOnDataTypeForNoDictionaryColumn(result, dataType));
    --- End diff --
   
    is this comment required now? please remove if not required


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1079#discussion_r125152873
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/scan/filter/FilterUtil.java ---
    @@ -395,6 +440,58 @@ public static DimColumnFilterInfo getNoDictionaryValKeyMemberForFilter(
       }
     
       /**
    +   * This method will get the no dictionary data based on filters and same
    +   * will be in ColumnFilterInfo
    +   *
    +   * @param evaluateResultListFinal
    +   * @param isIncludeFilter
    +   * @return ColumnFilterInfo
    +   */
    +  public static ColumnFilterInfo getMeasureValKeyMemberForFilter(
    +      List<String> evaluateResultListFinal, boolean isIncludeFilter, DataType dataType,
    +      CarbonMeasure carbonMeasure) throws FilterUnsupportedException {
    +    List<byte[]> filterValuesList = new ArrayList<byte[]>(20);
    +    String result = null;
    +    try {
    +      int length = evaluateResultListFinal.size();
    +      for (int i = 0; i < length; i++) {
    +        result = evaluateResultListFinal.get(i);
    +        if (CarbonCommonConstants.MEMBER_DEFAULT_VAL.equals(result)) {
    +          filterValuesList.add(new byte[0]);
    +          continue;
    +        }
    +        // TODO have to understand what method to be used for measures.
    +        // filterValuesList
    +        //  .add(DataTypeUtil.getBytesBasedOnDataTypeForNoDictionaryColumn(result, dataType));
    +
    +        filterValuesList
    +            .add(DataTypeUtil.getMeasureByteArrayBasedOnDataTypes(result, dataType, carbonMeasure));
    +
    +      }
    +    } catch (Throwable ex) {
    +      throw new FilterUnsupportedException("Unsupported Filter condition: " + result, ex);
    +    }
    +
    +    Comparator<byte[]> filterMeasureComaparator = new Comparator<byte[]>() {
    +
    +      @Override public int compare(byte[] filterMember1, byte[] filterMember2) {
    +        // TODO Auto-generated method stub
    +        return ByteUtil.UnsafeComparer.INSTANCE.compareTo(filterMember1, filterMember2);
    --- End diff --
   
    this is wrong, we cannot compare `double,float,decimal` through bytes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1079#discussion_r125152881
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/scan/filter/FilterUtil.java ---
    @@ -395,6 +440,58 @@ public static DimColumnFilterInfo getNoDictionaryValKeyMemberForFilter(
       }
     
       /**
    +   * This method will get the no dictionary data based on filters and same
    +   * will be in ColumnFilterInfo
    +   *
    +   * @param evaluateResultListFinal
    +   * @param isIncludeFilter
    +   * @return ColumnFilterInfo
    +   */
    +  public static ColumnFilterInfo getMeasureValKeyMemberForFilter(
    +      List<String> evaluateResultListFinal, boolean isIncludeFilter, DataType dataType,
    +      CarbonMeasure carbonMeasure) throws FilterUnsupportedException {
    +    List<byte[]> filterValuesList = new ArrayList<byte[]>(20);
    +    String result = null;
    +    try {
    +      int length = evaluateResultListFinal.size();
    +      for (int i = 0; i < length; i++) {
    +        result = evaluateResultListFinal.get(i);
    +        if (CarbonCommonConstants.MEMBER_DEFAULT_VAL.equals(result)) {
    +          filterValuesList.add(new byte[0]);
    +          continue;
    +        }
    +        // TODO have to understand what method to be used for measures.
    +        // filterValuesList
    +        //  .add(DataTypeUtil.getBytesBasedOnDataTypeForNoDictionaryColumn(result, dataType));
    +
    +        filterValuesList
    +            .add(DataTypeUtil.getMeasureByteArrayBasedOnDataTypes(result, dataType, carbonMeasure));
    +
    +      }
    +    } catch (Throwable ex) {
    +      throw new FilterUnsupportedException("Unsupported Filter condition: " + result, ex);
    +    }
    +
    +    Comparator<byte[]> filterMeasureComaparator = new Comparator<byte[]>() {
    +
    +      @Override public int compare(byte[] filterMember1, byte[] filterMember2) {
    +        // TODO Auto-generated method stub
    +        return ByteUtil.UnsafeComparer.INSTANCE.compareTo(filterMember1, filterMember2);
    --- End diff --
   
    Please compare actual values before convert to binary


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1079#discussion_r125152956
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/scan/filter/FilterUtil.java ---
    @@ -1042,12 +1144,17 @@ public static FilterExecuter getFilterExecuterTree(
        * @param dimension
        * @param dimColumnExecuterInfo
        */
    -  public static void prepareKeysFromSurrogates(DimColumnFilterInfo filterValues,
    +  public static void prepareKeysFromSurrogates(ColumnFilterInfo filterValues,
           SegmentProperties segmentProperties, CarbonDimension dimension,
    -      DimColumnExecuterFilterInfo dimColumnExecuterInfo) {
    -    byte[][] keysBasedOnFilter = getKeyArray(filterValues, dimension, segmentProperties);
    -    dimColumnExecuterInfo.setFilterKeys(keysBasedOnFilter);
    -
    +      DimColumnExecuterFilterInfo dimColumnExecuterInfo, CarbonMeasure measures,
    +      MeasureColumnExecuterFilterInfo msrColumnExecuterInfo) {
    +    if (null != measures) {
    --- End diff --
   
    I don't think this `if ` check is required. just pass dimension and measure to the method 'getKeyArray'


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1079#discussion_r125152979
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/scan/filter/executer/ExcludeFilterExecuterImpl.java ---
    @@ -18,56 +18,152 @@
     
     import java.io.IOException;
     import java.util.BitSet;
    +import java.util.Comparator;
     
     import org.apache.carbondata.core.datastore.block.SegmentProperties;
     import org.apache.carbondata.core.datastore.chunk.DimensionColumnDataChunk;
    +import org.apache.carbondata.core.datastore.chunk.MeasureColumnDataChunk;
     import org.apache.carbondata.core.datastore.chunk.impl.DimensionRawColumnChunk;
    +import org.apache.carbondata.core.datastore.chunk.impl.MeasureRawColumnChunk;
    +import org.apache.carbondata.core.metadata.datatype.DataType;
     import org.apache.carbondata.core.scan.filter.FilterUtil;
    +import org.apache.carbondata.core.scan.filter.partition.PartitionFilterUtil;
     import org.apache.carbondata.core.scan.filter.resolver.resolverinfo.DimColumnResolvedFilterInfo;
    +import org.apache.carbondata.core.scan.filter.resolver.resolverinfo.MeasureColumnResolvedFilterInfo;
     import org.apache.carbondata.core.scan.processor.BlocksChunkHolder;
     import org.apache.carbondata.core.util.BitSetGroup;
     import org.apache.carbondata.core.util.CarbonUtil;
    +import org.apache.carbondata.core.util.DataTypeUtil;
     
     public class ExcludeFilterExecuterImpl implements FilterExecuter {
     
       protected DimColumnResolvedFilterInfo dimColEvaluatorInfo;
       protected DimColumnExecuterFilterInfo dimColumnExecuterInfo;
    +  protected MeasureColumnResolvedFilterInfo msrColumnEvaluatorInfo;
    +  protected MeasureColumnExecuterFilterInfo msrColumnExecutorInfo;
       protected SegmentProperties segmentProperties;
    +  protected boolean isDimensionPresentInCurrentBlock = false;
    +  protected boolean isMeasurePresentInCurrentBlock = false;
       /**
        * is dimension column data is natural sorted
        */
    -  private boolean isNaturalSorted;
    +  private boolean isNaturalSorted = false;
    +
       public ExcludeFilterExecuterImpl(DimColumnResolvedFilterInfo dimColEvaluatorInfo,
    -      SegmentProperties segmentProperties) {
    -    this.dimColEvaluatorInfo = dimColEvaluatorInfo;
    -    dimColumnExecuterInfo = new DimColumnExecuterFilterInfo();
    +      MeasureColumnResolvedFilterInfo msrColumnEvaluatorInfo, SegmentProperties segmentProperties,
    +      boolean isMeasure) {
         this.segmentProperties = segmentProperties;
    -    FilterUtil.prepareKeysFromSurrogates(dimColEvaluatorInfo.getFilterValues(), segmentProperties,
    -        dimColEvaluatorInfo.getDimension(), dimColumnExecuterInfo);
    -    isNaturalSorted = dimColEvaluatorInfo.getDimension().isUseInvertedIndex() && dimColEvaluatorInfo
    -        .getDimension().isSortColumn();
    +    if (isMeasure == false) {
    --- End diff --
   
    just use `!isMeasure`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1079#discussion_r125153056
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/scan/filter/executer/ExcludeFilterExecuterImpl.java ---
    @@ -18,56 +18,152 @@
     
     import java.io.IOException;
     import java.util.BitSet;
    +import java.util.Comparator;
     
     import org.apache.carbondata.core.datastore.block.SegmentProperties;
     import org.apache.carbondata.core.datastore.chunk.DimensionColumnDataChunk;
    +import org.apache.carbondata.core.datastore.chunk.MeasureColumnDataChunk;
     import org.apache.carbondata.core.datastore.chunk.impl.DimensionRawColumnChunk;
    +import org.apache.carbondata.core.datastore.chunk.impl.MeasureRawColumnChunk;
    +import org.apache.carbondata.core.metadata.datatype.DataType;
     import org.apache.carbondata.core.scan.filter.FilterUtil;
    +import org.apache.carbondata.core.scan.filter.partition.PartitionFilterUtil;
     import org.apache.carbondata.core.scan.filter.resolver.resolverinfo.DimColumnResolvedFilterInfo;
    +import org.apache.carbondata.core.scan.filter.resolver.resolverinfo.MeasureColumnResolvedFilterInfo;
     import org.apache.carbondata.core.scan.processor.BlocksChunkHolder;
     import org.apache.carbondata.core.util.BitSetGroup;
     import org.apache.carbondata.core.util.CarbonUtil;
    +import org.apache.carbondata.core.util.DataTypeUtil;
     
     public class ExcludeFilterExecuterImpl implements FilterExecuter {
     
       protected DimColumnResolvedFilterInfo dimColEvaluatorInfo;
       protected DimColumnExecuterFilterInfo dimColumnExecuterInfo;
    +  protected MeasureColumnResolvedFilterInfo msrColumnEvaluatorInfo;
    +  protected MeasureColumnExecuterFilterInfo msrColumnExecutorInfo;
       protected SegmentProperties segmentProperties;
    +  protected boolean isDimensionPresentInCurrentBlock = false;
    +  protected boolean isMeasurePresentInCurrentBlock = false;
    --- End diff --
   
    I don't think all these flags are required. just do `null` check of  `dimColumnExecuterInfo` and  `msrColumnExecutorInfo` is enough


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1079#discussion_r125153179
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/util/DataTypeUtil.java ---
    @@ -113,6 +115,143 @@ public static Object getMeasureValueBasedOnDataType(String msrValue, DataType da
         }
       }
     
    +  public static Object getMeasureObjectFromDataType(byte[] data, DataType dataType) {
    +    ByteBuffer bb = ByteBuffer.wrap(data);
    --- End diff --
   
    This is unnecessary object for `decimal` so keep it in respective case statement


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1079#discussion_r125153193
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/util/DataTypeUtil.java ---
    @@ -113,6 +115,143 @@ public static Object getMeasureValueBasedOnDataType(String msrValue, DataType da
         }
       }
     
    +  public static Object getMeasureObjectFromDataType(byte[] data, DataType dataType) {
    +    ByteBuffer bb = ByteBuffer.wrap(data);
    +    switch (dataType) {
    +      case SHORT:
    +      case INT:
    +      case LONG:
    +        return bb.getLong();
    +      case DECIMAL:
    +        return byteToBigDecimal(data);
    +      default:
    +        return bb.getDouble();
    +    }
    +  }
    +
    +  /**
    +   * This method will convert a given ByteArray to its specific type
    +   *
    +   * @param msrValue
    +   * @param dataType
    +   * @param carbonMeasure
    +   * @return
    +   */
    +  //  public static byte[] getMeasureByteArrayBasedOnDataType(String msrValue, DataType dataType,
    +  //      CarbonMeasure carbonMeasure) {
    +  //    switch (dataType) {
    +  //      case DECIMAL:
    +  //        BigDecimal bigDecimal =
    +  //            new BigDecimal(msrValue).setScale(carbonMeasure.getScale(), RoundingMode.HALF_UP);
    +  //       return ByteUtil.toBytes(normalizeDecimalValue(bigDecimal, carbonMeasure.getPrecision()));
    +  //      case SHORT:
    +  //        return ByteUtil.toBytes((Short.parseShort(msrValue)));
    +  //      case INT:
    +  //        return ByteUtil.toBytes(Integer.parseInt(msrValue));
    +  //      case LONG:
    +  //        return ByteUtil.toBytes(Long.valueOf(msrValue));
    +  //      default:
    +  //        Double parsedValue = Double.valueOf(msrValue);
    +  //        if (Double.isInfinite(parsedValue) || Double.isNaN(parsedValue)) {
    +  //          return null;
    +  //        }
    +  //        return ByteUtil.toBytes(parsedValue);
    +  //    }
    +  //  }
    --- End diff --
   
    remove the commented code


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1079#discussion_r125153305
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/util/DataTypeUtil.java ---
    @@ -113,6 +115,143 @@ public static Object getMeasureValueBasedOnDataType(String msrValue, DataType da
         }
       }
     
    +  public static Object getMeasureObjectFromDataType(byte[] data, DataType dataType) {
    +    ByteBuffer bb = ByteBuffer.wrap(data);
    +    switch (dataType) {
    +      case SHORT:
    +      case INT:
    +      case LONG:
    +        return bb.getLong();
    +      case DECIMAL:
    +        return byteToBigDecimal(data);
    +      default:
    +        return bb.getDouble();
    +    }
    +  }
    +
    +  /**
    +   * This method will convert a given ByteArray to its specific type
    +   *
    +   * @param msrValue
    +   * @param dataType
    +   * @param carbonMeasure
    +   * @return
    +   */
    +  //  public static byte[] getMeasureByteArrayBasedOnDataType(String msrValue, DataType dataType,
    +  //      CarbonMeasure carbonMeasure) {
    +  //    switch (dataType) {
    +  //      case DECIMAL:
    +  //        BigDecimal bigDecimal =
    +  //            new BigDecimal(msrValue).setScale(carbonMeasure.getScale(), RoundingMode.HALF_UP);
    +  //       return ByteUtil.toBytes(normalizeDecimalValue(bigDecimal, carbonMeasure.getPrecision()));
    +  //      case SHORT:
    +  //        return ByteUtil.toBytes((Short.parseShort(msrValue)));
    +  //      case INT:
    +  //        return ByteUtil.toBytes(Integer.parseInt(msrValue));
    +  //      case LONG:
    +  //        return ByteUtil.toBytes(Long.valueOf(msrValue));
    +  //      default:
    +  //        Double parsedValue = Double.valueOf(msrValue);
    +  //        if (Double.isInfinite(parsedValue) || Double.isNaN(parsedValue)) {
    +  //          return null;
    +  //        }
    +  //        return ByteUtil.toBytes(parsedValue);
    +  //    }
    +  //  }
    +  public static byte[] getMeasureByteArrayBasedOnDataTypes(String msrValue, DataType dataType,
    +      CarbonMeasure carbonMeasure) {
    +    ByteBuffer b;
    +    switch (dataType) {
    +      case BYTE:
    +      case SHORT:
    +      case INT:
    +      case LONG:
    +        b = ByteBuffer.allocate(8);
    +        b.putLong(Long.valueOf(msrValue));
    +        b.flip();
    +        return b.array();
    +      case DOUBLE:
    +        b = ByteBuffer.allocate(8);
    +        b.putDouble(Double.valueOf(msrValue));
    +        b.flip();
    +        return b.array();
    +      case DECIMAL:
    +        BigDecimal bigDecimal =
    +            new BigDecimal(msrValue).setScale(carbonMeasure.getScale(), RoundingMode.HALF_UP);
    +        return DataTypeUtil
    +            .bigDecimalToByte(normalizeDecimalValue(bigDecimal, carbonMeasure.getPrecision()));
    +      default:
    +        throw new IllegalArgumentException("Invalid data type: " + dataType);
    +    }
    +  }
    +
    +  /**
    +   * This method will convert a given ByteArray to its specific type
    +   *
    +   * @param msrValue
    +   * @param dataType
    +   * @param carbonMeasure
    +   * @return
    +   */
    +  public static byte[] getMeasureByteArrayBasedOnDataType(ColumnPage measurePage, int index,
    --- End diff --
   
    This method is not used, please remove it


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1079#discussion_r125153317
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/util/DataTypeUtil.java ---
    @@ -113,6 +115,143 @@ public static Object getMeasureValueBasedOnDataType(String msrValue, DataType da
         }
       }
     
    +  public static Object getMeasureObjectFromDataType(byte[] data, DataType dataType) {
    +    ByteBuffer bb = ByteBuffer.wrap(data);
    +    switch (dataType) {
    +      case SHORT:
    +      case INT:
    +      case LONG:
    +        return bb.getLong();
    +      case DECIMAL:
    +        return byteToBigDecimal(data);
    +      default:
    +        return bb.getDouble();
    +    }
    +  }
    +
    +  /**
    +   * This method will convert a given ByteArray to its specific type
    +   *
    +   * @param msrValue
    +   * @param dataType
    +   * @param carbonMeasure
    +   * @return
    +   */
    +  //  public static byte[] getMeasureByteArrayBasedOnDataType(String msrValue, DataType dataType,
    +  //      CarbonMeasure carbonMeasure) {
    +  //    switch (dataType) {
    +  //      case DECIMAL:
    +  //        BigDecimal bigDecimal =
    +  //            new BigDecimal(msrValue).setScale(carbonMeasure.getScale(), RoundingMode.HALF_UP);
    +  //       return ByteUtil.toBytes(normalizeDecimalValue(bigDecimal, carbonMeasure.getPrecision()));
    +  //      case SHORT:
    +  //        return ByteUtil.toBytes((Short.parseShort(msrValue)));
    +  //      case INT:
    +  //        return ByteUtil.toBytes(Integer.parseInt(msrValue));
    +  //      case LONG:
    +  //        return ByteUtil.toBytes(Long.valueOf(msrValue));
    +  //      default:
    +  //        Double parsedValue = Double.valueOf(msrValue);
    +  //        if (Double.isInfinite(parsedValue) || Double.isNaN(parsedValue)) {
    +  //          return null;
    +  //        }
    +  //        return ByteUtil.toBytes(parsedValue);
    +  //    }
    +  //  }
    +  public static byte[] getMeasureByteArrayBasedOnDataTypes(String msrValue, DataType dataType,
    +      CarbonMeasure carbonMeasure) {
    +    ByteBuffer b;
    +    switch (dataType) {
    +      case BYTE:
    +      case SHORT:
    +      case INT:
    +      case LONG:
    +        b = ByteBuffer.allocate(8);
    +        b.putLong(Long.valueOf(msrValue));
    +        b.flip();
    +        return b.array();
    +      case DOUBLE:
    +        b = ByteBuffer.allocate(8);
    +        b.putDouble(Double.valueOf(msrValue));
    +        b.flip();
    +        return b.array();
    +      case DECIMAL:
    +        BigDecimal bigDecimal =
    +            new BigDecimal(msrValue).setScale(carbonMeasure.getScale(), RoundingMode.HALF_UP);
    +        return DataTypeUtil
    +            .bigDecimalToByte(normalizeDecimalValue(bigDecimal, carbonMeasure.getPrecision()));
    +      default:
    +        throw new IllegalArgumentException("Invalid data type: " + dataType);
    +    }
    +  }
    +
    +  /**
    +   * This method will convert a given ByteArray to its specific type
    +   *
    +   * @param msrValue
    +   * @param dataType
    +   * @param carbonMeasure
    +   * @return
    +   */
    +  public static byte[] getMeasureByteArrayBasedOnDataType(ColumnPage measurePage, int index,
    +      DataType dataType, CarbonMeasure carbonMeasure) {
    +    switch (dataType) {
    +      case DECIMAL:
    +        BigDecimal bigDecimal = new BigDecimal(measurePage.getDouble(index))
    +            .setScale(carbonMeasure.getScale(), RoundingMode.HALF_UP);
    +        return ByteUtil.toBytes(normalizeDecimalValue(bigDecimal, carbonMeasure.getPrecision()));
    +      case SHORT:
    +        return ByteUtil.toBytes(measurePage.getShort(index));
    +      case INT:
    +        return ByteUtil.toBytes(measurePage.getInt(index));
    +      case LONG:
    +        return ByteUtil.toBytes(measurePage.getLong(index));
    +      default:
    +        Double parsedValue = Double.valueOf(measurePage.getDouble(index));
    +        if (Double.isInfinite(parsedValue) || Double.isNaN(parsedValue)) {
    +          return null;
    +        }
    +        return ByteUtil.toBytes(parsedValue);
    +    }
    +  }
    +
    +  public static Object getMeasureObjectBasedOnDataType(ColumnPage measurePage, int index,
    +      DataType dataType, CarbonMeasure carbonMeasure) {
    +    //    switch (dataType) {
    +    //      case DECIMAL:
    +    //        BigDecimal bigDecimal = new BigDecimal(measurePage.getDouble(index))
    +    //            .setScale(carbonMeasure.getScale(), RoundingMode.HALF_UP);
    +    //        return normalizeDecimalValue(bigDecimal, carbonMeasure.getPrecision());
    +    //      case SHORT:
    +    //      case INT:
    +    //      case LONG:
    +    //        return measurePage.getLong(index);
    +    //      default:
    +    //        Double parsedValue = Double.valueOf(measurePage.getDouble(index));
    +    //        if (Double.isInfinite(parsedValue) || Double.isNaN(parsedValue)) {
    +    //          return null;
    +    //        }
    +    //        return parsedValue;
    +    //    }
    --- End diff --
   
    Please remove commented code


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1079#discussion_r125153423
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/scan/filter/FilterUtil.java ---
    @@ -395,6 +440,58 @@ public static DimColumnFilterInfo getNoDictionaryValKeyMemberForFilter(
       }
     
       /**
    +   * This method will get the no dictionary data based on filters and same
    +   * will be in ColumnFilterInfo
    +   *
    +   * @param evaluateResultListFinal
    +   * @param isIncludeFilter
    +   * @return ColumnFilterInfo
    +   */
    +  public static ColumnFilterInfo getMeasureValKeyMemberForFilter(
    +      List<String> evaluateResultListFinal, boolean isIncludeFilter, DataType dataType,
    +      CarbonMeasure carbonMeasure) throws FilterUnsupportedException {
    +    List<byte[]> filterValuesList = new ArrayList<byte[]>(20);
    +    String result = null;
    +    try {
    +      int length = evaluateResultListFinal.size();
    +      for (int i = 0; i < length; i++) {
    +        result = evaluateResultListFinal.get(i);
    +        if (CarbonCommonConstants.MEMBER_DEFAULT_VAL.equals(result)) {
    +          filterValuesList.add(new byte[0]);
    +          continue;
    +        }
    +        // TODO have to understand what method to be used for measures.
    +        // filterValuesList
    +        //  .add(DataTypeUtil.getBytesBasedOnDataTypeForNoDictionaryColumn(result, dataType));
    +
    +        filterValuesList
    +            .add(DataTypeUtil.getMeasureByteArrayBasedOnDataTypes(result, dataType, carbonMeasure));
    --- End diff --
   
    I believe we should directly keep objects without converting to binary for measures. It will avoid conversions later


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1079#discussion_r125153644
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/scan/filter/executer/ExcludeFilterExecuterImpl.java ---
    @@ -18,56 +18,152 @@
     
     import java.io.IOException;
     import java.util.BitSet;
    +import java.util.Comparator;
     
     import org.apache.carbondata.core.datastore.block.SegmentProperties;
     import org.apache.carbondata.core.datastore.chunk.DimensionColumnDataChunk;
    +import org.apache.carbondata.core.datastore.chunk.MeasureColumnDataChunk;
     import org.apache.carbondata.core.datastore.chunk.impl.DimensionRawColumnChunk;
    +import org.apache.carbondata.core.datastore.chunk.impl.MeasureRawColumnChunk;
    +import org.apache.carbondata.core.metadata.datatype.DataType;
     import org.apache.carbondata.core.scan.filter.FilterUtil;
    +import org.apache.carbondata.core.scan.filter.partition.PartitionFilterUtil;
     import org.apache.carbondata.core.scan.filter.resolver.resolverinfo.DimColumnResolvedFilterInfo;
    +import org.apache.carbondata.core.scan.filter.resolver.resolverinfo.MeasureColumnResolvedFilterInfo;
     import org.apache.carbondata.core.scan.processor.BlocksChunkHolder;
     import org.apache.carbondata.core.util.BitSetGroup;
     import org.apache.carbondata.core.util.CarbonUtil;
    +import org.apache.carbondata.core.util.DataTypeUtil;
     
     public class ExcludeFilterExecuterImpl implements FilterExecuter {
     
       protected DimColumnResolvedFilterInfo dimColEvaluatorInfo;
       protected DimColumnExecuterFilterInfo dimColumnExecuterInfo;
    +  protected MeasureColumnResolvedFilterInfo msrColumnEvaluatorInfo;
    +  protected MeasureColumnExecuterFilterInfo msrColumnExecutorInfo;
       protected SegmentProperties segmentProperties;
    +  protected boolean isDimensionPresentInCurrentBlock = false;
    +  protected boolean isMeasurePresentInCurrentBlock = false;
       /**
        * is dimension column data is natural sorted
        */
    -  private boolean isNaturalSorted;
    +  private boolean isNaturalSorted = false;
    +
       public ExcludeFilterExecuterImpl(DimColumnResolvedFilterInfo dimColEvaluatorInfo,
    -      SegmentProperties segmentProperties) {
    -    this.dimColEvaluatorInfo = dimColEvaluatorInfo;
    -    dimColumnExecuterInfo = new DimColumnExecuterFilterInfo();
    +      MeasureColumnResolvedFilterInfo msrColumnEvaluatorInfo, SegmentProperties segmentProperties,
    +      boolean isMeasure) {
         this.segmentProperties = segmentProperties;
    -    FilterUtil.prepareKeysFromSurrogates(dimColEvaluatorInfo.getFilterValues(), segmentProperties,
    -        dimColEvaluatorInfo.getDimension(), dimColumnExecuterInfo);
    -    isNaturalSorted = dimColEvaluatorInfo.getDimension().isUseInvertedIndex() && dimColEvaluatorInfo
    -        .getDimension().isSortColumn();
    +    if (isMeasure == false) {
    +      this.dimColEvaluatorInfo = dimColEvaluatorInfo;
    +      dimColumnExecuterInfo = new DimColumnExecuterFilterInfo();
    +
    +      FilterUtil.prepareKeysFromSurrogates(dimColEvaluatorInfo.getFilterValues(), segmentProperties,
    +          dimColEvaluatorInfo.getDimension(), dimColumnExecuterInfo, null, null);
    +      isDimensionPresentInCurrentBlock = true;
    +      isNaturalSorted =
    +          dimColEvaluatorInfo.getDimension().isUseInvertedIndex() && dimColEvaluatorInfo
    +              .getDimension().isSortColumn();
    +    } else {
    +      this.msrColumnEvaluatorInfo = msrColumnEvaluatorInfo;
    +      msrColumnExecutorInfo = new MeasureColumnExecuterFilterInfo();
    +      FilterUtil
    +          .prepareKeysFromSurrogates(msrColumnEvaluatorInfo.getFilterValues(), segmentProperties,
    +              null, null, msrColumnEvaluatorInfo.getMeasure(), msrColumnExecutorInfo);
    +      isMeasurePresentInCurrentBlock = true;
    +    }
    +
       }
     
       @Override public BitSetGroup applyFilter(BlocksChunkHolder blockChunkHolder) throws IOException {
    -    int blockIndex = segmentProperties.getDimensionOrdinalToBlockMapping()
    -        .get(dimColEvaluatorInfo.getColumnIndex());
    -    if (null == blockChunkHolder.getDimensionRawDataChunk()[blockIndex]) {
    -      blockChunkHolder.getDimensionRawDataChunk()[blockIndex] = blockChunkHolder.getDataBlock()
    -          .getDimensionChunk(blockChunkHolder.getFileReader(), blockIndex);
    +    if (isDimensionPresentInCurrentBlock == true) {
    +      int blockIndex = segmentProperties.getDimensionOrdinalToBlockMapping()
    +          .get(dimColEvaluatorInfo.getColumnIndex());
    +      if (null == blockChunkHolder.getDimensionRawDataChunk()[blockIndex]) {
    +        blockChunkHolder.getDimensionRawDataChunk()[blockIndex] = blockChunkHolder.getDataBlock()
    +            .getDimensionChunk(blockChunkHolder.getFileReader(), blockIndex);
    +      }
    +      DimensionRawColumnChunk dimensionRawColumnChunk =
    +          blockChunkHolder.getDimensionRawDataChunk()[blockIndex];
    +      DimensionColumnDataChunk[] dimensionColumnDataChunks =
    +          dimensionRawColumnChunk.convertToDimColDataChunks();
    +      BitSetGroup bitSetGroup = new BitSetGroup(dimensionRawColumnChunk.getPagesCount());
    +      for (int i = 0; i < dimensionColumnDataChunks.length; i++) {
    +        BitSet bitSet = getFilteredIndexes(dimensionColumnDataChunks[i],
    +            dimensionRawColumnChunk.getRowCount()[i]);
    +        bitSetGroup.setBitSet(bitSet, i);
    +      }
    +
    +      return bitSetGroup;
    +    } else if (isMeasurePresentInCurrentBlock == true) {
    +      int blockIndex = segmentProperties.getMeasuresOrdinalToBlockMapping()
    +          .get(msrColumnEvaluatorInfo.getColumnIndex());
    +      if (null == blockChunkHolder.getMeasureRawDataChunk()[blockIndex]) {
    +        blockChunkHolder.getMeasureRawDataChunk()[blockIndex] = blockChunkHolder.getDataBlock()
    +            .getMeasureChunk(blockChunkHolder.getFileReader(), blockIndex);
    +      }
    +      MeasureRawColumnChunk measureRawColumnChunk =
    +          blockChunkHolder.getMeasureRawDataChunk()[blockIndex];
    +      MeasureColumnDataChunk[] measureColumnDataChunks =
    +          measureRawColumnChunk.convertToMeasureColDataChunks();
    +      BitSetGroup bitSetGroup = new BitSetGroup(measureRawColumnChunk.getPagesCount());
    +      DataType msrType = getMeasureDataType(msrColumnEvaluatorInfo);
    +      for (int i = 0; i < measureColumnDataChunks.length; i++) {
    +        BitSet bitSet =
    +            getFilteredIndexes(measureColumnDataChunks[i], measureRawColumnChunk.getRowCount()[i],
    +                msrType);
    +        bitSetGroup.setBitSet(bitSet, i);
    +      }
    +      return bitSetGroup;
         }
    -    DimensionRawColumnChunk dimensionRawColumnChunk =
    -        blockChunkHolder.getDimensionRawDataChunk()[blockIndex];
    -    DimensionColumnDataChunk[] dimensionColumnDataChunks =
    -        dimensionRawColumnChunk.convertToDimColDataChunks();
    -    BitSetGroup bitSetGroup =
    -        new BitSetGroup(dimensionRawColumnChunk.getPagesCount());
    -    for (int i = 0; i < dimensionColumnDataChunks.length; i++) {
    -      BitSet bitSet = getFilteredIndexes(dimensionColumnDataChunks[i],
    -          dimensionRawColumnChunk.getRowCount()[i]);
    -      bitSetGroup.setBitSet(bitSet, i);
    +    return null;
    +  }
    +
    +  private DataType getMeasureDataType(MeasureColumnResolvedFilterInfo msrColumnEvaluatorInfo) {
    +    switch (msrColumnEvaluatorInfo.getType()) {
    +      case SHORT:
    +        return DataType.SHORT;
    +      case INT:
    +        return DataType.INT;
    +      case LONG:
    +        return DataType.LONG;
    +      case DECIMAL:
    +        return DataType.DECIMAL;
    +      default:
    +        return DataType.DOUBLE;
         }
    +  }
     
    -    return bitSetGroup;
    +  protected BitSet getFilteredIndexes(MeasureColumnDataChunk measureColumnDataChunk,
    +      int numerOfRows, DataType msrType) {
    +    // Here the algorithm is
    +    // Get the measure values from the chunk. compare sequentially with the
    +    // the filter values. The one that matches sets it Bitset.
    +    BitSet bitSet = new BitSet(numerOfRows);
    +    bitSet.flip(0, numerOfRows);
    +    byte[][] filterValues = msrColumnExecutorInfo.getFilterKeys();
    --- End diff --
   
    better set objects to `msrColumnExecutorInfo` and use objects with out converting


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1079#discussion_r125153654
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/scan/filter/executer/IncludeFilterExecuterImpl.java ---
    @@ -17,65 +17,174 @@
     package org.apache.carbondata.core.scan.filter.executer;
     
     import java.io.IOException;
    +import java.math.BigDecimal;
    +import java.nio.ByteBuffer;
     import java.util.BitSet;
    +import java.util.Comparator;
     
     import org.apache.carbondata.core.datastore.block.SegmentProperties;
     import org.apache.carbondata.core.datastore.chunk.DimensionColumnDataChunk;
    +import org.apache.carbondata.core.datastore.chunk.MeasureColumnDataChunk;
     import org.apache.carbondata.core.datastore.chunk.impl.DimensionRawColumnChunk;
    +import org.apache.carbondata.core.datastore.chunk.impl.MeasureRawColumnChunk;
    +import org.apache.carbondata.core.metadata.datatype.DataType;
     import org.apache.carbondata.core.scan.filter.FilterUtil;
    +import org.apache.carbondata.core.scan.filter.partition.PartitionFilterUtil;
     import org.apache.carbondata.core.scan.filter.resolver.resolverinfo.DimColumnResolvedFilterInfo;
    +import org.apache.carbondata.core.scan.filter.resolver.resolverinfo.MeasureColumnResolvedFilterInfo;
     import org.apache.carbondata.core.scan.processor.BlocksChunkHolder;
     import org.apache.carbondata.core.util.BitSetGroup;
     import org.apache.carbondata.core.util.ByteUtil;
     import org.apache.carbondata.core.util.CarbonUtil;
    +import org.apache.carbondata.core.util.DataTypeUtil;
     
     public class IncludeFilterExecuterImpl implements FilterExecuter {
     
       protected DimColumnResolvedFilterInfo dimColumnEvaluatorInfo;
       protected DimColumnExecuterFilterInfo dimColumnExecuterInfo;
    +  protected MeasureColumnResolvedFilterInfo msrColumnEvaluatorInfo;
    +  protected MeasureColumnExecuterFilterInfo msrColumnExecutorInfo;
       protected SegmentProperties segmentProperties;
    +  protected boolean isDimensionPresentInCurrentBlock = false;
    +  protected boolean isMeasurePresentInCurrentBlock = false;
       /**
        * is dimension column data is natural sorted
        */
    -  private boolean isNaturalSorted;
    +  private boolean isNaturalSorted = false;
    --- End diff --
   
    default is false only right, no need to add


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1079#discussion_r125153674
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/scan/filter/executer/IncludeFilterExecuterImpl.java ---
    @@ -17,65 +17,174 @@
     package org.apache.carbondata.core.scan.filter.executer;
     
     import java.io.IOException;
    +import java.math.BigDecimal;
    +import java.nio.ByteBuffer;
     import java.util.BitSet;
    +import java.util.Comparator;
     
     import org.apache.carbondata.core.datastore.block.SegmentProperties;
     import org.apache.carbondata.core.datastore.chunk.DimensionColumnDataChunk;
    +import org.apache.carbondata.core.datastore.chunk.MeasureColumnDataChunk;
     import org.apache.carbondata.core.datastore.chunk.impl.DimensionRawColumnChunk;
    +import org.apache.carbondata.core.datastore.chunk.impl.MeasureRawColumnChunk;
    +import org.apache.carbondata.core.metadata.datatype.DataType;
     import org.apache.carbondata.core.scan.filter.FilterUtil;
    +import org.apache.carbondata.core.scan.filter.partition.PartitionFilterUtil;
     import org.apache.carbondata.core.scan.filter.resolver.resolverinfo.DimColumnResolvedFilterInfo;
    +import org.apache.carbondata.core.scan.filter.resolver.resolverinfo.MeasureColumnResolvedFilterInfo;
     import org.apache.carbondata.core.scan.processor.BlocksChunkHolder;
     import org.apache.carbondata.core.util.BitSetGroup;
     import org.apache.carbondata.core.util.ByteUtil;
     import org.apache.carbondata.core.util.CarbonUtil;
    +import org.apache.carbondata.core.util.DataTypeUtil;
     
     public class IncludeFilterExecuterImpl implements FilterExecuter {
     
       protected DimColumnResolvedFilterInfo dimColumnEvaluatorInfo;
       protected DimColumnExecuterFilterInfo dimColumnExecuterInfo;
    +  protected MeasureColumnResolvedFilterInfo msrColumnEvaluatorInfo;
    +  protected MeasureColumnExecuterFilterInfo msrColumnExecutorInfo;
       protected SegmentProperties segmentProperties;
    +  protected boolean isDimensionPresentInCurrentBlock = false;
    +  protected boolean isMeasurePresentInCurrentBlock = false;
    --- End diff --
   
    remove this flags and use null check of `dimColumnExecuterInfo` and `msrColumnExecutorInfo`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---