[GitHub] carbondata pull request #2417: [WIP][Complex Column Enhancements]Primitive D...

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2417
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6704/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2417
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5531/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2417
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6710/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2417
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6711/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2417
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6712/



---

[qiuchenjian-2]

Github user gvramana commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2417#discussion_r199481192
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/datastore/chunk/store/ColumnPageWrapper.java ---
    @@ -30,32 +40,72 @@ public ColumnPageWrapper(ColumnPage columnPage) {
         this.columnPage = columnPage;
       }
     
    +  public ColumnPage getColumnPage() {
    +    return columnPage;
    +  }
    +
       @Override
       public int fillRawData(int rowId, int offset, byte[] data, KeyStructureInfo restructuringInfo) {
    -    throw new UnsupportedOperationException("internal error");
    +    throw new UnsupportedOperationException(
    +        "internal error: should be called for only dictionary columns");
       }
     
       @Override
       public int fillSurrogateKey(int rowId, int chunkIndex, int[] outputSurrogateKey,
           KeyStructureInfo restructuringInfo) {
    -    throw new UnsupportedOperationException("internal error");
    +    throw new UnsupportedOperationException(
    +        "internal error: should be called for only dictionary columns");
       }
     
       @Override
       public int fillVector(ColumnVectorInfo[] vectorInfo, int chunkIndex,
           KeyStructureInfo restructuringInfo) {
    -    throw new UnsupportedOperationException("internal error");
    +    // fill the vector with data in column page
    +    ColumnVectorInfo columnVectorInfo = vectorInfo[chunkIndex];
    +    CarbonColumnVector vector = columnVectorInfo.vector;
    +    fillData(null, columnVectorInfo, vector);
    +    return chunkIndex + 1;
       }
     
    +
       @Override
       public int fillVector(int[] filteredRowId, ColumnVectorInfo[] vectorInfo, int chunkIndex,
           KeyStructureInfo restructuringInfo) {
    -    throw new UnsupportedOperationException("internal error");
    +    ColumnVectorInfo columnVectorInfo = vectorInfo[chunkIndex];
    +    CarbonColumnVector vector = columnVectorInfo.vector;
    +    fillData(filteredRowId, columnVectorInfo, vector);
    +    return chunkIndex + 1;
       }
     
    -  @Override
    -  public byte[] getChunkData(int rowId) {
    -    return columnPage.getBytes(rowId);
    +  @Override public byte[] getChunkData(int rowId) {
    +    ColumnType columnType = columnPage.getColumnSpec().getColumnType();
    +    // TODO: No need to convert to Byte array, handle like measure
    +    // But interface currently doesn't support, need to add new interface.
    +    if (columnType == ColumnType.PLAIN_VALUE) {
    +      if (columnPage.getNullBits().get(rowId)) {
    +        // if this row is null, return default null represent in byte array
    +        return CarbonCommonConstants.MEMBER_DEFAULT_VAL_ARRAY;
    +      }
    +      if (columnPage.getDataType() == DataTypes.BYTE) {
    +        byte byteData = columnPage.getByte(rowId);
    +        return ByteUtil.toBytes(byteData);
    +      } else if (columnPage.getDataType() == DataTypes.SHORT) {
    --- End diff --
   
    Use Switch instead of ifelse


---

[qiuchenjian-2]

Github user gvramana commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2417#discussion_r199481217
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/datastore/chunk/store/ColumnPageWrapper.java ---
    @@ -77,14 +127,70 @@ public boolean isExplicitSorted() {
         return false;
       }
     
    -  @Override
    -  public int compareTo(int rowId, byte[] compareValue) {
    -    throw new UnsupportedOperationException("internal error");
    +  @Override public int compareTo(int rowId, byte[] compareValue) {
    +    throw new UnsupportedOperationException(
    +        "internal error: should be called for only dictionary columns");
       }
     
       @Override
       public void freeMemory() {
     
       }
     
    +  private void fillData(int[] rowMapping, ColumnVectorInfo columnVectorInfo,
    +      CarbonColumnVector vector) {
    +    int offsetRowId = columnVectorInfo.offset;
    +    int vectorOffset = columnVectorInfo.vectorOffset;
    +    int maxRowId = offsetRowId + columnVectorInfo.size;
    +    BitSet nullBitSet = columnPage.getNullBits();
    +    TableSpec.ColumnSpec columnSpec = columnPage.getColumnSpec();
    +    if (columnSpec.getColumnType() == PLAIN_VALUE) {
    +      for (int rowId = offsetRowId; rowId < maxRowId; rowId++) {
    +        int currentRowId = (rowMapping == null) ? rowId : rowMapping[rowId];
    +        if (nullBitSet.get(currentRowId)) {
    +          // to handle the null values
    +          vector.putNull(vectorOffset++);
    +        } else {
    +          if (columnSpec.getSchemaDataType() == DataTypes.STRING) {
    +            byte[] data = columnPage.getBytes(currentRowId);
    +            if (ByteUtil.UnsafeComparer.INSTANCE
    +                .equals(CarbonCommonConstants.MEMBER_DEFAULT_VAL_ARRAY, data)) {
    +              vector.putNull(vectorOffset++);
    +            } else {
    +              vector.putBytes(vectorOffset++, 0, data.length, data);
    +            }
    +          } else if (columnSpec.getSchemaDataType() == DataTypes.BOOLEAN) {
    +            boolean data = columnPage.getBoolean(currentRowId);
    +            vector.putBoolean(vectorOffset++, data);
    +          } else if (columnSpec.getSchemaDataType() == DataTypes.SHORT) {
    +            short data = columnPage.getShort(currentRowId);
    +            vector.putShort(vectorOffset++, data);
    --- End diff --
   
    Use Switch instead of ifelse


---

[qiuchenjian-2]

Github user gvramana commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2417#discussion_r199482930
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/encoding/DefaultEncodingFactory.java ---
    @@ -161,14 +174,16 @@ private static DataType fitLongMinMax(long max, long min) {
       }
     
       private static DataType fitMinMax(DataType dataType, Object max, Object min) {
    -    if (dataType == DataTypes.BYTE) {
    +    if ((dataType == DataTypes.BYTE) || (dataType == DataTypes.BOOLEAN)) {
    --- End diff --
   
    Use Switch instead of ifelse


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2417
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5557/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2417
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5537/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2417
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5540/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2417
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5560/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2417
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5561/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2417
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5562/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2417
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6741/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2417
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5586/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2417
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5573/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2417
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5590/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2417
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5591/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2417
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6746/



---