[GitHub] carbondata pull request #2417: [WIP][Complex Column Enhancements]Primitive D...

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2417
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6634/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2417
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5462/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2417
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5508/



---

[qiuchenjian-2]

Github user sounakr commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2417#discussion_r199081296
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/encoding/DefaultEncodingFactory.java ---
    @@ -38,6 +38,15 @@
     import org.apache.carbondata.core.metadata.datatype.DataTypes;
     import org.apache.carbondata.core.metadata.datatype.DecimalConverterFactory;
     
    +import static org.apache.carbondata.core.metadata.datatype.DataTypes.BOOLEAN;
    --- End diff --
   
    Done


---

[qiuchenjian-2]

Github user sounakr commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2417#discussion_r199081333
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/encoding/DefaultEncodingFactory.java ---
    @@ -58,16 +67,36 @@ public static EncodingFactory getInstance() {
       @Override
       public ColumnPageEncoder createEncoder(TableSpec.ColumnSpec columnSpec, ColumnPage inputPage) {
         // TODO: add log
    +    ColumnPageEncoder pageEncoder = null;
         if (columnSpec instanceof TableSpec.MeasureSpec) {
           return createEncoderForMeasure(inputPage);
    -    } else {
    -      if (newWay) {
    -        return createEncoderForDimension((TableSpec.DimensionSpec) columnSpec, inputPage);
    -      } else {
    -        assert columnSpec instanceof TableSpec.DimensionSpec;
    +    } else if (columnSpec instanceof TableSpec.DimensionSpec) {
    +      pageEncoder = createCodecForDimension((TableSpec.DimensionSpec) columnSpec, inputPage);
    +      if (pageEncoder == null) {
             return createEncoderForDimensionLegacy((TableSpec.DimensionSpec) columnSpec);
           }
         }
    +    return pageEncoder;
    +  }
    +
    +  private ColumnPageEncoder createCodecForDimension(TableSpec.DimensionSpec columnSpec,
    +      ColumnPage inputPage) {
    +    switch (columnSpec.getColumnType()) {
    +      case PLAIN_VALUE:
    +        if ((inputPage.getDataType() == DataTypes.BYTE) || (inputPage.getDataType()
    +            == DataTypes.SHORT) || (inputPage.getDataType() == DataTypes.INT) || (
    +            inputPage.getDataType() == DataTypes.LONG)) {
    +          return selectCodecByAlgorithmForIntegral(inputPage.getStatistics()).createEncoder(null);
    +        } else if ((inputPage.getDataType() == DataTypes.FLOAT) || (inputPage.getDataType()
    +            == DataTypes.DOUBLE)) {
    +          return selectCodecByAlgorithmForFloating(inputPage.getStatistics()).createEncoder(null);
    +        } else if (inputPage.getDataType() == DataTypes.STRING) {
    +          // TODO. Currently let string go through legacy encoding. Later will change the encoding.
    +          return null;
    +        }
    +        break;
    +    }
    +    return null;
       }
     
       private ColumnPageEncoder createEncoderForDimension(TableSpec.DimensionSpec columnSpec,
    --- End diff --
   
    Done


---

[qiuchenjian-2]

Github user sounakr commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2417#discussion_r199081355
 
    --- Diff: processing/src/main/java/org/apache/carbondata/processing/store/TablePage.java ---
    @@ -354,6 +341,18 @@ public EncodedTablePage getEncodedTablePage() {
                   .getColumnType());
           }
         }
    +    //    for (int i = 0; i < dimensionPages.length; i++) {
    --- End diff --
   
    Done


---

[qiuchenjian-2]

Github user sounakr commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2417#discussion_r199081394
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/util/DataTypeUtil.java ---
    @@ -436,10 +436,11 @@ public static boolean isFixedSizeDataType(DataType dataType) {
        *
        * @param dataInBytes    data
        * @param actualDataType actual data type
    +   * @param isTimeStampConversion
        * @return actual data after conversion
        */
       public static Object getDataBasedOnDataTypeForNoDictionaryColumn(byte[] dataInBytes,
    -      DataType actualDataType) {
    +      DataType actualDataType, boolean isTimeStampConversion) {
    --- End diff --
   
    Done


---

[qiuchenjian-2]

Github user sounakr commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2417#discussion_r199081933
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/ColumnPage.java ---
    @@ -405,13 +446,30 @@ public void putData(int rowId, Object value) {
         } else if (dataType == DataTypes.STRING
             || dataType == DataTypes.BYTE_ARRAY
             || dataType == DataTypes.VARCHAR) {
    -      putBytes(rowId, (byte[]) value);
    -      statsCollector.update((byte[]) value);
    +      byte[] valueWithLength;
    +      if (columnSpec.getColumnType() != ColumnType.PLAIN_VALUE) {
    +        // This case is for GLOBAL_DICTIONARY and DIRECT_DICTIONARY. In this
    +        // scenario the dataType is BYTE_ARRAY and passed bytearray should
    +        // be saved.
    +        putBytes(rowId, (byte[]) value);
    +        statsCollector.update((byte[]) value);
    +      } else {
    +        if (dataType == DataTypes.VARCHAR) {
    +          // Add length and then add the data.
    +          valueWithLength = addIntLengthToByteArray((byte[]) value);
    +        } else {
    +          valueWithLength = addShortLengthToByteArray((byte[]) value);
    +        }
    +        putBytes(rowId, valueWithLength);
    +        statsCollector.update((byte[]) valueWithLength);
    +      }
         } else {
           throw new RuntimeException("unsupported data type: " + dataType);
         }
       }
     
    +
    --- End diff --
   
    Done


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2417
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6641/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2417
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5469/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2417
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6644/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2417
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5513/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2417
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5473/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2417
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5515/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2417
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5520/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2417
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6693/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2417
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5547/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2417
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5522/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2417
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6695/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2417
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5550/



---