[GitHub] carbondata pull request #1402: [WIP] Change data type from enum to class

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1402#discussion_r144195221
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/metadata/datatype/MapType.java ---
    @@ -0,0 +1,36 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.carbondata.core.metadata.datatype;
    +
    +public class MapType extends DataType {
    +
    +  public static final DataType MAP = new MapType(DataTypes.MAP_TYPE_ID, 11, "MAP", -1);
    --- End diff --
   
    Even all complex types should not be a singleton.  these datatypes are dynamic as per schema


---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1402#discussion_r144195469
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/metadata/schema/table/CarbonTable.java ---
    @@ -18,7 +18,13 @@
     package org.apache.carbondata.core.metadata.schema.table;
     
     import java.io.Serializable;
    -import java.util.*;
    +import java.util.ArrayList;
    --- End diff --
   
    Use import java.util.*  for better readability and also guidelines suggests that more than 4 import from the same package
    should include *


---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1402#discussion_r144195868
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/scan/collector/impl/AbstractScannedResultCollector.java ---
    @@ -89,24 +90,23 @@ protected void fillMeasureData(Object[] msrValues, int offset,
       protected Object getMeasureData(ColumnPage dataChunk, int index,
           CarbonMeasure carbonMeasure) {
         if (!dataChunk.getNullBits().get(index)) {
    -      switch (carbonMeasure.getDataType()) {
    -        case SHORT:
    -          return (short)dataChunk.getLong(index);
    -        case INT:
    -          return (int)dataChunk.getLong(index);
    -        case LONG:
    -          return dataChunk.getLong(index);
    -        case DECIMAL:
    -          BigDecimal bigDecimalMsrValue =
    -              dataChunk.getDecimal(index);
    -          if (null != bigDecimalMsrValue && carbonMeasure.getScale() > bigDecimalMsrValue.scale()) {
    -            bigDecimalMsrValue =
    -                bigDecimalMsrValue.setScale(carbonMeasure.getScale(), RoundingMode.HALF_UP);
    -          }
    -          // convert data type as per the computing engine
    -          return DataTypeUtil.getDataTypeConverter().convertToDecimal(bigDecimalMsrValue);
    -        default:
    -          return dataChunk.getDouble(index);
    +      DataType dataType = carbonMeasure.getDataType();
    +      if (dataType == DataTypes.SHORT) {
    --- End diff --
   
    why don't use switch case  with id


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1402#discussion_r144204315
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/metadata/datatype/DateType.java ---
    @@ -0,0 +1,31 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.carbondata.core.metadata.datatype;
    +
    +public class DateType extends DataType {
    +
    +  public static final DataType DATE = new DateType(DataTypes.DATE_TYPE_ID, 1, "DATE", -1);
    +
    +  private DateType(int id, int precedenceOrder, String name, int sizeInBytes) {
    +    super(id, precedenceOrder, name, sizeInBytes);
    +  }
    +
    +  private Object readResolve() {
    --- End diff --
   
    It is added to ensure singleton while supporting serialization/deserialization by java. I will add comment to describe it.


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1402#discussion_r144204348
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/metadata/datatype/DecimalType.java ---
    @@ -0,0 +1,32 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.carbondata.core.metadata.datatype;
    +
    +public class DecimalType extends DataType {
    +
    +  public static final DataType DECIMAL =
    +      new DecimalType(DataTypes.DECIMAL_TYPE_ID, 8, "DECIMAL", -1);
    +
    +  private DecimalType(int id, int precedenceOrder, String name, int sizeInBytes) {
    +    super(id, precedenceOrder, name, sizeInBytes);
    +  }
    +
    +  private Object readResolve() {
    --- End diff --
   
    It is added to ensure singleton while supporting serialization/deserialization by java. I will add comment to describe it.


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1402#discussion_r144204706
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/metadata/datatype/DecimalType.java ---
    @@ -0,0 +1,32 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.carbondata.core.metadata.datatype;
    +
    +public class DecimalType extends DataType {
    +
    +  public static final DataType DECIMAL =
    --- End diff --
   
    Yes, I will do it in next PR. It needs more modification.


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1402#discussion_r144204715
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/metadata/datatype/MapType.java ---
    @@ -0,0 +1,36 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.carbondata.core.metadata.datatype;
    +
    +public class MapType extends DataType {
    +
    +  public static final DataType MAP = new MapType(DataTypes.MAP_TYPE_ID, 11, "MAP", -1);
    --- End diff --
   
    Yes, I will do it in next PR. It needs more modification.


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1402#discussion_r144204730
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/metadata/datatype/MapType.java ---
    @@ -0,0 +1,36 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.carbondata.core.metadata.datatype;
    +
    +public class MapType extends DataType {
    +
    +  public static final DataType MAP = new MapType(DataTypes.MAP_TYPE_ID, 11, "MAP", -1);
    --- End diff --
   
    Yes, I will do it in next PR. It needs more modification.


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1402#discussion_r144205561
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/scan/collector/impl/AbstractScannedResultCollector.java ---
    @@ -89,24 +90,23 @@ protected void fillMeasureData(Object[] msrValues, int offset,
       protected Object getMeasureData(ColumnPage dataChunk, int index,
           CarbonMeasure carbonMeasure) {
         if (!dataChunk.getNullBits().get(index)) {
    -      switch (carbonMeasure.getDataType()) {
    -        case SHORT:
    -          return (short)dataChunk.getLong(index);
    -        case INT:
    -          return (int)dataChunk.getLong(index);
    -        case LONG:
    -          return dataChunk.getLong(index);
    -        case DECIMAL:
    -          BigDecimal bigDecimalMsrValue =
    -              dataChunk.getDecimal(index);
    -          if (null != bigDecimalMsrValue && carbonMeasure.getScale() > bigDecimalMsrValue.scale()) {
    -            bigDecimalMsrValue =
    -                bigDecimalMsrValue.setScale(carbonMeasure.getScale(), RoundingMode.HALF_UP);
    -          }
    -          // convert data type as per the computing engine
    -          return DataTypeUtil.getDataTypeConverter().convertToDecimal(bigDecimalMsrValue);
    -        default:
    -          return dataChunk.getDouble(index);
    +      DataType dataType = carbonMeasure.getDataType();
    +      if (dataType == DataTypes.SHORT) {
    --- End diff --
   
    For primitive type and String type, we can compare object address, it is simpler.
    Only one place need to compare using id, `fromWrapperToExternalDataType` method. It needs it because GSON breaks the singleton pattern


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1402#discussion_r144205594
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/metadata/schema/table/CarbonTable.java ---
    @@ -18,7 +18,13 @@
     package org.apache.carbondata.core.metadata.schema.table;
     
     import java.io.Serializable;
    -import java.util.*;
    +import java.util.ArrayList;
    --- End diff --
   
    ok


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1402
 
    Build Success with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/310/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1402
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/435/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1402
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1063/



---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1402#discussion_r144220468
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/scan/collector/impl/AbstractScannedResultCollector.java ---
    @@ -89,24 +90,23 @@ protected void fillMeasureData(Object[] msrValues, int offset,
       protected Object getMeasureData(ColumnPage dataChunk, int index,
           CarbonMeasure carbonMeasure) {
         if (!dataChunk.getNullBits().get(index)) {
    -      switch (carbonMeasure.getDataType()) {
    -        case SHORT:
    -          return (short)dataChunk.getLong(index);
    -        case INT:
    -          return (int)dataChunk.getLong(index);
    -        case LONG:
    -          return dataChunk.getLong(index);
    -        case DECIMAL:
    -          BigDecimal bigDecimalMsrValue =
    -              dataChunk.getDecimal(index);
    -          if (null != bigDecimalMsrValue && carbonMeasure.getScale() > bigDecimalMsrValue.scale()) {
    -            bigDecimalMsrValue =
    -                bigDecimalMsrValue.setScale(carbonMeasure.getScale(), RoundingMode.HALF_UP);
    -          }
    -          // convert data type as per the computing engine
    -          return DataTypeUtil.getDataTypeConverter().convertToDecimal(bigDecimalMsrValue);
    -        default:
    -          return dataChunk.getDouble(index);
    +      DataType dataType = carbonMeasure.getDataType();
    +      if (dataType == DataTypes.SHORT) {
    --- End diff --
   
    But I feel changes are small and looks good to use switch case instead of using if else.


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1402
 
    LGTM


---

[qiuchenjian-2]

Github user asfgit closed the pull request at:

    https://github.com/apache/carbondata/pull/1402


---