[GitHub] carbondata pull request #2649: [WIP] Add support for Avro Map data type supp...

classic Classic list List threaded Threaded
29 messages Options
12
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata pull request #2649: [CARBONDATA-2869] Add support for Avro Map da...

qiuchenjian-2
Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2649#discussion_r213557758
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/metadata/schema/table/TableSchemaBuilder.java ---
    @@ -195,7 +196,8 @@ private ColumnSchema addColumn(StructField field, String parentName, AtomicInteg
         newColumn.setColumnReferenceId(newColumn.getColumnUniqueId());
         newColumn.setEncodingList(createEncoding(field.getDataType(), isSortColumn, isComplexChild));
         if (field.getDataType().isComplexType()) {
    -      if (field.getDataType().getName().equalsIgnoreCase("ARRAY")) {
    +      if (field.getDataType().getName().equalsIgnoreCase("ARRAY") || field.getDataType().getName()
    --- End diff --
   
    Please use `DataTypes.isArrayType(field.getDataType())`


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata pull request #2649: [CARBONDATA-2869] Add support for Avro Map da...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2649#discussion_r213560099
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/scan/complextypes/MapQueryType.java ---
    @@ -0,0 +1,47 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.carbondata.core.scan.complextypes;
    +
    +import java.nio.ByteBuffer;
    +
    +import org.apache.carbondata.core.util.DataTypeUtil;
    +
    +/**
    + * handles querying map data type columns
    + */
    +public class MapQueryType extends ArrayQueryType {
    +
    +  public MapQueryType(String name, String parentname, int blockIndex) {
    +    super(name, parentname, blockIndex);
    +  }
    +
    +  @Override public Object getDataBasedOnDataType(ByteBuffer dataBuffer) {
    --- End diff --
   
    Please add comment to tell how we convert array to map


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata pull request #2649: [CARBONDATA-2869] Add support for Avro Map da...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2649#discussion_r213566979
 
    --- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/file/AvroCarbonWriter.java ---
    @@ -134,6 +139,38 @@ private Object avroFieldToObject(Schema.Field avroField, Object fieldValue) {
             // also carbon internally needs float as double
             out = Double.parseDouble(fieldValue.toString());
             break;
    +      case MAP:
    +        // Note: Avro object takes care of removing the duplicates so we should not handle it again
    +        // Map will be internally stored as Array<Struct<Key,Value>>
    +        Map mapEntries = (HashMap) fieldValue;
    +        Object[] arrayMapChildObjects = new Object[mapEntries.size()];
    +        if (!mapEntries.isEmpty()) {
    +          Iterator iterator = ((HashMap) fieldValue).entrySet().iterator();
    --- End diff --
   
    Use `mapEntries` directly instead of typecast again


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2649: [CARBONDATA-2869] Add support for Avro Map data type...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user manishgupta88 commented on the issue:

    https://github.com/apache/carbondata/pull/2649
 
    @ravipesala ...fixed review comment. Kindly review and merge


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2649: [CARBONDATA-2869] Add support for Avro Map data type...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2649
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6451/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2649: [CARBONDATA-2869] Add support for Avro Map data type...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2649
 
    Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/66/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2649: [CARBONDATA-2869] Add support for Avro Map data type...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2649
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/8137/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2649: [CARBONDATA-2869] Add support for Avro Map data type...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2649
 
    LGTM


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata pull request #2649: [CARBONDATA-2869] Add support for Avro Map da...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user asfgit closed the pull request at:

    https://github.com/apache/carbondata/pull/2649


---
12