[DISCUSSION] Support Map type for complex type columns

[manishgupta88]

Hi All,

I am working on supporting complex type map columns. Please find below the
scope for the same.

*Scope:*
1. Create Table DDL support for complex map type.
1. Support loading of data for complex map type column [SDK, DataLoad DDL].
2. Design to consider n level nested support for map values and keys as
primitive types similar to Hive.
3. Support querying complex type map columns.
5. Support all datatypes in map complex type values includng struct and
array.
6. Describe formatted support to display the complex map type column.
7. Support configurable delimeters for key and value separator and a
separator for each map entry.

Please feel free to provide your comments and suggestions. I am working on
the design document and will upload soon.
*Jira Link:* https://issues.apache.org/jira/browse/CARBONDATA-45

Regards
Manish Gupta



--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/

[Indhumathi]

Hi all,

Please find the Attached MAP-DataType support Design document. Please let me
know for any further clarification on design document.
Any further suggestions/improvements are welcomed.

MAP_DATA-TYPE_SUPPORT.pdf
<http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/file/t341/MAP_DATA-TYPE_SUPPORT.pdf>  

Please find the link for design document.
https://docs.google.com/document/d/1HHe2fdkIh3Jyz1y3494_2kGRSc4muTWuilAmwg5lpVw/edit?usp=sharing
<https://docs.google.com/document/d/1HHe2fdkIh3Jyz1y3494_2kGRSc4muTWuilAmwg5lpVw/edit?usp=sharing>  

Regards
Indhumathi M



--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/

[chetdb]

1. Will lucene datamap and MV datamap be able to be created with map type. ?
2. Can COLUMN_META_CACHE ,NO_INVERTED_INDEX and bucketing table properties
be configured for map type column. ?
3. Currently carbon supports 2 level complex type schema for csv and n level
for avro. Will the same be supported for map type as well ?
4. Dictionary include/exclude for complex type child columns is not
supported now. Will the same apply for map type as well ?
5. In the design doc its mentioned that Local Dictionary for Map Datatype
column has to be supported. However in case of multi-level complex dataType
columns, primitive string/varchar/char columns are
considered for local dictionary generation. Please confirm the behavior to
be supported.







--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/

[sujith chacko]

Hi Indumathi,
Design document covers all aspects of the implementation which is good
except few minor points are missing i think like,
It will be better if we can list down the constraints in our
implementations if any, like
a)Can we use map data type columns as partition key columns?
b) It will be clear if you can mention all the data types which is
supported, like do we support all primitive types especially timestanp
since its one of the important type of key for map data type
e.g. you can store the events corresponding to a time periods.
c) Any restrictions in the level of nesting?
eg 50 or 100

Thanks,
Sujith

On Mon, 20 Aug 2018 at 4:24 PM, Indhumathi <[hidden email]> wrote:

> Hi all,
>
> Please find the Attached MAP-DataType support Design document. Please let
> me
> know for any further clarification on design document.
> Any further suggestions/improvements are welcomed.
>
> MAP_DATA-TYPE_SUPPORT.pdf
> <
> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/file/t341/MAP_DATA-TYPE_SUPPORT.pdf>
>
>
> Please find the link for design document.
>
> https://docs.google.com/document/d/1HHe2fdkIh3Jyz1y3494_2kGRSc4muTWuilAmwg5lpVw/edit?usp=sharing
> <
> https://docs.google.com/document/d/1HHe2fdkIh3Jyz1y3494_2kGRSc4muTWuilAmwg5lpVw/edit?usp=sharing>
>
>
> Regards
> Indhumathi M
>
>
>
> --
> Sent from:
> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
>

[Indhumathi]

Hi Sujith,

Thanks for your review. Please find my comments inline.

a)Can we use map data type columns as partition key columns?

** No. It has been already mentioned in Impacted Feature Analysis. **

b) It will be clear if you can mention all the data types which is
supported, like do we support all primitive types especially timestanp
since its one of the important type of key for map data type
e.g. you can store the events corresponding to a time periods.

** Supported DataTypes for Map has been updated in Design Document. Please
check **

c) Any restrictions in the level of nesting?
eg 50 or 100

** Similar to Array and Struct, Map will also have n - level nesting
supported through SDK.
    Without SDK, 2-level nesting will be supported **

Please find the updated Design document for Map-DataType.
MAP_DATA-TYPE_SUPPORT.pdf
<http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/file/t341/MAP_DATA-TYPE_SUPPORT.pdf>  



--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/

[Indhumathi]

Hi Chetan,

Thanks for your review. Please find my comments inline.

1. Will lucene datamap and MV datamap be able to be created with map type. ?

** No. This has been updated in Design Document. Please check **

2. Can COLUMN_META_CACHE ,NO_INVERTED_INDEX and bucketing table properties
be configured for map type column. ?

** No. Same behavior as that of Struct and Array **

3. Currently carbon supports 2 level complex type schema for csv and n level
for avro. Will the same be supported for map type as well ?

** Yes. Similar to Array and Struct, Map will also support 2 level complex
type schema
    for csv and n level for avro **

4. Dictionary include/exclude for complex type child columns is not
supported now. Will the same apply for map type as well ?

** Yes. Same behavior as that of Array and Struct. **

5. In the design doc its mentioned that Local Dictionary for Map Datatype
column has to be supported. However in case of multi-level complex dataType
columns, primitive string/varchar/char columns are
considered for local dictionary generation. Please confirm the behavior to
be supported.

** Internally, Map will be stored as Array<Struct&lt;key, values>>, hence
behavior will
   be same as that of Struct and Array. **

Please find the updated Design document for Map -Datatype support
MAP_DATA-TYPE_SUPPORT.pdf
<http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/file/t341/MAP_DATA-TYPE_SUPPORT.pdf>  



--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/