[GitHub] [carbondata] chetandb commented on a change in pull request #3275: [CARBONDATA-3425]Added documentation for mv

[GitBox]

chetandb commented on a change in pull request #3275: [CARBONDATA-3425]Added documentation for mv
URL: https://github.com/apache/carbondata/pull/3275#discussion_r304397130
 
 

 ##########
 File path: docs/datamap/mv-datamap-guide.md
 ##########
 @@ -0,0 +1,207 @@
+<!--
+    Licensed to the Apache Software Foundation (ASF) under one or more
+    contributor license agreements.  See the NOTICE file distributed with
+    this work for additional information regarding copyright ownership.
+    The ASF licenses this file to you under the Apache License, Version 2.0
+    (the "License"); you may not use this file except in compliance with
+    the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+    Unless required by applicable law or agreed to in writing, software
+    distributed under the License is distributed on an "AS IS" BASIS,
+    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+    See the License for the specific language governing permissions and
+    limitations under the License.
+-->
+
+# CarbonData MV DataMap
+
+* [Quick Example](#quick-example)
+* [MV DataMap](#mv-datamap-introduction)
+* [Loading Data](#loading-data)
+* [Querying Data](#querying-data)
+* [Compaction](#compacting-mv-tables)
+* [Data Management](#data-management-with-mv-tables)
+
+## Quick example
+
+Start spark-sql in terminal and run the following queries,
+```
+CREATE TABLE maintable(a int, b string, c int) stored by 'carbondata';
+insert into maintable select 1, 'ab', 2;
+CREATE DATAMAP datamap_1 on table maintable as SELECT a, sum(b) from maintable group by a;
+SELECT a, sum(b) from maintable group by a;
+EXPLAIN SELECT a, sum(b) from maintable group by a;
+```
+**NOTE**:
+  Run explain query and check if query hits the datamap table from the plan. Please refer to
+  EXPLAIN in [DataMap Related Commands](./datamap-management.md#datamap-related-commands) to
+  know whether datamap is used in the query
+
+## MV DataMap Introduction
+  MV tables are created as DataMaps and managed as tables internally by CarbonData. User can create
+  limitless MV datamaps on a table to improve query performance provided the storage requirements
+  and loading time is acceptable.
+
+  MV datamap can be a lazy or a non-lazy datamap. Once MV datamaps are created, CarbonData's
+  CarbonAnalyzer helps to select the most efficient MV datamap based on the user query and rewrite
+  the SQL to select the data from MV datamap instead of main table. Since the data size of MV
+  datamap is smaller and data is pre-processed, user queries are much faster.
+
+  For instance, main table called **sales** which is defined as
+
+  ```
+  CREATE TABLE sales (
+    order_time timestamp,
+    user_id string,
+    sex string,
+    country string,
+    quantity int,
+    price bigint)
+  STORED AS carbondata
+  ```
+
+  User can create MV tables using the Create DataMap DDL
+
+  ```
+  CREATE DATAMAP agg_sales
+  ON TABLE sales
+  USING "MV"
+  AS
+    SELECT country, sex, sum(quantity), avg(price)
+    FROM sales
+    GROUP BY country, sex
+  ```
+**NOTE**:
+ * Group by/Filter columns has to be provided in projection list while creating mv datamap
+ * If only single parent table is involved in mv datamap creation, then table properties of parent
+   (if not present in a aggregate function like sum(col)) listed below will be
+   inherited to datamap table
+  1. SORT_COLUMNS,
+  2. SORT_SCOPE,
+  3. TABLE_BLOCKSIZE,
+  4. FLAT_FOLDER,
+  5. LONG_STRING_COLUMNS,
+  6. LOCAL_DICTIONARY_ENABLE,
+  7. LOCAL_DICTIONARY_THRESHOLD,
+  8. LOCAL_DICTIONARY_EXCLUDE,
+  9. DICTIONARY_INCLUDE,
+  10. DICTIONARY_EXCLUDE,
+  11. INVERTED_INDEX,
+  12. NO_INVERTED_INDEX,
+  13. COLUMN_COMPRESSOR
+ * All columns of main table at once cannot participate in mv datamap table creation
+ * TableProperties can be provided in DMProperties excluding LOCAL_DICTIONARY_INCLUDE,
+   LOCAL_DICTIONARY_EXCLUDE, DICTIONARY_INCLUDE, DICTIONARY_EXCLUDE, INVERTED_INDEX,
 
 Review comment:
   Streaming property is not supported in dmproperties. This can be mentioned in this list.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services