[GitHub] [carbondata] akashrn5 opened a new pull request #3689: [WIP]add Secondary Index Document
Locked
1234
1234
 
|
akashrn5 opened a new pull request #3689: [WIP]add Secondary Index Document
URL: https://github.com/apache/carbondata/pull/3689 ### Why is this PR needed? ### What changes were proposed in this PR? ### Does this PR introduce any user interface change? - No - Yes. (please explain the change and update document) ### Is any new testcase added? - No - Yes ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [hidden email] With regards, Apache Git Services |
|
|
CarbonDataQA1 commented on issue #3689: [WIP]add Secondary Index Document
URL: https://github.com/apache/carbondata/pull/3689#issuecomment-606868582 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2598/ ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [hidden email] With regards, Apache Git Services |
|
|
In reply to this post by GitBox
CarbonDataQA1 commented on issue #3689: [WIP]add Secondary Index Document
URL: https://github.com/apache/carbondata/pull/3689#issuecomment-606872959 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/890/ ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [hidden email] With regards, Apache Git Services |
|
|
In reply to this post by GitBox
Indhumathi27 commented on a change in pull request #3689: [CARBONDATA-3680]Add Secondary Index Document
URL: https://github.com/apache/carbondata/pull/3689#discussion_r401371920 ########## File path: docs/index/secondary-index-guide.md ########## @@ -0,0 +1,155 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to you under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +# CarbonData Secondary Index + +* [Quick Example](#quick-example) +* [Secondary Index Table](#Secondary-Index-Introduction) +* [Loading Data](#loading-data) +* [Querying Data](#querying-data) +* [Compaction](#compacting-SI-table) +* [Data Management](#data-management-with-SI-tables) + +## Quick example + +Start spark-sql in terminal and run the following queries, +``` +CREATE TABLE maintable(a int, b string, c string) stored as carbondata; +insert into maintable select 1, 'ab', 'cd'; +CREATE index inex1 on table maintable(c) AS 'carbondata'; +SELECT a from maintable where c = 'cd'; +// NOTE: run explain query and check if query hits the SI table from the plan +EXPLAIN SELECT a from maintable where c = 'cd'; +``` + +## Secondary Index Introduction + Sencondary index tables are created as a child table and manager as tables internally by Carbondata. + User can create the SI tables based on the column position in table(Recommended for right columns) Review comment: ```suggestion Users can create secondary index based on the column position in main table(Recommended for right columns) ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [hidden email] With regards, Apache Git Services |
|
|
In reply to this post by GitBox
Indhumathi27 commented on a change in pull request #3689: [CARBONDATA-3680]Add Secondary Index Document
URL: https://github.com/apache/carbondata/pull/3689#discussion_r401373694 ########## File path: docs/index/secondary-index-guide.md ########## @@ -0,0 +1,155 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to you under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +# CarbonData Secondary Index + +* [Quick Example](#quick-example) +* [Secondary Index Table](#Secondary-Index-Introduction) +* [Loading Data](#loading-data) +* [Querying Data](#querying-data) +* [Compaction](#compacting-SI-table) +* [Data Management](#data-management-with-SI-tables) + +## Quick example + +Start spark-sql in terminal and run the following queries, +``` +CREATE TABLE maintable(a int, b string, c string) stored as carbondata; +insert into maintable select 1, 'ab', 'cd'; +CREATE index inex1 on table maintable(c) AS 'carbondata'; +SELECT a from maintable where c = 'cd'; +// NOTE: run explain query and check if query hits the SI table from the plan +EXPLAIN SELECT a from maintable where c = 'cd'; +``` + +## Secondary Index Introduction + Sencondary index tables are created as a child table and manager as tables internally by Carbondata. + User can create the SI tables based on the column position in table(Recommended for right columns) + and the queries should have filter on that column to improve the filter query performance. + + SI tables will always be loaded non-lazy way. Once SI table is created, Carbondata's + CarbonOptimizer with the help of CarbonSITransformationRule, transforms the query plan to hit the + SI table based on the filter condition or set of filter conditions present in the query. + So first level of pruning will be done on SI table as it stores blocklets and main table/parent + table pruning will be based on the SI output, which helps in giving the faster query results with + better pruning. + + For instance, main table called **sales** which is defined as + + ``` + CREATE TABLE sales ( + order_time timestamp, + user_id string, + sex string, + country string, + quantity int, + price bigint) + STORED AS carbondata + ``` + + User can create SI table using the Create Index DDL + + ``` + CREATE INDEX index_sales + ON TABLE sales(user_id) + AS + 'carbondata' + TBLPROPERTIES('table_blocksize'='1') + ``` +**NOTE**: + * Secondary Index tables are registered with hive and stored n HiveSERDEPROPERTIES s json formatted Review comment: Please check and correct the spelling ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [hidden email] With regards, Apache Git Services |
|
|
In reply to this post by GitBox
Indhumathi27 commented on a change in pull request #3689: [CARBONDATA-3680]Add Secondary Index Document
URL: https://github.com/apache/carbondata/pull/3689#discussion_r401374771 ########## File path: docs/index/secondary-index-guide.md ########## @@ -0,0 +1,155 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to you under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +# CarbonData Secondary Index + +* [Quick Example](#quick-example) +* [Secondary Index Table](#Secondary-Index-Introduction) +* [Loading Data](#loading-data) +* [Querying Data](#querying-data) +* [Compaction](#compacting-SI-table) +* [Data Management](#data-management-with-SI-tables) + +## Quick example + +Start spark-sql in terminal and run the following queries, +``` +CREATE TABLE maintable(a int, b string, c string) stored as carbondata; +insert into maintable select 1, 'ab', 'cd'; +CREATE index inex1 on table maintable(c) AS 'carbondata'; +SELECT a from maintable where c = 'cd'; +// NOTE: run explain query and check if query hits the SI table from the plan +EXPLAIN SELECT a from maintable where c = 'cd'; +``` + +## Secondary Index Introduction + Sencondary index tables are created as a child table and manager as tables internally by Carbondata. + User can create the SI tables based on the column position in table(Recommended for right columns) + and the queries should have filter on that column to improve the filter query performance. + + SI tables will always be loaded non-lazy way. Once SI table is created, Carbondata's + CarbonOptimizer with the help of CarbonSITransformationRule, transforms the query plan to hit the + SI table based on the filter condition or set of filter conditions present in the query. + So first level of pruning will be done on SI table as it stores blocklets and main table/parent + table pruning will be based on the SI output, which helps in giving the faster query results with + better pruning. + + For instance, main table called **sales** which is defined as + + ``` + CREATE TABLE sales ( + order_time timestamp, + user_id string, + sex string, + country string, + quantity int, + price bigint) + STORED AS carbondata + ``` + + User can create SI table using the Create Index DDL + + ``` + CREATE INDEX index_sales + ON TABLE sales(user_id) + AS + 'carbondata' + TBLPROPERTIES('table_blocksize'='1') + ``` +**NOTE**: + * Secondary Index tables are registered with hive and stored n HiveSERDEPROPERTIES s json formatted + value. Maximum characters supported for SERDEPROPERTIES by hive is 4000 characters which cannot be + changed. + + +#### How SI tables are selected + +When a user filter query is submitted, during query planning phase, CarbonData with help of Review comment: ```suggestion When a user executes a filter query, during query planning phase, CarbonData with help of ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [hidden email] With regards, Apache Git Services |
|
|
In reply to this post by GitBox
Indhumathi27 commented on a change in pull request #3689: [CARBONDATA-3680]Add Secondary Index Document
URL: https://github.com/apache/carbondata/pull/3689#discussion_r401371348 ########## File path: docs/index/secondary-index-guide.md ########## @@ -0,0 +1,155 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to you under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +# CarbonData Secondary Index + +* [Quick Example](#quick-example) +* [Secondary Index Table](#Secondary-Index-Introduction) +* [Loading Data](#loading-data) +* [Querying Data](#querying-data) +* [Compaction](#compacting-SI-table) +* [Data Management](#data-management-with-SI-tables) + +## Quick example + +Start spark-sql in terminal and run the following queries, +``` +CREATE TABLE maintable(a int, b string, c string) stored as carbondata; +insert into maintable select 1, 'ab', 'cd'; +CREATE index inex1 on table maintable(c) AS 'carbondata'; +SELECT a from maintable where c = 'cd'; +// NOTE: run explain query and check if query hits the SI table from the plan +EXPLAIN SELECT a from maintable where c = 'cd'; +``` + +## Secondary Index Introduction + Sencondary index tables are created as a child table and manager as tables internally by Carbondata. Review comment: ```suggestion Sencondary index tables are created as indexes and managed as child tables internally by Carbondata. ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [hidden email] With regards, Apache Git Services |
|
|
In reply to this post by GitBox
Indhumathi27 commented on a change in pull request #3689: [CARBONDATA-3680]Add Secondary Index Document
URL: https://github.com/apache/carbondata/pull/3689#discussion_r401379775 ########## File path: docs/index/secondary-index-guide.md ########## @@ -0,0 +1,155 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to you under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +# CarbonData Secondary Index + +* [Quick Example](#quick-example) +* [Secondary Index Table](#Secondary-Index-Introduction) +* [Loading Data](#loading-data) +* [Querying Data](#querying-data) +* [Compaction](#compacting-SI-table) +* [Data Management](#data-management-with-SI-tables) Review comment: Please remove Data Management ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [hidden email] With regards, Apache Git Services |
|
|
In reply to this post by GitBox
Indhumathi27 commented on a change in pull request #3689: [CARBONDATA-3680]Add Secondary Index Document
URL: https://github.com/apache/carbondata/pull/3689#discussion_r401383769 ########## File path: docs/index/secondary-index-guide.md ########## @@ -0,0 +1,155 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to you under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +# CarbonData Secondary Index + +* [Quick Example](#quick-example) +* [Secondary Index Table](#Secondary-Index-Introduction) +* [Loading Data](#loading-data) +* [Querying Data](#querying-data) +* [Compaction](#compacting-SI-table) +* [Data Management](#data-management-with-SI-tables) + +## Quick example + +Start spark-sql in terminal and run the following queries, +``` +CREATE TABLE maintable(a int, b string, c string) stored as carbondata; +insert into maintable select 1, 'ab', 'cd'; +CREATE index inex1 on table maintable(c) AS 'carbondata'; +SELECT a from maintable where c = 'cd'; +// NOTE: run explain query and check if query hits the SI table from the plan +EXPLAIN SELECT a from maintable where c = 'cd'; +``` + +## Secondary Index Introduction + Sencondary index tables are created as a child table and manager as tables internally by Carbondata. + User can create the SI tables based on the column position in table(Recommended for right columns) + and the queries should have filter on that column to improve the filter query performance. + + SI tables will always be loaded non-lazy way. Once SI table is created, Carbondata's + CarbonOptimizer with the help of CarbonSITransformationRule, transforms the query plan to hit the + SI table based on the filter condition or set of filter conditions present in the query. + So first level of pruning will be done on SI table as it stores blocklets and main table/parent + table pruning will be based on the SI output, which helps in giving the faster query results with + better pruning. + + For instance, main table called **sales** which is defined as + + ``` + CREATE TABLE sales ( + order_time timestamp, + user_id string, + sex string, + country string, + quantity int, + price bigint) + STORED AS carbondata + ``` + + User can create SI table using the Create Index DDL + + ``` + CREATE INDEX index_sales + ON TABLE sales(user_id) + AS + 'carbondata' + TBLPROPERTIES('table_blocksize'='1') + ``` +**NOTE**: + * Secondary Index tables are registered with hive and stored n HiveSERDEPROPERTIES s json formatted + value. Maximum characters supported for SERDEPROPERTIES by hive is 4000 characters which cannot be + changed. + + +#### How SI tables are selected + +When a user filter query is submitted, during query planning phase, CarbonData with help of +CarbonSITransformationRule, checkes if there are any index tables present on the filter column of +query. If there are any, then filter query plan will be transformed such a way that, execution will +first hit the corresponding SI table and give input to main table for further pruning. + + +For the main table **sales** and SI table **index_sales** created above, following queries +``` +SELECT country, sex from sales where user_id = 'xxx' + +SELECT country, sex from sales where user_id = 'xxx' and country = 'INDIA' +``` + +will be transformed by CarbonData's CarbonSITransformationRule to query against SI table +**index_sales** first which will be input to the main table **sales** + + +## Loading data + +### Loading data to Secondary Index table(s). + +*case1:* When SI table is created and the main table does not have any data. In this case every +consecutive load will load to SI table once main table data load is finished. + +*case2:* When SI table is created and main table already contains some data, then SI creation will +also load to SI table with same number of segments as main table. There after, consecutive load to +main table will load to SI table also. + + **NOTE**: + * In case of any failure to SI table, then we make the SI table disable by setting a hive serde + property and subsequent load to main table will also load to SI with failed load data also and + make it enable and available for query. + +## Querying data +Direct query can be made on SI tables to see the data present in position reference columns. +When a filter query is fired, if the filter column is a secondary index column, then plan is +transformed accordingly to hit SI table first to make better pruning with main table and in turn +helps for faster query results. + +User can verify whether a query can leverage SI table or not by executing `EXPLAIN` +command, which will show the transformed logical plan, and thus user can check whether SI table +table is selected. + + +## Compacting SI table + +### Compacting SI table table through Main Table compaction +Running Compaction command (`ALTER TABLE COMPACT`)[COMPACTION TYPE-> MINOR/MAJOR] on main table will +automatically delete all the old segments of SI and creates a new segment with same name as main +table compacted segmet and loads data to it. + +### Compacting SI table's individual segment(s) through REBUILD command Review comment: REbuild command is not available for SI. Users have to enable carbon property "carbon.si.segment.merge" to enable SI segment Compaction / merge small files ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [hidden email] With regards, Apache Git Services |
|
|
In reply to this post by GitBox
Indhumathi27 commented on a change in pull request #3689: [CARBONDATA-3680]Add Secondary Index Document
URL: https://github.com/apache/carbondata/pull/3689#discussion_r401379282 ########## File path: docs/index/secondary-index-guide.md ########## @@ -0,0 +1,155 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to you under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +# CarbonData Secondary Index + +* [Quick Example](#quick-example) +* [Secondary Index Table](#Secondary-Index-Introduction) +* [Loading Data](#loading-data) +* [Querying Data](#querying-data) +* [Compaction](#compacting-SI-table) +* [Data Management](#data-management-with-SI-tables) + +## Quick example + +Start spark-sql in terminal and run the following queries, +``` +CREATE TABLE maintable(a int, b string, c string) stored as carbondata; +insert into maintable select 1, 'ab', 'cd'; +CREATE index inex1 on table maintable(c) AS 'carbondata'; +SELECT a from maintable where c = 'cd'; +// NOTE: run explain query and check if query hits the SI table from the plan +EXPLAIN SELECT a from maintable where c = 'cd'; +``` + +## Secondary Index Introduction + Sencondary index tables are created as a child table and manager as tables internally by Carbondata. + User can create the SI tables based on the column position in table(Recommended for right columns) + and the queries should have filter on that column to improve the filter query performance. + + SI tables will always be loaded non-lazy way. Once SI table is created, Carbondata's + CarbonOptimizer with the help of CarbonSITransformationRule, transforms the query plan to hit the + SI table based on the filter condition or set of filter conditions present in the query. + So first level of pruning will be done on SI table as it stores blocklets and main table/parent + table pruning will be based on the SI output, which helps in giving the faster query results with + better pruning. + + For instance, main table called **sales** which is defined as + + ``` + CREATE TABLE sales ( + order_time timestamp, + user_id string, + sex string, + country string, + quantity int, + price bigint) + STORED AS carbondata + ``` + + User can create SI table using the Create Index DDL + + ``` + CREATE INDEX index_sales + ON TABLE sales(user_id) + AS + 'carbondata' + TBLPROPERTIES('table_blocksize'='1') + ``` +**NOTE**: + * Secondary Index tables are registered with hive and stored n HiveSERDEPROPERTIES s json formatted + value. Maximum characters supported for SERDEPROPERTIES by hive is 4000 characters which cannot be + changed. + + +#### How SI tables are selected + +When a user filter query is submitted, during query planning phase, CarbonData with help of +CarbonSITransformationRule, checkes if there are any index tables present on the filter column of +query. If there are any, then filter query plan will be transformed such a way that, execution will +first hit the corresponding SI table and give input to main table for further pruning. + + +For the main table **sales** and SI table **index_sales** created above, following queries +``` +SELECT country, sex from sales where user_id = 'xxx' + +SELECT country, sex from sales where user_id = 'xxx' and country = 'INDIA' +``` + +will be transformed by CarbonData's CarbonSITransformationRule to query against SI table +**index_sales** first which will be input to the main table **sales** + + +## Loading data + +### Loading data to Secondary Index table(s). + +*case1:* When SI table is created and the main table does not have any data. In this case every +consecutive load will load to SI table once main table data load is finished. + +*case2:* When SI table is created and main table already contains some data, then SI creation will +also load to SI table with same number of segments as main table. There after, consecutive load to +main table will load to SI table also. + + **NOTE**: + * In case of any failure to SI table, then we make the SI table disable by setting a hive serde + property and subsequent load to main table will also load to SI with failed load data also and Review comment: ```suggestion property. Users can fire REFRESH INDEX command to sync or else the subsequent main table load will load the old failed loads along with current load and makes the SI table enable and available for query ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [hidden email] With regards, Apache Git Services |
|
|
In reply to this post by GitBox
Indhumathi27 commented on a change in pull request #3689: [CARBONDATA-3680]Add Secondary Index Document
URL: https://github.com/apache/carbondata/pull/3689#discussion_r401380164 ########## File path: docs/index/secondary-index-guide.md ########## @@ -0,0 +1,155 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to you under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +# CarbonData Secondary Index + +* [Quick Example](#quick-example) +* [Secondary Index Table](#Secondary-Index-Introduction) +* [Loading Data](#loading-data) +* [Querying Data](#querying-data) +* [Compaction](#compacting-SI-table) +* [Data Management](#data-management-with-SI-tables) Review comment: Can add document for REFRESH INDEX, REGISTER INDEX, SHOW INDEX, DROP INDEX commands also ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [hidden email] With regards, Apache Git Services |
|
|
In reply to this post by GitBox
Indhumathi27 commented on a change in pull request #3689: [CARBONDATA-3680]Add Secondary Index Document
URL: https://github.com/apache/carbondata/pull/3689#discussion_r401381166 ########## File path: docs/index/secondary-index-guide.md ########## @@ -0,0 +1,155 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to you under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +# CarbonData Secondary Index + +* [Quick Example](#quick-example) +* [Secondary Index Table](#Secondary-Index-Introduction) +* [Loading Data](#loading-data) +* [Querying Data](#querying-data) +* [Compaction](#compacting-SI-table) +* [Data Management](#data-management-with-SI-tables) + +## Quick example + +Start spark-sql in terminal and run the following queries, +``` +CREATE TABLE maintable(a int, b string, c string) stored as carbondata; +insert into maintable select 1, 'ab', 'cd'; +CREATE index inex1 on table maintable(c) AS 'carbondata'; +SELECT a from maintable where c = 'cd'; +// NOTE: run explain query and check if query hits the SI table from the plan +EXPLAIN SELECT a from maintable where c = 'cd'; +``` + +## Secondary Index Introduction + Sencondary index tables are created as a child table and manager as tables internally by Carbondata. + User can create the SI tables based on the column position in table(Recommended for right columns) + and the queries should have filter on that column to improve the filter query performance. + + SI tables will always be loaded non-lazy way. Once SI table is created, Carbondata's + CarbonOptimizer with the help of CarbonSITransformationRule, transforms the query plan to hit the + SI table based on the filter condition or set of filter conditions present in the query. + So first level of pruning will be done on SI table as it stores blocklets and main table/parent + table pruning will be based on the SI output, which helps in giving the faster query results with + better pruning. + + For instance, main table called **sales** which is defined as + + ``` + CREATE TABLE sales ( + order_time timestamp, + user_id string, + sex string, + country string, + quantity int, + price bigint) + STORED AS carbondata + ``` + + User can create SI table using the Create Index DDL + + ``` + CREATE INDEX index_sales + ON TABLE sales(user_id) + AS + 'carbondata' + TBLPROPERTIES('table_blocksize'='1') + ``` +**NOTE**: + * Secondary Index tables are registered with hive and stored n HiveSERDEPROPERTIES s json formatted + value. Maximum characters supported for SERDEPROPERTIES by hive is 4000 characters which cannot be + changed. + + +#### How SI tables are selected + +When a user filter query is submitted, during query planning phase, CarbonData with help of +CarbonSITransformationRule, checkes if there are any index tables present on the filter column of +query. If there are any, then filter query plan will be transformed such a way that, execution will +first hit the corresponding SI table and give input to main table for further pruning. + + +For the main table **sales** and SI table **index_sales** created above, following queries +``` +SELECT country, sex from sales where user_id = 'xxx' + +SELECT country, sex from sales where user_id = 'xxx' and country = 'INDIA' +``` + +will be transformed by CarbonData's CarbonSITransformationRule to query against SI table +**index_sales** first which will be input to the main table **sales** + + +## Loading data + +### Loading data to Secondary Index table(s). + +*case1:* When SI table is created and the main table does not have any data. In this case every +consecutive load will load to SI table once main table data load is finished. + +*case2:* When SI table is created and main table already contains some data, then SI creation will +also load to SI table with same number of segments as main table. There after, consecutive load to +main table will load to SI table also. + + **NOTE**: + * In case of any failure to SI table, then we make the SI table disable by setting a hive serde + property and subsequent load to main table will also load to SI with failed load data also and + make it enable and available for query. + +## Querying data +Direct query can be made on SI tables to see the data present in position reference columns. +When a filter query is fired, if the filter column is a secondary index column, then plan is +transformed accordingly to hit SI table first to make better pruning with main table and in turn +helps for faster query results. + +User can verify whether a query can leverage SI table or not by executing `EXPLAIN` +command, which will show the transformed logical plan, and thus user can check whether SI table +table is selected. Review comment: Can add documentation for queries which do not push down to SI ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [hidden email] With regards, Apache Git Services |
|
|
In reply to this post by GitBox
Indhumathi27 commented on a change in pull request #3689: [CARBONDATA-3680]Add Secondary Index Document
URL: https://github.com/apache/carbondata/pull/3689#discussion_r401371348 ########## File path: docs/index/secondary-index-guide.md ########## @@ -0,0 +1,155 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to you under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +# CarbonData Secondary Index + +* [Quick Example](#quick-example) +* [Secondary Index Table](#Secondary-Index-Introduction) +* [Loading Data](#loading-data) +* [Querying Data](#querying-data) +* [Compaction](#compacting-SI-table) +* [Data Management](#data-management-with-SI-tables) + +## Quick example + +Start spark-sql in terminal and run the following queries, +``` +CREATE TABLE maintable(a int, b string, c string) stored as carbondata; +insert into maintable select 1, 'ab', 'cd'; +CREATE index inex1 on table maintable(c) AS 'carbondata'; +SELECT a from maintable where c = 'cd'; +// NOTE: run explain query and check if query hits the SI table from the plan +EXPLAIN SELECT a from maintable where c = 'cd'; +``` + +## Secondary Index Introduction + Sencondary index tables are created as a child table and manager as tables internally by Carbondata. Review comment: ```suggestion Secondary index tables are created as indexes and managed as child tables internally by Carbondata. ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [hidden email] With regards, Apache Git Services |
|
|
In reply to this post by GitBox
akashrn5 commented on a change in pull request #3689: [CARBONDATA-3680]Add Secondary Index Document
URL: https://github.com/apache/carbondata/pull/3689#discussion_r401407346 ########## File path: docs/index/secondary-index-guide.md ########## @@ -0,0 +1,155 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to you under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +# CarbonData Secondary Index + +* [Quick Example](#quick-example) +* [Secondary Index Table](#Secondary-Index-Introduction) +* [Loading Data](#loading-data) +* [Querying Data](#querying-data) +* [Compaction](#compacting-SI-table) +* [Data Management](#data-management-with-SI-tables) + +## Quick example + +Start spark-sql in terminal and run the following queries, +``` +CREATE TABLE maintable(a int, b string, c string) stored as carbondata; +insert into maintable select 1, 'ab', 'cd'; +CREATE index inex1 on table maintable(c) AS 'carbondata'; +SELECT a from maintable where c = 'cd'; +// NOTE: run explain query and check if query hits the SI table from the plan +EXPLAIN SELECT a from maintable where c = 'cd'; +``` + +## Secondary Index Introduction + Sencondary index tables are created as a child table and manager as tables internally by Carbondata. + User can create the SI tables based on the column position in table(Recommended for right columns) + and the queries should have filter on that column to improve the filter query performance. + + SI tables will always be loaded non-lazy way. Once SI table is created, Carbondata's + CarbonOptimizer with the help of CarbonSITransformationRule, transforms the query plan to hit the + SI table based on the filter condition or set of filter conditions present in the query. + So first level of pruning will be done on SI table as it stores blocklets and main table/parent + table pruning will be based on the SI output, which helps in giving the faster query results with + better pruning. + + For instance, main table called **sales** which is defined as + + ``` + CREATE TABLE sales ( + order_time timestamp, + user_id string, + sex string, + country string, + quantity int, + price bigint) + STORED AS carbondata + ``` + + User can create SI table using the Create Index DDL + + ``` + CREATE INDEX index_sales + ON TABLE sales(user_id) + AS + 'carbondata' + TBLPROPERTIES('table_blocksize'='1') + ``` +**NOTE**: + * Secondary Index tables are registered with hive and stored n HiveSERDEPROPERTIES s json formatted + value. Maximum characters supported for SERDEPROPERTIES by hive is 4000 characters which cannot be + changed. + + +#### How SI tables are selected + +When a user filter query is submitted, during query planning phase, CarbonData with help of +CarbonSITransformationRule, checkes if there are any index tables present on the filter column of +query. If there are any, then filter query plan will be transformed such a way that, execution will +first hit the corresponding SI table and give input to main table for further pruning. + + +For the main table **sales** and SI table **index_sales** created above, following queries +``` +SELECT country, sex from sales where user_id = 'xxx' + +SELECT country, sex from sales where user_id = 'xxx' and country = 'INDIA' +``` + +will be transformed by CarbonData's CarbonSITransformationRule to query against SI table +**index_sales** first which will be input to the main table **sales** + + +## Loading data + +### Loading data to Secondary Index table(s). + +*case1:* When SI table is created and the main table does not have any data. In this case every +consecutive load will load to SI table once main table data load is finished. + +*case2:* When SI table is created and main table already contains some data, then SI creation will +also load to SI table with same number of segments as main table. There after, consecutive load to +main table will load to SI table also. + + **NOTE**: + * In case of any failure to SI table, then we make the SI table disable by setting a hive serde + property and subsequent load to main table will also load to SI with failed load data also and + make it enable and available for query. + +## Querying data +Direct query can be made on SI tables to see the data present in position reference columns. +When a filter query is fired, if the filter column is a secondary index column, then plan is +transformed accordingly to hit SI table first to make better pruning with main table and in turn +helps for faster query results. + +User can verify whether a query can leverage SI table or not by executing `EXPLAIN` +command, which will show the transformed logical plan, and thus user can check whether SI table +table is selected. Review comment: thats already explained right, if no filester on SI, then it will not push down no need to add more queries as eamples ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [hidden email] With regards, Apache Git Services |
|
|
In reply to this post by GitBox
akashrn5 commented on a change in pull request #3689: [CARBONDATA-3680]Add Secondary Index Document
URL: https://github.com/apache/carbondata/pull/3689#discussion_r401407346 ########## File path: docs/index/secondary-index-guide.md ########## @@ -0,0 +1,155 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to you under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +# CarbonData Secondary Index + +* [Quick Example](#quick-example) +* [Secondary Index Table](#Secondary-Index-Introduction) +* [Loading Data](#loading-data) +* [Querying Data](#querying-data) +* [Compaction](#compacting-SI-table) +* [Data Management](#data-management-with-SI-tables) + +## Quick example + +Start spark-sql in terminal and run the following queries, +``` +CREATE TABLE maintable(a int, b string, c string) stored as carbondata; +insert into maintable select 1, 'ab', 'cd'; +CREATE index inex1 on table maintable(c) AS 'carbondata'; +SELECT a from maintable where c = 'cd'; +// NOTE: run explain query and check if query hits the SI table from the plan +EXPLAIN SELECT a from maintable where c = 'cd'; +``` + +## Secondary Index Introduction + Sencondary index tables are created as a child table and manager as tables internally by Carbondata. + User can create the SI tables based on the column position in table(Recommended for right columns) + and the queries should have filter on that column to improve the filter query performance. + + SI tables will always be loaded non-lazy way. Once SI table is created, Carbondata's + CarbonOptimizer with the help of CarbonSITransformationRule, transforms the query plan to hit the + SI table based on the filter condition or set of filter conditions present in the query. + So first level of pruning will be done on SI table as it stores blocklets and main table/parent + table pruning will be based on the SI output, which helps in giving the faster query results with + better pruning. + + For instance, main table called **sales** which is defined as + + ``` + CREATE TABLE sales ( + order_time timestamp, + user_id string, + sex string, + country string, + quantity int, + price bigint) + STORED AS carbondata + ``` + + User can create SI table using the Create Index DDL + + ``` + CREATE INDEX index_sales + ON TABLE sales(user_id) + AS + 'carbondata' + TBLPROPERTIES('table_blocksize'='1') + ``` +**NOTE**: + * Secondary Index tables are registered with hive and stored n HiveSERDEPROPERTIES s json formatted + value. Maximum characters supported for SERDEPROPERTIES by hive is 4000 characters which cannot be + changed. + + +#### How SI tables are selected + +When a user filter query is submitted, during query planning phase, CarbonData with help of +CarbonSITransformationRule, checkes if there are any index tables present on the filter column of +query. If there are any, then filter query plan will be transformed such a way that, execution will +first hit the corresponding SI table and give input to main table for further pruning. + + +For the main table **sales** and SI table **index_sales** created above, following queries +``` +SELECT country, sex from sales where user_id = 'xxx' + +SELECT country, sex from sales where user_id = 'xxx' and country = 'INDIA' +``` + +will be transformed by CarbonData's CarbonSITransformationRule to query against SI table +**index_sales** first which will be input to the main table **sales** + + +## Loading data + +### Loading data to Secondary Index table(s). + +*case1:* When SI table is created and the main table does not have any data. In this case every +consecutive load will load to SI table once main table data load is finished. + +*case2:* When SI table is created and main table already contains some data, then SI creation will +also load to SI table with same number of segments as main table. There after, consecutive load to +main table will load to SI table also. + + **NOTE**: + * In case of any failure to SI table, then we make the SI table disable by setting a hive serde + property and subsequent load to main table will also load to SI with failed load data also and + make it enable and available for query. + +## Querying data +Direct query can be made on SI tables to see the data present in position reference columns. +When a filter query is fired, if the filter column is a secondary index column, then plan is +transformed accordingly to hit SI table first to make better pruning with main table and in turn +helps for faster query results. + +User can verify whether a query can leverage SI table or not by executing `EXPLAIN` +command, which will show the transformed logical plan, and thus user can check whether SI table +table is selected. Review comment: thats already explained right, if no filter on SI, then it will not push down no need to add more queries as eamples ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [hidden email] With regards, Apache Git Services |
|
|
In reply to this post by GitBox
akashrn5 commented on a change in pull request #3689: [CARBONDATA-3680]Add Secondary Index Document
URL: https://github.com/apache/carbondata/pull/3689#discussion_r401408219 ########## File path: docs/index/secondary-index-guide.md ########## @@ -0,0 +1,155 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to you under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +# CarbonData Secondary Index + +* [Quick Example](#quick-example) +* [Secondary Index Table](#Secondary-Index-Introduction) +* [Loading Data](#loading-data) +* [Querying Data](#querying-data) +* [Compaction](#compacting-SI-table) +* [Data Management](#data-management-with-SI-tables) + +## Quick example + +Start spark-sql in terminal and run the following queries, +``` +CREATE TABLE maintable(a int, b string, c string) stored as carbondata; +insert into maintable select 1, 'ab', 'cd'; +CREATE index inex1 on table maintable(c) AS 'carbondata'; +SELECT a from maintable where c = 'cd'; +// NOTE: run explain query and check if query hits the SI table from the plan +EXPLAIN SELECT a from maintable where c = 'cd'; +``` + +## Secondary Index Introduction + Sencondary index tables are created as a child table and manager as tables internally by Carbondata. + User can create the SI tables based on the column position in table(Recommended for right columns) + and the queries should have filter on that column to improve the filter query performance. + + SI tables will always be loaded non-lazy way. Once SI table is created, Carbondata's + CarbonOptimizer with the help of CarbonSITransformationRule, transforms the query plan to hit the + SI table based on the filter condition or set of filter conditions present in the query. + So first level of pruning will be done on SI table as it stores blocklets and main table/parent + table pruning will be based on the SI output, which helps in giving the faster query results with + better pruning. + + For instance, main table called **sales** which is defined as + + ``` + CREATE TABLE sales ( + order_time timestamp, + user_id string, + sex string, + country string, + quantity int, + price bigint) + STORED AS carbondata + ``` + + User can create SI table using the Create Index DDL + + ``` + CREATE INDEX index_sales + ON TABLE sales(user_id) + AS + 'carbondata' + TBLPROPERTIES('table_blocksize'='1') + ``` +**NOTE**: + * Secondary Index tables are registered with hive and stored n HiveSERDEPROPERTIES s json formatted + value. Maximum characters supported for SERDEPROPERTIES by hive is 4000 characters which cannot be + changed. + + +#### How SI tables are selected + +When a user filter query is submitted, during query planning phase, CarbonData with help of +CarbonSITransformationRule, checkes if there are any index tables present on the filter column of +query. If there are any, then filter query plan will be transformed such a way that, execution will +first hit the corresponding SI table and give input to main table for further pruning. + + +For the main table **sales** and SI table **index_sales** created above, following queries +``` +SELECT country, sex from sales where user_id = 'xxx' + +SELECT country, sex from sales where user_id = 'xxx' and country = 'INDIA' +``` + +will be transformed by CarbonData's CarbonSITransformationRule to query against SI table +**index_sales** first which will be input to the main table **sales** + + +## Loading data + +### Loading data to Secondary Index table(s). + +*case1:* When SI table is created and the main table does not have any data. In this case every +consecutive load will load to SI table once main table data load is finished. + +*case2:* When SI table is created and main table already contains some data, then SI creation will +also load to SI table with same number of segments as main table. There after, consecutive load to +main table will load to SI table also. + + **NOTE**: + * In case of any failure to SI table, then we make the SI table disable by setting a hive serde + property and subsequent load to main table will also load to SI with failed load data also and + make it enable and available for query. + +## Querying data +Direct query can be made on SI tables to see the data present in position reference columns. +When a filter query is fired, if the filter column is a secondary index column, then plan is +transformed accordingly to hit SI table first to make better pruning with main table and in turn +helps for faster query results. + +User can verify whether a query can leverage SI table or not by executing `EXPLAIN` +command, which will show the transformed logical plan, and thus user can check whether SI table +table is selected. + + +## Compacting SI table + +### Compacting SI table table through Main Table compaction +Running Compaction command (`ALTER TABLE COMPACT`)[COMPACTION TYPE-> MINOR/MAJOR] on main table will +automatically delete all the old segments of SI and creates a new segment with same name as main +table compacted segmet and loads data to it. + +### Compacting SI table's individual segment(s) through REBUILD command Review comment: no we have a rebuild command, please refer org.apache.spark.sql.secondaryindex.command.SIRebuildSegmentCommand ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [hidden email] With regards, Apache Git Services |
|
|
In reply to this post by GitBox
Indhumathi27 commented on a change in pull request #3689: [CARBONDATA-3680]Add Secondary Index Document
URL: https://github.com/apache/carbondata/pull/3689#discussion_r401380164 ########## File path: docs/index/secondary-index-guide.md ########## @@ -0,0 +1,155 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to you under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +# CarbonData Secondary Index + +* [Quick Example](#quick-example) +* [Secondary Index Table](#Secondary-Index-Introduction) +* [Loading Data](#loading-data) +* [Querying Data](#querying-data) +* [Compaction](#compacting-SI-table) +* [Data Management](#data-management-with-SI-tables) Review comment: Can add document for REGISTER INDEX, SHOW INDEX, DROP INDEX commands also ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [hidden email] With regards, Apache Git Services |
|
|
In reply to this post by GitBox
Indhumathi27 commented on a change in pull request #3689: [CARBONDATA-3680]Add Secondary Index Document
URL: https://github.com/apache/carbondata/pull/3689#discussion_r401411591 ########## File path: docs/index/secondary-index-guide.md ########## @@ -0,0 +1,155 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to you under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +# CarbonData Secondary Index + +* [Quick Example](#quick-example) +* [Secondary Index Table](#Secondary-Index-Introduction) +* [Loading Data](#loading-data) +* [Querying Data](#querying-data) +* [Compaction](#compacting-SI-table) +* [Data Management](#data-management-with-SI-tables) + +## Quick example + +Start spark-sql in terminal and run the following queries, +``` +CREATE TABLE maintable(a int, b string, c string) stored as carbondata; +insert into maintable select 1, 'ab', 'cd'; +CREATE index inex1 on table maintable(c) AS 'carbondata'; +SELECT a from maintable where c = 'cd'; +// NOTE: run explain query and check if query hits the SI table from the plan +EXPLAIN SELECT a from maintable where c = 'cd'; +``` + +## Secondary Index Introduction + Sencondary index tables are created as a child table and manager as tables internally by Carbondata. + User can create the SI tables based on the column position in table(Recommended for right columns) + and the queries should have filter on that column to improve the filter query performance. + + SI tables will always be loaded non-lazy way. Once SI table is created, Carbondata's + CarbonOptimizer with the help of CarbonSITransformationRule, transforms the query plan to hit the + SI table based on the filter condition or set of filter conditions present in the query. + So first level of pruning will be done on SI table as it stores blocklets and main table/parent + table pruning will be based on the SI output, which helps in giving the faster query results with + better pruning. + + For instance, main table called **sales** which is defined as + + ``` + CREATE TABLE sales ( + order_time timestamp, + user_id string, + sex string, + country string, + quantity int, + price bigint) + STORED AS carbondata + ``` + + User can create SI table using the Create Index DDL + + ``` + CREATE INDEX index_sales + ON TABLE sales(user_id) + AS + 'carbondata' + TBLPROPERTIES('table_blocksize'='1') + ``` +**NOTE**: + * Secondary Index tables are registered with hive and stored n HiveSERDEPROPERTIES s json formatted + value. Maximum characters supported for SERDEPROPERTIES by hive is 4000 characters which cannot be + changed. + + +#### How SI tables are selected + +When a user filter query is submitted, during query planning phase, CarbonData with help of +CarbonSITransformationRule, checkes if there are any index tables present on the filter column of +query. If there are any, then filter query plan will be transformed such a way that, execution will +first hit the corresponding SI table and give input to main table for further pruning. + + +For the main table **sales** and SI table **index_sales** created above, following queries +``` +SELECT country, sex from sales where user_id = 'xxx' + +SELECT country, sex from sales where user_id = 'xxx' and country = 'INDIA' +``` + +will be transformed by CarbonData's CarbonSITransformationRule to query against SI table +**index_sales** first which will be input to the main table **sales** + + +## Loading data + +### Loading data to Secondary Index table(s). + +*case1:* When SI table is created and the main table does not have any data. In this case every +consecutive load will load to SI table once main table data load is finished. + +*case2:* When SI table is created and main table already contains some data, then SI creation will +also load to SI table with same number of segments as main table. There after, consecutive load to +main table will load to SI table also. + + **NOTE**: + * In case of any failure to SI table, then we make the SI table disable by setting a hive serde + property and subsequent load to main table will also load to SI with failed load data also and + make it enable and available for query. + +## Querying data +Direct query can be made on SI tables to see the data present in position reference columns. +When a filter query is fired, if the filter column is a secondary index column, then plan is +transformed accordingly to hit SI table first to make better pruning with main table and in turn +helps for faster query results. + +User can verify whether a query can leverage SI table or not by executing `EXPLAIN` +command, which will show the transformed logical plan, and thus user can check whether SI table +table is selected. Review comment: with filter on SI and donotPushToSI. For expressions like NOT, ISNOTNULL etc ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [hidden email] With regards, Apache Git Services |
|
|
In reply to this post by GitBox
Indhumathi27 commented on a change in pull request #3689: [CARBONDATA-3680]Add Secondary Index Document
URL: https://github.com/apache/carbondata/pull/3689#discussion_r401383769 ########## File path: docs/index/secondary-index-guide.md ########## @@ -0,0 +1,155 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to you under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +# CarbonData Secondary Index + +* [Quick Example](#quick-example) +* [Secondary Index Table](#Secondary-Index-Introduction) +* [Loading Data](#loading-data) +* [Querying Data](#querying-data) +* [Compaction](#compacting-SI-table) +* [Data Management](#data-management-with-SI-tables) + +## Quick example + +Start spark-sql in terminal and run the following queries, +``` +CREATE TABLE maintable(a int, b string, c string) stored as carbondata; +insert into maintable select 1, 'ab', 'cd'; +CREATE index inex1 on table maintable(c) AS 'carbondata'; +SELECT a from maintable where c = 'cd'; +// NOTE: run explain query and check if query hits the SI table from the plan +EXPLAIN SELECT a from maintable where c = 'cd'; +``` + +## Secondary Index Introduction + Sencondary index tables are created as a child table and manager as tables internally by Carbondata. + User can create the SI tables based on the column position in table(Recommended for right columns) + and the queries should have filter on that column to improve the filter query performance. + + SI tables will always be loaded non-lazy way. Once SI table is created, Carbondata's + CarbonOptimizer with the help of CarbonSITransformationRule, transforms the query plan to hit the + SI table based on the filter condition or set of filter conditions present in the query. + So first level of pruning will be done on SI table as it stores blocklets and main table/parent + table pruning will be based on the SI output, which helps in giving the faster query results with + better pruning. + + For instance, main table called **sales** which is defined as + + ``` + CREATE TABLE sales ( + order_time timestamp, + user_id string, + sex string, + country string, + quantity int, + price bigint) + STORED AS carbondata + ``` + + User can create SI table using the Create Index DDL + + ``` + CREATE INDEX index_sales + ON TABLE sales(user_id) + AS + 'carbondata' + TBLPROPERTIES('table_blocksize'='1') + ``` +**NOTE**: + * Secondary Index tables are registered with hive and stored n HiveSERDEPROPERTIES s json formatted + value. Maximum characters supported for SERDEPROPERTIES by hive is 4000 characters which cannot be + changed. + + +#### How SI tables are selected + +When a user filter query is submitted, during query planning phase, CarbonData with help of +CarbonSITransformationRule, checkes if there are any index tables present on the filter column of +query. If there are any, then filter query plan will be transformed such a way that, execution will +first hit the corresponding SI table and give input to main table for further pruning. + + +For the main table **sales** and SI table **index_sales** created above, following queries +``` +SELECT country, sex from sales where user_id = 'xxx' + +SELECT country, sex from sales where user_id = 'xxx' and country = 'INDIA' +``` + +will be transformed by CarbonData's CarbonSITransformationRule to query against SI table +**index_sales** first which will be input to the main table **sales** + + +## Loading data + +### Loading data to Secondary Index table(s). + +*case1:* When SI table is created and the main table does not have any data. In this case every +consecutive load will load to SI table once main table data load is finished. + +*case2:* When SI table is created and main table already contains some data, then SI creation will +also load to SI table with same number of segments as main table. There after, consecutive load to +main table will load to SI table also. + + **NOTE**: + * In case of any failure to SI table, then we make the SI table disable by setting a hive serde + property and subsequent load to main table will also load to SI with failed load data also and + make it enable and available for query. + +## Querying data +Direct query can be made on SI tables to see the data present in position reference columns. +When a filter query is fired, if the filter column is a secondary index column, then plan is +transformed accordingly to hit SI table first to make better pruning with main table and in turn +helps for faster query results. + +User can verify whether a query can leverage SI table or not by executing `EXPLAIN` +command, which will show the transformed logical plan, and thus user can check whether SI table +table is selected. + + +## Compacting SI table + +### Compacting SI table table through Main Table compaction +Running Compaction command (`ALTER TABLE COMPACT`)[COMPACTION TYPE-> MINOR/MAJOR] on main table will +automatically delete all the old segments of SI and creates a new segment with same name as main +table compacted segmet and loads data to it. + +### Compacting SI table's individual segment(s) through REBUILD command Review comment: REbuild command is not available for SI. Users have to enable carbon property "carbon.si.segment.merge" to enable SI segment Compaction / merge small files ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [hidden email] With regards, Apache Git Services |
|
|
In reply to this post by GitBox
Indhumathi27 commented on a change in pull request #3689: [CARBONDATA-3680]Add Secondary Index Document
URL: https://github.com/apache/carbondata/pull/3689#discussion_r401412761 ########## File path: docs/index/secondary-index-guide.md ########## @@ -0,0 +1,155 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to you under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +# CarbonData Secondary Index + +* [Quick Example](#quick-example) +* [Secondary Index Table](#Secondary-Index-Introduction) +* [Loading Data](#loading-data) +* [Querying Data](#querying-data) +* [Compaction](#compacting-SI-table) +* [Data Management](#data-management-with-SI-tables) + +## Quick example + +Start spark-sql in terminal and run the following queries, +``` +CREATE TABLE maintable(a int, b string, c string) stored as carbondata; +insert into maintable select 1, 'ab', 'cd'; +CREATE index inex1 on table maintable(c) AS 'carbondata'; +SELECT a from maintable where c = 'cd'; +// NOTE: run explain query and check if query hits the SI table from the plan +EXPLAIN SELECT a from maintable where c = 'cd'; +``` + +## Secondary Index Introduction + Sencondary index tables are created as a child table and manager as tables internally by Carbondata. + User can create the SI tables based on the column position in table(Recommended for right columns) + and the queries should have filter on that column to improve the filter query performance. + + SI tables will always be loaded non-lazy way. Once SI table is created, Carbondata's + CarbonOptimizer with the help of CarbonSITransformationRule, transforms the query plan to hit the + SI table based on the filter condition or set of filter conditions present in the query. + So first level of pruning will be done on SI table as it stores blocklets and main table/parent + table pruning will be based on the SI output, which helps in giving the faster query results with + better pruning. + + For instance, main table called **sales** which is defined as + + ``` + CREATE TABLE sales ( + order_time timestamp, + user_id string, + sex string, + country string, + quantity int, + price bigint) + STORED AS carbondata + ``` + + User can create SI table using the Create Index DDL + + ``` + CREATE INDEX index_sales + ON TABLE sales(user_id) + AS + 'carbondata' + TBLPROPERTIES('table_blocksize'='1') + ``` +**NOTE**: + * Secondary Index tables are registered with hive and stored n HiveSERDEPROPERTIES s json formatted + value. Maximum characters supported for SERDEPROPERTIES by hive is 4000 characters which cannot be + changed. + + +#### How SI tables are selected + +When a user filter query is submitted, during query planning phase, CarbonData with help of +CarbonSITransformationRule, checkes if there are any index tables present on the filter column of +query. If there are any, then filter query plan will be transformed such a way that, execution will +first hit the corresponding SI table and give input to main table for further pruning. + + +For the main table **sales** and SI table **index_sales** created above, following queries +``` +SELECT country, sex from sales where user_id = 'xxx' + +SELECT country, sex from sales where user_id = 'xxx' and country = 'INDIA' +``` + +will be transformed by CarbonData's CarbonSITransformationRule to query against SI table +**index_sales** first which will be input to the main table **sales** + + +## Loading data + +### Loading data to Secondary Index table(s). + +*case1:* When SI table is created and the main table does not have any data. In this case every +consecutive load will load to SI table once main table data load is finished. + +*case2:* When SI table is created and main table already contains some data, then SI creation will +also load to SI table with same number of segments as main table. There after, consecutive load to +main table will load to SI table also. + + **NOTE**: + * In case of any failure to SI table, then we make the SI table disable by setting a hive serde + property and subsequent load to main table will also load to SI with failed load data also and + make it enable and available for query. + +## Querying data +Direct query can be made on SI tables to see the data present in position reference columns. +When a filter query is fired, if the filter column is a secondary index column, then plan is +transformed accordingly to hit SI table first to make better pruning with main table and in turn +helps for faster query results. + +User can verify whether a query can leverage SI table or not by executing `EXPLAIN` +command, which will show the transformed logical plan, and thus user can check whether SI table +table is selected. + + +## Compacting SI table + +### Compacting SI table table through Main Table compaction +Running Compaction command (`ALTER TABLE COMPACT`)[COMPACTION TYPE-> MINOR/MAJOR] on main table will +automatically delete all the old segments of SI and creates a new segment with same name as main +table compacted segmet and loads data to it. + +### Compacting SI table's individual segment(s) through REBUILD command Review comment: sorry. only rebuild command. can ignore the comment ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [hidden email] With regards, Apache Git Services |
«
Return to Apache CarbonData JIRA issues
|
1 view|%1 views
| Free forum by Nabble | Edit this page |