[GitHub] [carbondata] jackylk opened a new pull request #3612: [WIP] Separate Materialized View command from DataMap command
Locked
12345
12345
 
|
CarbonDataQA1 commented on issue #3612: [CARBONDATA-3694] Separate Materialized View command from DataMap command
URL: https://github.com/apache/carbondata/pull/3612#issuecomment-586813087 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2015/ ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [hidden email] With regards, Apache Git Services |
|
|
In reply to this post by GitBox
Indhumathi27 commented on a change in pull request #3612: [CARBONDATA-3694] Separate Materialized View command from DataMap command
URL: https://github.com/apache/carbondata/pull/3612#discussion_r379997083 ########## File path: datamap/mv/core/src/main/scala/org/apache/carbondata/mv/extension/command/CreateMaterializedViewCommand.scala ########## @@ -0,0 +1,105 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.carbondata.mv.extension.command + +import scala.collection.JavaConverters._ +import scala.collection.mutable + +import org.apache.spark.sql._ +import org.apache.spark.sql.execution.command._ + +import org.apache.carbondata.common.exceptions.sql.MalformedMaterializedViewException +import org.apache.carbondata.common.logging.LogServiceFactory +import org.apache.carbondata.core.datamap.{DataMapProvider, DataMapStoreManager} +import org.apache.carbondata.core.datamap.status.DataMapStatusManager +import org.apache.carbondata.core.metadata.schema.datamap.{DataMapClassProvider, DataMapProperty} +import org.apache.carbondata.core.metadata.schema.table.DataMapSchema +import org.apache.carbondata.core.util.CarbonProperties +import org.apache.carbondata.datamap.DataMapManager +import org.apache.carbondata.events._ + +/** + * Create Materialized View Command implementation + * It will create the MV table, load the MV table (if deferred rebuild is false), + * and register the MV schema in [[DataMapStoreManager]] + */ +case class CreateMaterializedViewCommand( + mvName: String, + properties: Map[String, String], + queryString: Option[String], + ifNotExistsSet: Boolean = false, + deferredRebuild: Boolean = false) + extends AtomicRunnableCommand { + + private val LOGGER = LogServiceFactory.getLogService(this.getClass.getName) + private var dataMapProvider: DataMapProvider = _ + private var dataMapSchema: DataMapSchema = _ + + override def processMetadata(sparkSession: SparkSession): Seq[Row] = { + + setAuditInfo(Map("mvName" -> mvName) ++ properties) + + val mutableMap = mutable.Map[String, String](properties.toSeq: _*) + mutableMap.put(DataMapProperty.DEFERRED_REBUILD, deferredRebuild.toString) + + dataMapSchema = new DataMapSchema(mvName, DataMapClassProvider.MV.name()) + dataMapSchema.setProperties(mutableMap.asJava) + dataMapProvider = DataMapManager.get.getDataMapProvider(null, dataMapSchema, sparkSession) + if (DataMapStoreManager.getInstance().getAllDataMapSchemas.asScala + .exists(_.getDataMapName.equalsIgnoreCase(dataMapSchema.getDataMapName))) { + if (!ifNotExistsSet) { + throw new MalformedMaterializedViewException( + s"Materialized view with name ${dataMapSchema.getDataMapName} already exists") + } else { + return Seq.empty + } + } + + val systemFolderLocation: String = CarbonProperties.getInstance().getSystemFolderLocation + val operationContext: OperationContext = new OperationContext() + val preExecEvent = CreateDataMapPreExecutionEvent(sparkSession, systemFolderLocation, null) + OperationListenerBus.getInstance().fireEvent(preExecEvent, operationContext) + Review comment: Lazy datamaps has to be disabled after creation ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [hidden email] With regards, Apache Git Services |
|
|
In reply to this post by GitBox
Indhumathi27 commented on a change in pull request #3612: [CARBONDATA-3694] Separate Materialized View command from DataMap command
URL: https://github.com/apache/carbondata/pull/3612#discussion_r379997083 ########## File path: datamap/mv/core/src/main/scala/org/apache/carbondata/mv/extension/command/CreateMaterializedViewCommand.scala ########## @@ -0,0 +1,105 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.carbondata.mv.extension.command + +import scala.collection.JavaConverters._ +import scala.collection.mutable + +import org.apache.spark.sql._ +import org.apache.spark.sql.execution.command._ + +import org.apache.carbondata.common.exceptions.sql.MalformedMaterializedViewException +import org.apache.carbondata.common.logging.LogServiceFactory +import org.apache.carbondata.core.datamap.{DataMapProvider, DataMapStoreManager} +import org.apache.carbondata.core.datamap.status.DataMapStatusManager +import org.apache.carbondata.core.metadata.schema.datamap.{DataMapClassProvider, DataMapProperty} +import org.apache.carbondata.core.metadata.schema.table.DataMapSchema +import org.apache.carbondata.core.util.CarbonProperties +import org.apache.carbondata.datamap.DataMapManager +import org.apache.carbondata.events._ + +/** + * Create Materialized View Command implementation + * It will create the MV table, load the MV table (if deferred rebuild is false), + * and register the MV schema in [[DataMapStoreManager]] + */ +case class CreateMaterializedViewCommand( + mvName: String, + properties: Map[String, String], + queryString: Option[String], + ifNotExistsSet: Boolean = false, + deferredRebuild: Boolean = false) + extends AtomicRunnableCommand { + + private val LOGGER = LogServiceFactory.getLogService(this.getClass.getName) + private var dataMapProvider: DataMapProvider = _ + private var dataMapSchema: DataMapSchema = _ + + override def processMetadata(sparkSession: SparkSession): Seq[Row] = { + + setAuditInfo(Map("mvName" -> mvName) ++ properties) + + val mutableMap = mutable.Map[String, String](properties.toSeq: _*) + mutableMap.put(DataMapProperty.DEFERRED_REBUILD, deferredRebuild.toString) + + dataMapSchema = new DataMapSchema(mvName, DataMapClassProvider.MV.name()) + dataMapSchema.setProperties(mutableMap.asJava) + dataMapProvider = DataMapManager.get.getDataMapProvider(null, dataMapSchema, sparkSession) + if (DataMapStoreManager.getInstance().getAllDataMapSchemas.asScala + .exists(_.getDataMapName.equalsIgnoreCase(dataMapSchema.getDataMapName))) { + if (!ifNotExistsSet) { + throw new MalformedMaterializedViewException( + s"Materialized view with name ${dataMapSchema.getDataMapName} already exists") + } else { + return Seq.empty + } + } + + val systemFolderLocation: String = CarbonProperties.getInstance().getSystemFolderLocation + val operationContext: OperationContext = new OperationContext() + val preExecEvent = CreateDataMapPreExecutionEvent(sparkSession, systemFolderLocation, null) + OperationListenerBus.getInstance().fireEvent(preExecEvent, operationContext) + Review comment: Lazy datamaps has to be disabled after creation ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [hidden email] With regards, Apache Git Services |
|
|
In reply to this post by GitBox
Indhumathi27 commented on a change in pull request #3612: [CARBONDATA-3694] Separate Materialized View command from DataMap command
URL: https://github.com/apache/carbondata/pull/3612#discussion_r379997174 ########## File path: datamap/mv/core/src/main/scala/org/apache/carbondata/mv/extension/command/CreateMaterializedViewCommand.scala ########## @@ -0,0 +1,105 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.carbondata.mv.extension.command + +import scala.collection.JavaConverters._ +import scala.collection.mutable + +import org.apache.spark.sql._ +import org.apache.spark.sql.execution.command._ + +import org.apache.carbondata.common.exceptions.sql.MalformedMaterializedViewException +import org.apache.carbondata.common.logging.LogServiceFactory +import org.apache.carbondata.core.datamap.{DataMapProvider, DataMapStoreManager} +import org.apache.carbondata.core.datamap.status.DataMapStatusManager +import org.apache.carbondata.core.metadata.schema.datamap.{DataMapClassProvider, DataMapProperty} +import org.apache.carbondata.core.metadata.schema.table.DataMapSchema +import org.apache.carbondata.core.util.CarbonProperties +import org.apache.carbondata.datamap.DataMapManager +import org.apache.carbondata.events._ + +/** + * Create Materialized View Command implementation + * It will create the MV table, load the MV table (if deferred rebuild is false), + * and register the MV schema in [[DataMapStoreManager]] + */ +case class CreateMaterializedViewCommand( + mvName: String, + properties: Map[String, String], + queryString: Option[String], + ifNotExistsSet: Boolean = false, + deferredRebuild: Boolean = false) + extends AtomicRunnableCommand { + + private val LOGGER = LogServiceFactory.getLogService(this.getClass.getName) + private var dataMapProvider: DataMapProvider = _ + private var dataMapSchema: DataMapSchema = _ + + override def processMetadata(sparkSession: SparkSession): Seq[Row] = { + + setAuditInfo(Map("mvName" -> mvName) ++ properties) + + val mutableMap = mutable.Map[String, String](properties.toSeq: _*) + mutableMap.put(DataMapProperty.DEFERRED_REBUILD, deferredRebuild.toString) + + dataMapSchema = new DataMapSchema(mvName, DataMapClassProvider.MV.name()) + dataMapSchema.setProperties(mutableMap.asJava) + dataMapProvider = DataMapManager.get.getDataMapProvider(null, dataMapSchema, sparkSession) + if (DataMapStoreManager.getInstance().getAllDataMapSchemas.asScala + .exists(_.getDataMapName.equalsIgnoreCase(dataMapSchema.getDataMapName))) { + if (!ifNotExistsSet) { + throw new MalformedMaterializedViewException( + s"Materialized view with name ${dataMapSchema.getDataMapName} already exists") + } else { + return Seq.empty + } + } + + val systemFolderLocation: String = CarbonProperties.getInstance().getSystemFolderLocation + val operationContext: OperationContext = new OperationContext() + val preExecEvent = CreateDataMapPreExecutionEvent(sparkSession, systemFolderLocation, null) + OperationListenerBus.getInstance().fireEvent(preExecEvent, operationContext) + + dataMapProvider.initMeta(queryString.orNull) Review comment: Lazy datamaps has to be disabled after creation ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [hidden email] With regards, Apache Git Services |
|
|
In reply to this post by GitBox
Indhumathi27 commented on a change in pull request #3612: [CARBONDATA-3694] Separate Materialized View command from DataMap command
URL: https://github.com/apache/carbondata/pull/3612#discussion_r379999285 ########## File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/datamap/CarbonCreateDataMapCommand.scala ########## @@ -106,7 +106,8 @@ case class CarbonCreateDataMapCommand( .exists(_.getDataMapName.equalsIgnoreCase(dataMapSchema.getDataMapName))) { if (!ifNotExistsSet) { throw new MalformedDataMapCommandException( - "DataMap with name " + dataMapSchema.getDataMapName + " already exists in storage") + "Materialized view with name " + dataMapSchema.getDataMapName + Review comment: Since you have added a command to create mv, is this check in createdatamap is required? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [hidden email] With regards, Apache Git Services |
|
|
In reply to this post by GitBox
Indhumathi27 commented on a change in pull request #3612: [CARBONDATA-3694] Separate Materialized View command from DataMap command
URL: https://github.com/apache/carbondata/pull/3612#discussion_r379999974 ########## File path: docs/datamap/mv-datamap-guide.md ########## @@ -105,7 +105,7 @@ EXPLAIN SELECT a, sum(b) from maintable group by a; property is inherited from parent table, which allows user to provide different tableproperties for child table * MV creation with limit or union all ctas queries is unsupported - * MV datamap does not support Streaming + * MV does not support Streaming Review comment: I think complete document has to be changed. Using mv instead of datamap ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [hidden email] With regards, Apache Git Services |
|
|
In reply to this post by GitBox
jackylk commented on a change in pull request #3612: [CARBONDATA-3694] Separate Materialized View command from DataMap command
URL: https://github.com/apache/carbondata/pull/3612#discussion_r380000879 ########## File path: docs/datamap/mv-datamap-guide.md ########## @@ -105,7 +105,7 @@ EXPLAIN SELECT a, sum(b) from maintable group by a; property is inherited from parent table, which allows user to provide different tableproperties for child table * MV creation with limit or union all ctas queries is unsupported - * MV datamap does not support Streaming + * MV does not support Streaming Review comment: Yes, I will change doc later in another PR ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [hidden email] With regards, Apache Git Services |
|
|
In reply to this post by GitBox
jackylk commented on a change in pull request #3612: [CARBONDATA-3694] Separate Materialized View command from DataMap command
URL: https://github.com/apache/carbondata/pull/3612#discussion_r380002484 ########## File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/datamap/CarbonCreateDataMapCommand.scala ########## @@ -106,7 +106,8 @@ case class CarbonCreateDataMapCommand( .exists(_.getDataMapName.equalsIgnoreCase(dataMapSchema.getDataMapName))) { if (!ifNotExistsSet) { throw new MalformedDataMapCommandException( - "DataMap with name " + dataMapSchema.getDataMapName + " already exists in storage") + "Materialized view with name " + dataMapSchema.getDataMapName + Review comment: I will revert this change. This PR only modifies MV related command, DataMap and index related command modification will be done in #3614 ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [hidden email] With regards, Apache Git Services |
|
|
In reply to this post by GitBox
jackylk commented on a change in pull request #3612: [CARBONDATA-3694] Separate Materialized View command from DataMap command
URL: https://github.com/apache/carbondata/pull/3612#discussion_r380002851 ########## File path: datamap/mv/core/src/main/scala/org/apache/carbondata/mv/extension/command/CreateMaterializedViewCommand.scala ########## @@ -0,0 +1,105 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.carbondata.mv.extension.command + +import scala.collection.JavaConverters._ +import scala.collection.mutable + +import org.apache.spark.sql._ +import org.apache.spark.sql.execution.command._ + +import org.apache.carbondata.common.exceptions.sql.MalformedMaterializedViewException +import org.apache.carbondata.common.logging.LogServiceFactory +import org.apache.carbondata.core.datamap.{DataMapProvider, DataMapStoreManager} +import org.apache.carbondata.core.datamap.status.DataMapStatusManager +import org.apache.carbondata.core.metadata.schema.datamap.{DataMapClassProvider, DataMapProperty} +import org.apache.carbondata.core.metadata.schema.table.DataMapSchema +import org.apache.carbondata.core.util.CarbonProperties +import org.apache.carbondata.datamap.DataMapManager +import org.apache.carbondata.events._ + +/** + * Create Materialized View Command implementation + * It will create the MV table, load the MV table (if deferred rebuild is false), + * and register the MV schema in [[DataMapStoreManager]] + */ +case class CreateMaterializedViewCommand( + mvName: String, + properties: Map[String, String], + queryString: Option[String], + ifNotExistsSet: Boolean = false, + deferredRebuild: Boolean = false) + extends AtomicRunnableCommand { + + private val LOGGER = LogServiceFactory.getLogService(this.getClass.getName) + private var dataMapProvider: DataMapProvider = _ + private var dataMapSchema: DataMapSchema = _ + + override def processMetadata(sparkSession: SparkSession): Seq[Row] = { + + setAuditInfo(Map("mvName" -> mvName) ++ properties) + + val mutableMap = mutable.Map[String, String](properties.toSeq: _*) + mutableMap.put(DataMapProperty.DEFERRED_REBUILD, deferredRebuild.toString) + + dataMapSchema = new DataMapSchema(mvName, DataMapClassProvider.MV.name()) + dataMapSchema.setProperties(mutableMap.asJava) + dataMapProvider = DataMapManager.get.getDataMapProvider(null, dataMapSchema, sparkSession) + if (DataMapStoreManager.getInstance().getAllDataMapSchemas.asScala + .exists(_.getDataMapName.equalsIgnoreCase(dataMapSchema.getDataMapName))) { + if (!ifNotExistsSet) { + throw new MalformedMaterializedViewException( + s"Materialized view with name ${dataMapSchema.getDataMapName} already exists") + } else { + return Seq.empty + } + } + + val systemFolderLocation: String = CarbonProperties.getInstance().getSystemFolderLocation + val operationContext: OperationContext = new OperationContext() + val preExecEvent = CreateDataMapPreExecutionEvent(sparkSession, systemFolderLocation, null) + OperationListenerBus.getInstance().fireEvent(preExecEvent, operationContext) + + dataMapProvider.initMeta(queryString.orNull) Review comment: It is disabled inside initMeta ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [hidden email] With regards, Apache Git Services |
|
|
In reply to this post by GitBox
CarbonDataQA1 commented on issue #3612: [CARBONDATA-3694] Separate Materialized View command from DataMap command
URL: https://github.com/apache/carbondata/pull/3612#issuecomment-586843876 Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/313/ ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [hidden email] With regards, Apache Git Services |
|
|
In reply to this post by GitBox
CarbonDataQA1 commented on issue #3612: [CARBONDATA-3694] Separate Materialized View command from DataMap command
URL: https://github.com/apache/carbondata/pull/3612#issuecomment-586868248 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2016/ ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [hidden email] With regards, Apache Git Services |
|
|
In reply to this post by GitBox
akashrn5 commented on issue #3612: [CARBONDATA-3694] Separate Materialized View command from DataMap command
URL: https://github.com/apache/carbondata/pull/3612#issuecomment-587286207 retest this please ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [hidden email] With regards, Apache Git Services |
|
|
In reply to this post by GitBox
CarbonDataQA1 commented on issue #3612: [CARBONDATA-3694] Separate Materialized View command from DataMap command
URL: https://github.com/apache/carbondata/pull/3612#issuecomment-587290706 Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/329/ ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [hidden email] With regards, Apache Git Services |
|
|
In reply to this post by GitBox
CarbonDataQA1 commented on issue #3612: [CARBONDATA-3694] Separate Materialized View command from DataMap command
URL: https://github.com/apache/carbondata/pull/3612#issuecomment-587315169 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2031/ ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [hidden email] With regards, Apache Git Services |
|
|
In reply to this post by GitBox
akashrn5 commented on issue #3612: [CARBONDATA-3694] Separate Materialized View command from DataMap command
URL: https://github.com/apache/carbondata/pull/3612#issuecomment-587339708 LGTM ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [hidden email] With regards, Apache Git Services |
|
|
In reply to this post by GitBox
asfgit closed pull request #3612: [CARBONDATA-3694] Separate Materialized View command from DataMap command
URL: https://github.com/apache/carbondata/pull/3612 ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [hidden email] With regards, Apache Git Services |
«
Return to Apache CarbonData JIRA issues
|
1 view|%1 views
| Free forum by Nabble | Edit this page |