Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1476 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1618/ --- |
In reply to this post by qiuchenjian-2
Github user jackylk commented on the issue:
https://github.com/apache/carbondata/pull/1476 Please refer to #1447, you can do it in similar way. Add a new strategy for pre-aggregate table and add checks there. --- |
In reply to this post by qiuchenjian-2
Github user kunal642 commented on the issue:
https://github.com/apache/carbondata/pull/1476 @jackylk We have an Event architecture which can be used to perform pre and post actions. I think it is better to add the validations in pre listeners. Another point is that if we add validation in a strategy like in #1447 then we would have to look up the carbon table twice(once for validation and other in the actual flow) which would decrease the performance. In listener interface, we can be shared the carbon table among various listeners. --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1476 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1051/ --- |
In reply to this post by qiuchenjian-2
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1476#discussion_r150492366 --- Diff: integration/spark-common/src/main/scala/org/apache/spark/sql/execution/command/carbonTableSchemaCommon.scala --- @@ -428,7 +428,7 @@ class TableNewProcessor(cm: TableModel) { columnSchema.setSortColumn(false) if(isParentColumnRelation) { val dataMapField = map.get.get(field).get - columnSchema.setAggFunction(dataMapField.aggregateFunction) + columnSchema.setAggFunction(dataMapField.aggregateFunction); --- End diff -- don't add semi colon --- |
In reply to this post by qiuchenjian-2
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1476#discussion_r150492650 --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/preaaggregate/CreatePreAggregateTableCommand.scala --- @@ -24,7 +24,7 @@ import org.apache.carbondata.common.logging.LogServiceFactory import org.apache.carbondata.core.constants.CarbonCommonConstants import org.apache.carbondata.core.exception.InvalidConfigurationException import org.apache.carbondata.core.metadata.AbsoluteTableIdentifier -import org.apache.carbondata.core.metadata.schema.table.TableInfo +import org.apache.carbondata.core.metadata.schema.table.{RelationIdentifier, TableInfo} --- End diff -- Please don't change the class unnecessarily --- |
In reply to this post by qiuchenjian-2
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1476#discussion_r150493082 --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/preaaggregate/PreAggregateListeners.scala --- @@ -79,3 +79,190 @@ object LoadPostAggregateListener extends OperationEventListener { } } } + +object PreAggregateDataTypeChangePreListener extends OperationEventListener { + /** + * Called on a specified event occurrence + * + * @param event + * @param operationContext + */ + override def onEvent(event: Event, operationContext: OperationContext): Unit = { + val dataTypeChangePreListener = event.asInstanceOf[AlterTableDataTypeChangePreEvent] + val carbonTable = dataTypeChangePreListener.carbonTable + val alterTableDataTypeChangeModel = dataTypeChangePreListener.alterTableDataTypeChangeModel + val columnToBeAltered: String = alterTableDataTypeChangeModel.columnName + val dataMapSchemas = carbonTable.getTableInfo.getDataMapSchemaList + if (dataMapSchemas != null && !dataMapSchemas.isEmpty) { + dataMapSchemas.asScala.foreach { --- End diff -- move to above line --- |
In reply to this post by qiuchenjian-2
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1476#discussion_r150493129 --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/preaaggregate/PreAggregateListeners.scala --- @@ -79,3 +79,190 @@ object LoadPostAggregateListener extends OperationEventListener { } } } + +object PreAggregateDataTypeChangePreListener extends OperationEventListener { + /** + * Called on a specified event occurrence + * + * @param event + * @param operationContext + */ + override def onEvent(event: Event, operationContext: OperationContext): Unit = { + val dataTypeChangePreListener = event.asInstanceOf[AlterTableDataTypeChangePreEvent] + val carbonTable = dataTypeChangePreListener.carbonTable + val alterTableDataTypeChangeModel = dataTypeChangePreListener.alterTableDataTypeChangeModel + val columnToBeAltered: String = alterTableDataTypeChangeModel.columnName + val dataMapSchemas = carbonTable.getTableInfo.getDataMapSchemaList + if (dataMapSchemas != null && !dataMapSchemas.isEmpty) { + dataMapSchemas.asScala.foreach { + dataMapSchema => --- End diff -- move to above line --- |
In reply to this post by qiuchenjian-2
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1476#discussion_r150493381 --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/preaaggregate/PreAggregateListeners.scala --- @@ -79,3 +79,190 @@ object LoadPostAggregateListener extends OperationEventListener { } } } + +object PreAggregateDataTypeChangePreListener extends OperationEventListener { + /** + * Called on a specified event occurrence + * + * @param event + * @param operationContext + */ + override def onEvent(event: Event, operationContext: OperationContext): Unit = { + val dataTypeChangePreListener = event.asInstanceOf[AlterTableDataTypeChangePreEvent] + val carbonTable = dataTypeChangePreListener.carbonTable + val alterTableDataTypeChangeModel = dataTypeChangePreListener.alterTableDataTypeChangeModel + val columnToBeAltered: String = alterTableDataTypeChangeModel.columnName + val dataMapSchemas = carbonTable.getTableInfo.getDataMapSchemaList + if (dataMapSchemas != null && !dataMapSchemas.isEmpty) { + dataMapSchemas.asScala.foreach { + dataMapSchema => + val childColumns = dataMapSchema.getChildSchema.getListOfColumns + if (childColumns.asScala.exists(_.getColumnName == columnToBeAltered)) { --- End diff -- I think it is better use equalsIgnorecase --- |
In reply to this post by qiuchenjian-2
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1476#discussion_r150493685 --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/preaaggregate/PreAggregateListeners.scala --- @@ -79,3 +79,190 @@ object LoadPostAggregateListener extends OperationEventListener { } } } + +object PreAggregateDataTypeChangePreListener extends OperationEventListener { + /** + * Called on a specified event occurrence + * + * @param event + * @param operationContext + */ + override def onEvent(event: Event, operationContext: OperationContext): Unit = { + val dataTypeChangePreListener = event.asInstanceOf[AlterTableDataTypeChangePreEvent] + val carbonTable = dataTypeChangePreListener.carbonTable + val alterTableDataTypeChangeModel = dataTypeChangePreListener.alterTableDataTypeChangeModel + val columnToBeAltered: String = alterTableDataTypeChangeModel.columnName + val dataMapSchemas = carbonTable.getTableInfo.getDataMapSchemaList + if (dataMapSchemas != null && !dataMapSchemas.isEmpty) { + dataMapSchemas.asScala.foreach { + dataMapSchema => + val childColumns = dataMapSchema.getChildSchema.getListOfColumns + if (childColumns.asScala.exists(_.getColumnName == columnToBeAltered)) { + throw new UnsupportedOperationException(s"Column $columnToBeAltered exists in a " + + s"pre-aggregate table ${ + dataMapSchema.toString}. Cannot change datatype") + } + } + + if (carbonTable.isPreAggregateTable) { + throw new UnsupportedOperationException(s"Cannot change data type for columns in " + + s"pre-aggreagate table ${ + carbonTable.getDatabaseName}.${ carbonTable.getFactTableName }") + } + } + } +} + +object PreAggregateDeleteSegmentByDatePreListener extends OperationEventListener { + /** + * Called on a specified event occurrence + * + * @param event + * @param operationContext + */ + override def onEvent(event: Event, operationContext: OperationContext): Unit = { + val deleteSegmentByDatePreEvent = event.asInstanceOf[DeleteSegmentByDatePreEvent] + val carbonTable = deleteSegmentByDatePreEvent.carbonTable + if (carbonTable != null) { + if (carbonTable.hasPreAggregateTables) { + throw new UnsupportedOperationException( + "Delete segment operation is not supported on tables which have a pre-aggregate table. " + + "Drop pre-aggregation table to continue") + } + if (carbonTable.isPreAggregateTable) { + throw new UnsupportedOperationException( + "Delete segment operation is not supported on pre-aggregate table") + } + } + } +} + +object PreAggregateDeleteSegmentByIdPreListener extends OperationEventListener { + /** + * Called on a specified event occurrence + * + * @param event + * @param operationContext + */ + override def onEvent(event: Event, operationContext: OperationContext): Unit = { + val tableEvent = event.asInstanceOf[DeleteSegmentByIdPreEvent] + val carbonTable = tableEvent.carbonTable + if (carbonTable != null) { + if (carbonTable.hasPreAggregateTables) { + throw new UnsupportedOperationException( + "Delete segment operation is not supported on tables which have a pre-aggregate table") + } + if (carbonTable.isPreAggregateTable) { + throw new UnsupportedOperationException( + "Delete segment operation is not supported on pre-aggregate table") + } + } + } + +} + +object PreAggregateDropColumnPreListener extends OperationEventListener { + /** + * Called on a specified event occurrence + * + * @param event + * @param operationContext + */ + override def onEvent(event: Event, operationContext: OperationContext): Unit = { + val dataTypeChangePreListener = event.asInstanceOf[AlterTableDropColumnPreEvent] + val carbonTable = dataTypeChangePreListener.carbonTable + val alterTableDropColumnModel = dataTypeChangePreListener.alterTableDropColumnModel + val columnsToBeDropped = alterTableDropColumnModel.columns + val dataMapSchemas = carbonTable.getTableInfo.getDataMapSchemaList + if (dataMapSchemas != null && !dataMapSchemas.isEmpty) { + dataMapSchemas.asScala.foreach { + dataMapSchema => --- End diff -- move to above line --- |
In reply to this post by qiuchenjian-2
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1476#discussion_r150494490 --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonSessionState.scala --- @@ -26,7 +26,11 @@ import org.apache.spark.sql.catalyst.optimizer.Optimizer import org.apache.spark.sql.catalyst.parser.ParserInterface import org.apache.spark.sql.catalyst.plans.logical.{Filter, LogicalPlan, SubqueryAlias} import org.apache.spark.sql.execution.SparkOptimizer -import org.apache.spark.sql.execution.command.preaaggregate.{DropPreAggregateTablePostListener, LoadPostAggregateListener} +import org.apache.spark.sql.execution.command.preaaggregate._ +<<<<<<< HEAD +======= +import org.apache.spark.sql.execution.command.preaaggregate.listeners._ +>>>>>>> a49f44c... added support for alter operations --- End diff -- Seems something wrong with merge --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1476 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1054/ --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1476 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1056/ --- |
In reply to this post by qiuchenjian-2
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1476 LGTM --- |
In reply to this post by qiuchenjian-2
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1476 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1670/ --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1476 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1060/ --- |
In reply to this post by qiuchenjian-2
|
Free forum by Nabble | Edit this page |