[GitHub] carbondata pull request #1476: [CARBONDATA-1528][CARBONDATA-1527] [PreAgg] R...

classic Classic list List threaded Threaded
37 messages Options
12
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #1476: [CARBONDATA-1527] [PreAgg] Restrict update/delete fo...

qiuchenjian-2
Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1476
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1618/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #1476: [CARBONDATA-1527] [PreAgg] Restrict update/delete fo...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user jackylk commented on the issue:

    https://github.com/apache/carbondata/pull/1476
 
    Please refer to #1447, you can do it in similar way. Add a new strategy for pre-aggregate table and add checks there.


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #1476: [CARBONDATA-1527] [PreAgg] Restrict update/delete fo...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user kunal642 commented on the issue:

    https://github.com/apache/carbondata/pull/1476
 
    @jackylk We have an Event architecture which can be used to perform pre and post actions. I think it is better to add the validations in pre listeners.
   
    Another point is that if we add validation in a strategy like in #1447 then we would have to look up the carbon table twice(once for validation and other in the actual flow) which would decrease the performance. In listener interface, we can be shared the carbon table among various listeners.


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #1476: [CARBONDATA-1527] [CARBONDATA-1528] [PreAgg] Restric...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1476
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1051/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata pull request #1476: [CARBONDATA-1527] [CARBONDATA-1528] [PreAgg] ...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1476#discussion_r150492366
 
    --- Diff: integration/spark-common/src/main/scala/org/apache/spark/sql/execution/command/carbonTableSchemaCommon.scala ---
    @@ -428,7 +428,7 @@ class TableNewProcessor(cm: TableModel) {
         columnSchema.setSortColumn(false)
         if(isParentColumnRelation) {
           val dataMapField = map.get.get(field).get
    -      columnSchema.setAggFunction(dataMapField.aggregateFunction)
    +      columnSchema.setAggFunction(dataMapField.aggregateFunction);
    --- End diff --
   
    don't add semi colon


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata pull request #1476: [CARBONDATA-1527] [CARBONDATA-1528] [PreAgg] ...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1476#discussion_r150492650
 
    --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/preaaggregate/CreatePreAggregateTableCommand.scala ---
    @@ -24,7 +24,7 @@ import org.apache.carbondata.common.logging.LogServiceFactory
     import org.apache.carbondata.core.constants.CarbonCommonConstants
     import org.apache.carbondata.core.exception.InvalidConfigurationException
     import org.apache.carbondata.core.metadata.AbsoluteTableIdentifier
    -import org.apache.carbondata.core.metadata.schema.table.TableInfo
    +import org.apache.carbondata.core.metadata.schema.table.{RelationIdentifier, TableInfo}
    --- End diff --
   
    Please don't change the class unnecessarily


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata pull request #1476: [CARBONDATA-1527] [CARBONDATA-1528] [PreAgg] ...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1476#discussion_r150493082
 
    --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/preaaggregate/PreAggregateListeners.scala ---
    @@ -79,3 +79,190 @@ object LoadPostAggregateListener extends OperationEventListener {
         }
       }
     }
    +
    +object PreAggregateDataTypeChangePreListener extends OperationEventListener {
    +  /**
    +   * Called on a specified event occurrence
    +   *
    +   * @param event
    +   * @param operationContext
    +   */
    +  override def onEvent(event: Event, operationContext: OperationContext): Unit = {
    +    val dataTypeChangePreListener = event.asInstanceOf[AlterTableDataTypeChangePreEvent]
    +    val carbonTable = dataTypeChangePreListener.carbonTable
    +    val alterTableDataTypeChangeModel = dataTypeChangePreListener.alterTableDataTypeChangeModel
    +    val columnToBeAltered: String = alterTableDataTypeChangeModel.columnName
    +    val dataMapSchemas = carbonTable.getTableInfo.getDataMapSchemaList
    +    if (dataMapSchemas != null && !dataMapSchemas.isEmpty) {
    +      dataMapSchemas.asScala.foreach {
    --- End diff --
   
    move to above line


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata pull request #1476: [CARBONDATA-1527] [CARBONDATA-1528] [PreAgg] ...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1476#discussion_r150493129
 
    --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/preaaggregate/PreAggregateListeners.scala ---
    @@ -79,3 +79,190 @@ object LoadPostAggregateListener extends OperationEventListener {
         }
       }
     }
    +
    +object PreAggregateDataTypeChangePreListener extends OperationEventListener {
    +  /**
    +   * Called on a specified event occurrence
    +   *
    +   * @param event
    +   * @param operationContext
    +   */
    +  override def onEvent(event: Event, operationContext: OperationContext): Unit = {
    +    val dataTypeChangePreListener = event.asInstanceOf[AlterTableDataTypeChangePreEvent]
    +    val carbonTable = dataTypeChangePreListener.carbonTable
    +    val alterTableDataTypeChangeModel = dataTypeChangePreListener.alterTableDataTypeChangeModel
    +    val columnToBeAltered: String = alterTableDataTypeChangeModel.columnName
    +    val dataMapSchemas = carbonTable.getTableInfo.getDataMapSchemaList
    +    if (dataMapSchemas != null && !dataMapSchemas.isEmpty) {
    +      dataMapSchemas.asScala.foreach {
    +        dataMapSchema =>
    --- End diff --
   
    move to above line


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata pull request #1476: [CARBONDATA-1527] [CARBONDATA-1528] [PreAgg] ...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1476#discussion_r150493381
 
    --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/preaaggregate/PreAggregateListeners.scala ---
    @@ -79,3 +79,190 @@ object LoadPostAggregateListener extends OperationEventListener {
         }
       }
     }
    +
    +object PreAggregateDataTypeChangePreListener extends OperationEventListener {
    +  /**
    +   * Called on a specified event occurrence
    +   *
    +   * @param event
    +   * @param operationContext
    +   */
    +  override def onEvent(event: Event, operationContext: OperationContext): Unit = {
    +    val dataTypeChangePreListener = event.asInstanceOf[AlterTableDataTypeChangePreEvent]
    +    val carbonTable = dataTypeChangePreListener.carbonTable
    +    val alterTableDataTypeChangeModel = dataTypeChangePreListener.alterTableDataTypeChangeModel
    +    val columnToBeAltered: String = alterTableDataTypeChangeModel.columnName
    +    val dataMapSchemas = carbonTable.getTableInfo.getDataMapSchemaList
    +    if (dataMapSchemas != null && !dataMapSchemas.isEmpty) {
    +      dataMapSchemas.asScala.foreach {
    +        dataMapSchema =>
    +          val childColumns = dataMapSchema.getChildSchema.getListOfColumns
    +          if (childColumns.asScala.exists(_.getColumnName == columnToBeAltered)) {
    --- End diff --
   
    I think it is better use equalsIgnorecase


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata pull request #1476: [CARBONDATA-1527] [CARBONDATA-1528] [PreAgg] ...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1476#discussion_r150493685
 
    --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/preaaggregate/PreAggregateListeners.scala ---
    @@ -79,3 +79,190 @@ object LoadPostAggregateListener extends OperationEventListener {
         }
       }
     }
    +
    +object PreAggregateDataTypeChangePreListener extends OperationEventListener {
    +  /**
    +   * Called on a specified event occurrence
    +   *
    +   * @param event
    +   * @param operationContext
    +   */
    +  override def onEvent(event: Event, operationContext: OperationContext): Unit = {
    +    val dataTypeChangePreListener = event.asInstanceOf[AlterTableDataTypeChangePreEvent]
    +    val carbonTable = dataTypeChangePreListener.carbonTable
    +    val alterTableDataTypeChangeModel = dataTypeChangePreListener.alterTableDataTypeChangeModel
    +    val columnToBeAltered: String = alterTableDataTypeChangeModel.columnName
    +    val dataMapSchemas = carbonTable.getTableInfo.getDataMapSchemaList
    +    if (dataMapSchemas != null && !dataMapSchemas.isEmpty) {
    +      dataMapSchemas.asScala.foreach {
    +        dataMapSchema =>
    +          val childColumns = dataMapSchema.getChildSchema.getListOfColumns
    +          if (childColumns.asScala.exists(_.getColumnName == columnToBeAltered)) {
    +            throw new UnsupportedOperationException(s"Column $columnToBeAltered exists in a " +
    +                                                    s"pre-aggregate table ${
    +                                                      dataMapSchema.toString}. Cannot change datatype")
    +          }
    +      }
    +
    +      if (carbonTable.isPreAggregateTable) {
    +        throw new UnsupportedOperationException(s"Cannot change data type for columns in " +
    +                                                s"pre-aggreagate table ${
    +                                                  carbonTable.getDatabaseName}.${ carbonTable.getFactTableName }")
    +      }
    +    }
    +  }
    +}
    +
    +object PreAggregateDeleteSegmentByDatePreListener extends OperationEventListener {
    +  /**
    +   * Called on a specified event occurrence
    +   *
    +   * @param event
    +   * @param operationContext
    +   */
    +  override def onEvent(event: Event, operationContext: OperationContext): Unit = {
    +    val deleteSegmentByDatePreEvent = event.asInstanceOf[DeleteSegmentByDatePreEvent]
    +    val carbonTable = deleteSegmentByDatePreEvent.carbonTable
    +    if (carbonTable != null) {
    +      if (carbonTable.hasPreAggregateTables) {
    +        throw new UnsupportedOperationException(
    +          "Delete segment operation is not supported on tables which have a pre-aggregate table. " +
    +          "Drop pre-aggregation table to continue")
    +      }
    +      if (carbonTable.isPreAggregateTable) {
    +        throw new UnsupportedOperationException(
    +          "Delete segment operation is not supported on pre-aggregate table")
    +      }
    +    }
    +  }
    +}
    +
    +object PreAggregateDeleteSegmentByIdPreListener extends OperationEventListener {
    +  /**
    +   * Called on a specified event occurrence
    +   *
    +   * @param event
    +   * @param operationContext
    +   */
    +  override def onEvent(event: Event, operationContext: OperationContext): Unit = {
    +    val tableEvent = event.asInstanceOf[DeleteSegmentByIdPreEvent]
    +    val carbonTable = tableEvent.carbonTable
    +    if (carbonTable != null) {
    +      if (carbonTable.hasPreAggregateTables) {
    +        throw new UnsupportedOperationException(
    +          "Delete segment operation is not supported on tables which have a pre-aggregate table")
    +      }
    +      if (carbonTable.isPreAggregateTable) {
    +        throw new UnsupportedOperationException(
    +          "Delete segment operation is not supported on pre-aggregate table")
    +      }
    +    }
    +  }
    +
    +}
    +
    +object PreAggregateDropColumnPreListener extends OperationEventListener {
    +  /**
    +   * Called on a specified event occurrence
    +   *
    +   * @param event
    +   * @param operationContext
    +   */
    +  override def onEvent(event: Event, operationContext: OperationContext): Unit = {
    +    val dataTypeChangePreListener = event.asInstanceOf[AlterTableDropColumnPreEvent]
    +    val carbonTable = dataTypeChangePreListener.carbonTable
    +    val alterTableDropColumnModel = dataTypeChangePreListener.alterTableDropColumnModel
    +    val columnsToBeDropped = alterTableDropColumnModel.columns
    +    val dataMapSchemas = carbonTable.getTableInfo.getDataMapSchemaList
    +    if (dataMapSchemas != null && !dataMapSchemas.isEmpty) {
    +      dataMapSchemas.asScala.foreach {
    +        dataMapSchema =>
    --- End diff --
   
    move to above line


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata pull request #1476: [CARBONDATA-1527] [CARBONDATA-1528] [PreAgg] ...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1476#discussion_r150494490
 
    --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonSessionState.scala ---
    @@ -26,7 +26,11 @@ import org.apache.spark.sql.catalyst.optimizer.Optimizer
     import org.apache.spark.sql.catalyst.parser.ParserInterface
     import org.apache.spark.sql.catalyst.plans.logical.{Filter, LogicalPlan, SubqueryAlias}
     import org.apache.spark.sql.execution.SparkOptimizer
    -import org.apache.spark.sql.execution.command.preaaggregate.{DropPreAggregateTablePostListener, LoadPostAggregateListener}
    +import org.apache.spark.sql.execution.command.preaaggregate._
    +<<<<<<< HEAD
    +=======
    +import org.apache.spark.sql.execution.command.preaaggregate.listeners._
    +>>>>>>> a49f44c... added support for alter operations
    --- End diff --
   
    Seems something wrong with merge


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #1476: [CARBONDATA-1527] [CARBONDATA-1528] [PreAgg] Restric...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1476
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1054/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #1476: [CARBONDATA-1527] [CARBONDATA-1528] [PreAgg] Restric...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1476
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1056/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #1476: [CARBONDATA-1527] [CARBONDATA-1528] [PreAgg] Restric...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1476
 
    LGTM


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #1476: [CARBONDATA-1527] [CARBONDATA-1528] [PreAgg] Restric...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1476
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1670/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #1476: [CARBONDATA-1527] [CARBONDATA-1528] [PreAgg] Restric...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1476
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1060/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata pull request #1476: [CARBONDATA-1527] [CARBONDATA-1528] [PreAgg] ...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user asfgit closed the pull request at:

    https://github.com/apache/carbondata/pull/1476


---
12