[GitHub] carbondata pull request #1416: [WIP] Adding event listener interface to Carb...

classic Classic list List threaded Threaded
32 messages Options
12
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata pull request #1416: [WIP] [CARBONDATA-1592] Adding event listener...

qiuchenjian-2
Github user gvramana commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1416#discussion_r145422709
 
    --- Diff: integration/spark2/src/main/scala/org/apache/carbondata/events/Events.scala ---
    @@ -0,0 +1,43 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.carbondata.events
    +
    +import org.apache.carbondata.core.metadata.CarbonTableIdentifier
    +import org.apache.carbondata.processing.loading.model.CarbonLoadModel
    +
    +/**
    + * event for database operations
    + */
    +trait DatabaseEvent extends Event {
    +  val databaseName: String
    +}
    +
    +/**
    + * event for table related operations
    + */
    +trait TableEvent extends DatabaseEvent {
    +  val carbonTableIdentifier: CarbonTableIdentifier
    +  override lazy val databaseName: String = carbonTableIdentifier.getDatabaseName
    +}
    +
    +/**
    + * event for load operations
    + */
    +trait LoadEvent extends TableEvent {
    --- End diff --
   
    Move this trait to LoadEvents class


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata pull request #1416: [WIP] [CARBONDATA-1592] Adding event listener...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user gvramana commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1416#discussion_r145424680
 
    --- Diff: integration/spark2/src/main/scala/org/apache/carbondata/events/EventListenerExample.scala ---
    @@ -0,0 +1,59 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.carbondata.events
    +
    +import java.io.File
    +
    +import org.apache.spark.sql.SparkSession
    +
    +import org.apache.carbondata.core.metadata.CarbonTableIdentifier
    +import org.apache.carbondata.processing.loading.model.CarbonLoadModel
    +
    +/**
    + * Example for event listener
    + */
    +object EventListenerExample extends App {
    +
    +  ListenerBus.getInstance()
    +    .addListener(LoadTablePreExecutionEvent.getClass.getName, new EventListener {
    +
    +      override def onEvent(event: Event): Unit = {
    +        println("Test event listener event")
    +      }
    +    })
    +
    +  import org.apache.spark.sql.CarbonSession._
    +
    +  val rootPath = new File(this.getClass.getResource("/").getPath
    +                          + "../../../..").getCanonicalPath
    +  val storeLocation = s"$rootPath/examples/spark2/target/store"
    +  val warehouse = s"$rootPath/examples/spark2/target/warehouse"
    +  val sparkSession = SparkSession
    +    .builder()
    +    .master("local")
    +    .appName("CarbonSessionExample")
    +    .config("spark.sql.warehouse.dir", warehouse)
    +    .config("spark.driver.host", "localhost")
    +    .getOrCreateCarbonSession(storeLocation)
    +  val carbonTableIdentifier = new CarbonTableIdentifier("db1", "tbl1", "tbl1-id")
    +  val carbonLoadModel = new CarbonLoadModel
    --- End diff --
   
    Example to be used how to subscribe to data load event


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #1416: [WIP] [CARBONDATA-1592] Adding event listener interf...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user jackylk commented on the issue:

    https://github.com/apache/carbondata/pull/1416
 
    Can you explain more on the intention of this PR?


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #1416: [WIP] [CARBONDATA-1592] Adding event listener interf...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1416
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1175/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #1416: [WIP] [CARBONDATA-1592] Adding event listener interf...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1416
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/544/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #1416: [WIP] [CARBONDATA-1592] Adding event listener interf...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user manishgupta88 commented on the issue:

    https://github.com/apache/carbondata/pull/1416
 
    @jackylk ..This PR will provide an event listener interface using which you can register the events based on the need of your operations. For example  before dropping a table you need to record the information from that table that can be used for clean up of resources/caches used by that table on executor or driver side.  In this case you can register an event PreDropEventListener which will perform the required task and add an event which will triggered just before drop.


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata pull request #1416: [WIP] [CARBONDATA-1592] Adding event listener...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user manishgupta88 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1416#discussion_r146236785
 
    --- Diff: integration/spark2/src/main/scala/org/apache/carbondata/events/Events.scala ---
    @@ -0,0 +1,43 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.carbondata.events
    +
    +import org.apache.carbondata.core.metadata.CarbonTableIdentifier
    +import org.apache.carbondata.processing.loading.model.CarbonLoadModel
    +
    +/**
    + * event for database operations
    + */
    +trait DatabaseEvent extends Event {
    +  val databaseName: String
    +}
    +
    +/**
    + * event for table related operations
    + */
    +trait TableEvent extends DatabaseEvent {
    +  val carbonTableIdentifier: CarbonTableIdentifier
    +  override lazy val databaseName: String = carbonTableIdentifier.getDatabaseName
    +}
    +
    +/**
    + * event for load operations
    + */
    +trait LoadEvent extends TableEvent {
    --- End diff --
   
    I think it is better to keep all the possible events in Events.scala class in order to know what all different type of events are possible and keep their concrete implementation in different classes to clearly define their behavior.


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #1416: [WIP] [CARBONDATA-1592] Adding event listener interf...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1416
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1284/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #1416: [WIP] [CARBONDATA-1592] Adding event listener interf...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1416
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/650/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #1416: [WIP] [CARBONDATA-1592] Adding event listener interf...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1416
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/854/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata pull request #1416: [WIP] [CARBONDATA-1592] Adding event listener...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user manishgupta88 closed the pull request at:

    https://github.com/apache/carbondata/pull/1416


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #1416: [WIP] [CARBONDATA-1592] Adding event listener interf...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user manishgupta88 commented on the issue:

    https://github.com/apache/carbondata/pull/1416
 
    Code already merged as part of PR #1473


---
12