[GitHub] carbondata pull request #1490: [CARBONDATA-1579][PREAGG][DATAMAP] Support Da...

[qiuchenjian-2]

GitHub user ravipesala opened a pull request:

    https://github.com/apache/carbondata/pull/1490

    [CARBONDATA-1579][PREAGG][DATAMAP] Support DataMap show

    Be sure to do all of the following checklist to help us incorporate
    your contribution quickly and easily:
      Added support for show datamap on table.
      ```
       SHOW DATAMAP ON TABLE test
      ```
      The above command shows all datamaps on the table name test.
     Output of it shows the following attributes
      ```
      +-----------+---------+----------------+
    |DataMapName|ClassName|Associated Table|
    +-----------+---------+----------------+
      ```
   
    This PR is dependent on the PRs https://github.com/apache/carbondata/pull/1481  and https://github.com/apache/carbondata/pull/1489 . First those PR need to be merged
     - [X] Any interfaces changed?
       NO
     - [X] Any backward compatibility impacted?
       NO
     - [X] Document update required?
       Yes, sql command needs to be added to document
     - [X] Testing done
           Tests are added        
     - [X] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
   


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/ravipesala/incubator-carbondata datamap-show

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/1490.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1490
   
----
commit cac54d9a1578796cf1d90abacab069a0413a49a0
Author: ravipesala <[hidden email]>
Date:   2017-11-12T11:38:09Z

    Added create datamap parser and saved to schema file

commit aebcdc01ce66ed79bac17db61fd62a325622e9c4
Author: ravipesala <[hidden email]>
Date:   2017-11-12T06:57:47Z

    Added parser for drop datamap and handled events

commit dd1ee26742f8c8f8731260ac623173c5309cb153
Author: ravipesala <[hidden email]>
Date:   2017-11-12T17:26:17Z

    Added parser for Show Datamap command

----


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1490
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1033/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1490
 
    retest this please


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1490
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1035/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1490
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1651/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1490
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1653/



---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1490#discussion_r150506530
 
    --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/datamap/CarbonDataMapShowCommand.scala ---
    @@ -0,0 +1,67 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.sql.execution.command.datamap
    +
    +import scala.collection.JavaConverters._
    +
    +import org.apache.spark.sql.{CarbonEnv, Row, SparkSession}
    +import org.apache.spark.sql.catalyst.expressions.{Attribute, AttributeReference}
    +import org.apache.spark.sql.execution.command.{Checker, DataProcessCommand, RunnableCommand}
    +import org.apache.spark.sql.hive.CarbonRelation
    +import org.apache.spark.sql.types.StringType
    +
    +/**
    + * Show the datamaps on the table
    + * @param databaseNameOp
    + * @param tableName
    + */
    +case class CarbonDataMapShowCommand(
    +    databaseNameOp: Option[String],
    +    tableName: String)
    +  extends RunnableCommand with DataProcessCommand {
    +
    +  override def output: Seq[Attribute] = {
    +    Seq(AttributeReference("DataMapName", StringType, nullable = false)(),
    +      AttributeReference("ClassName", StringType, nullable = false)(),
    +      AttributeReference("Associated Table", StringType, nullable = false)())
    +  }
    +
    +  override def run(sparkSession: SparkSession): Seq[Row] = {
    +    processData(sparkSession)
    +  }
    +
    +  override def processData(sparkSession: SparkSession): Seq[Row] = {
    +    Checker.validateTableExists(databaseNameOp, tableName, sparkSession)
    +    val carbonTable = CarbonEnv.getInstance(sparkSession).carbonMetastore.
    +      lookupRelation(databaseNameOp, tableName)(sparkSession).asInstanceOf[CarbonRelation].
    +      tableMeta.carbonTable
    +    val schemaList = carbonTable.getTableInfo.getDataMapSchemaList
    +    if (schemaList != null && schemaList.size() > 0) {
    +      schemaList.asScala.map { s =>
    +        var table = "-NA-"
    --- End diff --
   
    I think `(NA)` is better


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1490#discussion_r150506799
 
    --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/parser/CarbonSpark2SqlParser.scala ---
    @@ -142,6 +142,12 @@ class CarbonSpark2SqlParser extends CarbonDDLSqlParser {
             CarbonDropDataMapCommand(dmname, ifexists.isDefined, dbName, tableName)
         }
     
    +  protected lazy val showDataMap: Parser[LogicalPlan] =
    --- End diff --
   
    please add syntax description


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1490#discussion_r150507280
 
    --- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/datamap/TestDataMapCommand.scala ---
    @@ -81,8 +82,25 @@ class TestDataMapCommand extends QueryTest with BeforeAndAfterAll {
         assert(dataMapSchemaList.size() == 3)
       }
     
    +  test("test show datamap without preaggregate") {
    +    sql("drop table if exists datamapshowtest")
    +    sql("create table datamapshowtest (a string, b string, c string) stored by 'carbondata'")
    +    sql("create datamap datamap1 on table datamapshowtest using 'new.class' dmproperties('key'='value')")
    +    sql("create datamap datamap2 on table datamapshowtest using 'new.class' dmproperties('key'='value')")
    +    checkExistence(sql("show datamap on table datamapshowtest"), true, "datamap1", "datamap2", "-NA-", "new.class")
    --- End diff --
   
    In output string, it should show `datamapshowtest` instead of `-NA-` for the table name, right?


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1490#discussion_r150507602
 
    --- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/datamap/TestDataMapCommand.scala ---
    @@ -81,8 +82,25 @@ class TestDataMapCommand extends QueryTest with BeforeAndAfterAll {
         assert(dataMapSchemaList.size() == 3)
       }
     
    +  test("test show datamap without preaggregate") {
    +    sql("drop table if exists datamapshowtest")
    +    sql("create table datamapshowtest (a string, b string, c string) stored by 'carbondata'")
    +    sql("create datamap datamap1 on table datamapshowtest using 'new.class' dmproperties('key'='value')")
    +    sql("create datamap datamap2 on table datamapshowtest using 'new.class' dmproperties('key'='value')")
    +    checkExistence(sql("show datamap on table datamapshowtest"), true, "datamap1", "datamap2", "-NA-", "new.class")
    +  }
    +
    +  test("test show datamap with preaggregate") {
    +    sql("drop table if exists datamapshowtest")
    +    sql("create table datamapshowtest (a string, b string, c string) stored by 'carbondata'")
    +    sql("create datamap datamap1 on table datamapshowtest using 'preaggregate' as select count(a) from datamapshowtest")
    +    sql("create datamap datamap2 on table datamapshowtest using 'new.class' dmproperties('key'='value')")
    +    checkExistence(sql("show datamap on table datamapshowtest"), true, "datamap1", "datamap2", "-NA-", "new.class", "default.datamap1")
    +  }
    +
    --- End diff --
   
    please add a testcase to show datamap after dropping the datamap


---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1490#discussion_r150679772
 
    --- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/datamap/TestDataMapCommand.scala ---
    @@ -81,8 +82,25 @@ class TestDataMapCommand extends QueryTest with BeforeAndAfterAll {
         assert(dataMapSchemaList.size() == 3)
       }
     
    +  test("test show datamap without preaggregate") {
    +    sql("drop table if exists datamapshowtest")
    +    sql("create table datamapshowtest (a string, b string, c string) stored by 'carbondata'")
    +    sql("create datamap datamap1 on table datamapshowtest using 'new.class' dmproperties('key'='value')")
    +    sql("create datamap datamap2 on table datamapshowtest using 'new.class' dmproperties('key'='value')")
    +    checkExistence(sql("show datamap on table datamapshowtest"), true, "datamap1", "datamap2", "-NA-", "new.class")
    --- End diff --
   
    No, it is NA only. The associated table available only for preaggregate datamaps


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1490
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1110/



---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1490#discussion_r150808687
 
    --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/datamap/CarbonDataMapShowCommand.scala ---
    @@ -0,0 +1,67 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.sql.execution.command.datamap
    +
    +import scala.collection.JavaConverters._
    +
    +import org.apache.spark.sql.{CarbonEnv, Row, SparkSession}
    +import org.apache.spark.sql.catalyst.expressions.{Attribute, AttributeReference}
    +import org.apache.spark.sql.execution.command.{Checker, DataProcessCommand, RunnableCommand}
    +import org.apache.spark.sql.hive.CarbonRelation
    +import org.apache.spark.sql.types.StringType
    +
    +/**
    + * Show the datamaps on the table
    + * @param databaseNameOp
    + * @param tableName
    + */
    +case class CarbonDataMapShowCommand(
    +    databaseNameOp: Option[String],
    +    tableName: String)
    +  extends RunnableCommand with DataProcessCommand {
    +
    +  override def output: Seq[Attribute] = {
    +    Seq(AttributeReference("DataMapName", StringType, nullable = false)(),
    +      AttributeReference("ClassName", StringType, nullable = false)(),
    +      AttributeReference("Associated Table", StringType, nullable = false)())
    +  }
    +
    +  override def run(sparkSession: SparkSession): Seq[Row] = {
    +    processData(sparkSession)
    +  }
    +
    +  override def processData(sparkSession: SparkSession): Seq[Row] = {
    +    Checker.validateTableExists(databaseNameOp, tableName, sparkSession)
    +    val carbonTable = CarbonEnv.getInstance(sparkSession).carbonMetastore.
    +      lookupRelation(databaseNameOp, tableName)(sparkSession).asInstanceOf[CarbonRelation].
    +      tableMeta.carbonTable
    +    val schemaList = carbonTable.getTableInfo.getDataMapSchemaList
    +    if (schemaList != null && schemaList.size() > 0) {
    +      schemaList.asScala.map { s =>
    +        var table = "-NA-"
    --- End diff --
   
    ok


---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1490#discussion_r150809947
 
    --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/parser/CarbonSpark2SqlParser.scala ---
    @@ -142,6 +142,12 @@ class CarbonSpark2SqlParser extends CarbonDDLSqlParser {
             CarbonDropDataMapCommand(dmname, ifexists.isDefined, dbName, tableName)
         }
     
    +  protected lazy val showDataMap: Parser[LogicalPlan] =
    --- End diff --
   
    ok


---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1490#discussion_r150812888
 
    --- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/datamap/TestDataMapCommand.scala ---
    @@ -81,8 +82,25 @@ class TestDataMapCommand extends QueryTest with BeforeAndAfterAll {
         assert(dataMapSchemaList.size() == 3)
       }
     
    +  test("test show datamap without preaggregate") {
    +    sql("drop table if exists datamapshowtest")
    +    sql("create table datamapshowtest (a string, b string, c string) stored by 'carbondata'")
    +    sql("create datamap datamap1 on table datamapshowtest using 'new.class' dmproperties('key'='value')")
    +    sql("create datamap datamap2 on table datamapshowtest using 'new.class' dmproperties('key'='value')")
    +    checkExistence(sql("show datamap on table datamapshowtest"), true, "datamap1", "datamap2", "-NA-", "new.class")
    +  }
    +
    +  test("test show datamap with preaggregate") {
    +    sql("drop table if exists datamapshowtest")
    +    sql("create table datamapshowtest (a string, b string, c string) stored by 'carbondata'")
    +    sql("create datamap datamap1 on table datamapshowtest using 'preaggregate' as select count(a) from datamapshowtest")
    +    sql("create datamap datamap2 on table datamapshowtest using 'new.class' dmproperties('key'='value')")
    +    checkExistence(sql("show datamap on table datamapshowtest"), true, "datamap1", "datamap2", "-NA-", "new.class", "default.datamap1")
    +  }
    +
    --- End diff --
   
    added


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1490
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1113/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1490
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1709/



---

[qiuchenjian-2]

Github user jackylk commented on the issue:

    https://github.com/apache/carbondata/pull/1490
 
    retest this please


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1490
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1712/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1490
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1120/



---