[GitHub] carbondata pull request #1878: [CARBONDATA-2094] Filter DataMap Tables in Sh...

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1878
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2128/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1878
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3277/



---

[qiuchenjian-2]

Github user kumarvishal09 commented on the issue:

    https://github.com/apache/carbondata/pull/1878
 
    retest this please


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1878
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2176/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1878
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3413/



---

[qiuchenjian-2]

Github user kumarvishal09 commented on the issue:

    https://github.com/apache/carbondata/pull/1878
 
    LGTM


---

[qiuchenjian-2]

Github user BJangir closed the pull request at:

    https://github.com/apache/carbondata/pull/1878


---

[qiuchenjian-2]

Github user xubo245 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1878#discussion_r165567560
 
    --- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/preaggregate/TestPreAggregateDrop.scala ---
    @@ -84,7 +86,8 @@ class TestPreAggregateDrop extends QueryTest with BeforeAndAfterAll {
         sql("create datamap preagg_same1 on table maintable using 'preaggregate' as select" +
             " a,sum(c) from maintable group by a")
         showTables = sql("show tables")
    --- End diff --
   
    remove this line


---

[qiuchenjian-2]

Github user xubo245 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1878#discussion_r165807476
 
    --- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/preaggregate/TestPreAggCreateCommand.scala ---
    @@ -270,6 +284,28 @@ class TestPreAggCreateCommand extends QueryTest with BeforeAndAfterAll {
         sql("DROP DATAMAP IF EXISTS agg0 ON TABLE maintable")
       }
     
    +  test("remove  agg tables from show table command") {
    +    sql("DROP TABLE IF EXISTS tbl_1")
    +    sql("create table if not exists  tbl_1(imei string,age int,mac string ,prodate timestamp,update timestamp,gamepoint double,contrid double) stored by 'carbondata' ")
    +    sql("create datamap agg1 on table tbl_1 using 'preaggregate' as select mac, sum(age) from tbl_1 group by mac")
    +    sql("create table if not exists  sparktable(imei string,age int,mac string ,prodate timestamp,update timestamp,gamepoint double,contrid double) ")
    +    checkExistence(sql("show tables"), false, "tbl_1_agg1")
    +    checkExistence(sql("show tables"), true, "sparktable","tbl_1")
    +  }
    +
    +
    +  test("remove TimeSeries agg tables from show table command") {
    +    sql("DROP TABLE IF EXISTS tbl_1")
    +    sql("create table if not exists  tbl_1(imei string,age int,mac string ,prodate timestamp,update timestamp,gamepoint double,contrid double) stored by 'carbondata' ")
    +    sql(
    +      "create datamap agg2 on table tbl_1 using 'preaggregate' DMPROPERTIES ('timeseries" +
    --- End diff --
   
    we don't support create timeseries table like this :  'timeseries.eventTime'='prodate', 'timeseries.hierarchy'


---

[qiuchenjian-2]

Github user xubo245 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1878#discussion_r165807488
 
    --- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/preaggregate/TestPreAggCreateCommand.scala ---
    @@ -216,6 +216,20 @@ class TestPreAggCreateCommand extends QueryTest with BeforeAndAfterAll {
       }
     
       val timeSeries = TIMESERIES.toString
    +  test("remove agg tables from show table command") {
    +    sql("DROP TABLE IF EXISTS tbl_1")
    +    sql("DROP TABLE IF EXISTS sparktable")
    +    sql("create table if not exists  tbl_1(imei string,age int,mac string ,prodate timestamp,update timestamp,gamepoint double,contrid double) stored by 'carbondata' ")
    +    sql("create table if not exists sparktable(a int,b string)")
    +    sql(
    +      s"""create datamap preagg_sum on table tbl_1 using 'preaggregate' as select mac,avg(age) from tbl_1 group by mac"""
    +        .stripMargin)
    +    sql(
    +      "create datamap agg2 on table tbl_1 using 'preaggregate' DMPROPERTIES ('timeseries" +
    +      ".eventTime'='prodate', 'timeseries.hierarchy'='hour=1,day=1,month=1,year=1') as select prodate," +
    --- End diff --
   
    we don't support create timeseries table like this now : 'timeseries.eventTime'='prodate', 'timeseries.hierarchy'


---

[qiuchenjian-2]

Github user xubo245 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1878#discussion_r166847284
 
    --- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/preaggregate/TestPreAggregateDrop.scala ---
    @@ -46,8 +46,9 @@ class TestPreAggregateDrop extends QueryTest with BeforeAndAfterAll {
           " a,sum(c) from maintable group by a")
         sql("drop datamap if exists preagg2 on table maintable")
         val showTables = sql("show tables")
    +    val showdatamaps =sql("show datamap on table maintable")
         checkExistence(showTables, false, "maintable_preagg2")
    --- End diff --
   
    this test case is invalid after you change the function of "show tables


---

[qiuchenjian-2]

Github user xubo245 commented on the issue:

    https://github.com/apache/carbondata/pull/1878
 
    There are many test case is unuseful/invalid after you change the function of  "show tables"



---

[qiuchenjian-2]

Github user xubo245 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1878#discussion_r166847806
 
    --- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/preaggregate/TestPreAggregateDrop.scala ---
    @@ -46,8 +46,9 @@ class TestPreAggregateDrop extends QueryTest with BeforeAndAfterAll {
           " a,sum(c) from maintable group by a")
         sql("drop datamap if exists preagg2 on table maintable")
         val showTables = sql("show tables")
    +    val showdatamaps =sql("show datamap on table maintable")
         checkExistence(showTables, false, "maintable_preagg2")
    --- End diff --
   
    for example:
   
      test("dropping 1 aggregate table should not drop others") {
         sql(
           "create datamap preagg1 on table maintable using 'preaggregate' as select" +
           " a,sum(b) from maintable group by a")
         sql(
           "create datamap preagg2 on table maintable using 'preaggregate' as select" +
           " a,sum(c) from maintable group by a")
         sql("drop datamap if exists preagg2 on table maintable")
         val showTables = sql("show tables")
         checkExistence(showTables, false, "maintable_preagg2")
   
    showTables will always not contain maintable_preagg2, so we need adapt new show table function. There are some test case need check again.  @BJangir @jackylk @chenliang613 @sraghunandan @ravipesala


---

[qiuchenjian-2]

Github user xubo245 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1878#discussion_r166848593
 
    --- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/preaggregate/TestPreAggCreateCommand.scala ---
    @@ -216,6 +216,20 @@ class TestPreAggCreateCommand extends QueryTest with BeforeAndAfterAll {
       }
     
       val timeSeries = TIMESERIES.toString
    +  test("remove agg tables from show table command") {
    +    sql("DROP TABLE IF EXISTS tbl_1")
    +    sql("DROP TABLE IF EXISTS sparktable")
    +    sql("create table if not exists  tbl_1(imei string,age int,mac string ,prodate timestamp,update timestamp,gamepoint double,contrid double) stored by 'carbondata' ")
    +    sql("create table if not exists sparktable(a int,b string)")
    +    sql(
    +      s"""create datamap preagg_sum on table tbl_1 using 'preaggregate' as select mac,avg(age) from tbl_1 group by mac"""
    +        .stripMargin)
    +    sql(
    +      "create datamap agg2 on table tbl_1 using 'preaggregate' DMPROPERTIES ('timeseries" +
    +      ".eventTime'='prodate', 'timeseries.hierarchy'='hour=1,day=1,month=1,year=1') as select prodate," +
    --- End diff --
   
    I fix it on https://github.com/apache/carbondata/pull/1929, please review it @BJangir @jackylk @ravipesala @sraghunandan


---