[GitHub] carbondata pull request #1443: [CARBONDATA-1524[CARBONDATA-1525] Added suppo...

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1443
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1503/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1443
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1507/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1443
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/914/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1443
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/915/



---

[qiuchenjian-2]

Github user kunal642 commented on the issue:

    https://github.com/apache/carbondata/pull/1443
 
    retest this please


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1443
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/916/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1443
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/919/



---

[qiuchenjian-2]

Github user kunal642 commented on the issue:

    https://github.com/apache/carbondata/pull/1443
 
    @ravipesala Please review


---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1443#discussion_r149931432
 
    --- Diff: examples/spark2/src/main/scala/org/apache/carbondata/examples/CarbonSessionExample.scala ---
    @@ -56,7 +56,8 @@ object CarbonSessionExample {
           s"""
              | LOAD DATA LOCAL INPATH '$path'
              | INTO TABLE carbon_table
    -         | OPTIONS('HEADER'='true', 'COMPLEX_DELIMITER_LEVEL_1'='#')
    +         | OPTIONS('FILEHEADER'='shortField,intField,bigintField,doubleField,stringField,timestampField,decimalField,dateField,charField,floatField,complexData',
    --- End diff --
   
    don't change


---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1443#discussion_r149931865
 
    --- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/preaggregate/TestPreAggregateDrop.scala ---
    @@ -0,0 +1,65 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.carbondata.integration.spark.testsuite.preaggregate
    +
    +import org.apache.spark.sql.test.util.QueryTest
    +import org.scalatest.BeforeAndAfterAll
    +
    +class TestPreAggregateDrop extends QueryTest with BeforeAndAfterAll {
    +
    +  override def beforeAll {
    +    sql("drop table if exists maintable")
    +    sql("drop table if exists preagg1")
    +    sql("drop table if exists preagg2")
    +    sql("create table maintable (a string, b string, c string) stored by 'carbondata'")
    +  }
    +
    +  test("create and drop preaggregate table") {
    +    sql(
    +      "create table preagg1 stored BY 'carbondata' tblproperties('parent'='maintable') as select" +
    +      " a,sum(b) from maintable group by a")
    +    sql("drop table if exists preagg1")
    +    checkExistence(sql("show tables"), false, "preagg1")
    +  }
    +
    +  test("dropping 1 aggregate table should not drop others") {
    +    sql(
    +      "create table preagg1 stored BY 'carbondata' tblproperties('parent'='maintable') as select" +
    +      " a,sum(b) from maintable group by a")
    +    sql(
    +      "create table preagg2 stored BY 'carbondata' tblproperties('parent'='maintable') as select" +
    +      " a,sum(c) from maintable group by a")
    +    sql("drop table if exists preagg2")
    +    checkExistence(sql("show tables"), false, "preagg2")
    --- End diff --
   
    Add check for other preagg1 exists


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1443
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1529/



---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1443#discussion_r149936562
 
    --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonFileMetastore.scala ---
    @@ -446,6 +447,35 @@ class CarbonFileMetastore extends CarbonMetaStore {
         }
       }
     
    +  protected def updateParentTableInfo(parentRelationIdentifier: RelationIdentifier,
    +      childCarbonTable: CarbonTable)(sparkSession: SparkSession): Unit = {
    +    val dbName = parentRelationIdentifier.getDatabaseName
    +    val tableName = parentRelationIdentifier.getTableName
    +    try {
    +      val parentCarbonTable = CarbonEnv.getInstance(sparkSession).carbonMetastore
    +        .lookupRelation(Some(dbName), tableName)(sparkSession).asInstanceOf[CarbonRelation]
    +        .tableMeta.carbonTable
    +      val childSchemas = parentCarbonTable.getTableInfo.getDataMapSchemaList
    --- End diff --
   
    add a null check and throw exception saying unexpected


---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1443#discussion_r149936691
 
    --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonFileMetastore.scala ---
    @@ -446,6 +447,35 @@ class CarbonFileMetastore extends CarbonMetaStore {
         }
       }
     
    +  protected def updateParentTableInfo(parentRelationIdentifier: RelationIdentifier,
    +      childCarbonTable: CarbonTable)(sparkSession: SparkSession): Unit = {
    +    val dbName = parentRelationIdentifier.getDatabaseName
    +    val tableName = parentRelationIdentifier.getTableName
    +    try {
    +      val parentCarbonTable = CarbonEnv.getInstance(sparkSession).carbonMetastore
    +        .lookupRelation(Some(dbName), tableName)(sparkSession).asInstanceOf[CarbonRelation]
    +        .tableMeta.carbonTable
    +      val childSchemas = parentCarbonTable.getTableInfo.getDataMapSchemaList
    +      val childSchemaIterator = childSchemas.iterator()
    +      while (childSchemaIterator.hasNext) {
    +        val childSchema = childSchemaIterator.next()
    +        if (childSchema.getChildSchema.equals(childCarbonTable.getTableInfo.getFactTable)) {
    +          childSchemaIterator.remove()
    +        }
    +      }
    +      val schemaConverter = new ThriftWrapperSchemaConverterImpl
    +      PreAggregateUtil
    +        .updateSchemaInfo(parentCarbonTable,
    +          schemaConverter
    +            .fromWrapperToExternalTableInfo(parentCarbonTable.getTableInfo,
    +              dbName,
    +              tableName))(sparkSession)
    +    } catch {
    +      case ex: Exception =>
    +        LOGGER.error(ex, s"Table $dbName.$tableName does not exist.")
    --- End diff --
   
    throw exception


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1443
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1530/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1443
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1531/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1443
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/932/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1443
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1534/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1443
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1547/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1443
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/949/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1443
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1564/



---