[GitHub] [carbondata] nihal0107 opened a new pull request #4071: [CARBONDATA-4102] Added UT and FT to improve coverage of SI module.

classic Classic list List threaded Threaded
103 messages Options
1234 ... 6
Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] nihal0107 opened a new pull request #4071: [CARBONDATA-4102] Added UT and FT to improve coverage of SI module.

GitBox

nihal0107 opened a new pull request #4071:
URL: https://github.com/apache/carbondata/pull/4071


    ### Why is this PR needed?
    Added UT and FT to improve coverage of SI module and also removed the dead or unused code.
   
    ### What changes were proposed in this PR?
    Added UT and FT to improve coverage of SI module and also removed the dead or unused code.
       
    ### Does this PR introduce any user interface change?
    - No
   
    ### Is any new testcase added?
    - Yes
   
       
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4071: [CARBONDATA-4102] Added UT and FT to improve coverage of SI module.

GitBox

CarbonDataQA2 commented on pull request #4071:
URL: https://github.com/apache/carbondata/pull/4071#issuecomment-756055862


   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5287/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4071: [CARBONDATA-4102] Added UT and FT to improve coverage of SI module.

GitBox
In reply to this post by GitBox

CarbonDataQA2 commented on pull request #4071:
URL: https://github.com/apache/carbondata/pull/4071#issuecomment-756057859


   Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3527/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4071: [CARBONDATA-4102] Added UT and FT to improve coverage of SI module.

GitBox
In reply to this post by GitBox

CarbonDataQA2 commented on pull request #4071:
URL: https://github.com/apache/carbondata/pull/4071#issuecomment-756127331


   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5288/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4071: [CARBONDATA-4102] Added UT and FT to improve coverage of SI module.

GitBox
In reply to this post by GitBox

CarbonDataQA2 commented on pull request #4071:
URL: https://github.com/apache/carbondata/pull/4071#issuecomment-756127667


   Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3528/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] Indhumathi27 commented on pull request #4071: [CARBONDATA-4102] Added UT and FT to improve coverage of SI module.

GitBox
In reply to this post by GitBox

Indhumathi27 commented on pull request #4071:
URL: https://github.com/apache/carbondata/pull/4071#issuecomment-761995983


   @nihal0107 Can you attach the report of how which SI coverage is increased from this PR


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] nihal0107 commented on pull request #4071: [CARBONDATA-4102] Added UT and FT to improve coverage of SI module.

GitBox
In reply to this post by GitBox

nihal0107 commented on pull request #4071:
URL: https://github.com/apache/carbondata/pull/4071#issuecomment-762304031


   @Indhumathi27 please find the comparison with master branch and current branch
   
   x-special/nautilus-clipboard
   copy
   file:///home/nihal/Pictures/after1.png
   file:///home/nihal/Pictures/before2.png
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] nihal0107 removed a comment on pull request #4071: [CARBONDATA-4102] Added UT and FT to improve coverage of SI module.

GitBox
In reply to this post by GitBox

nihal0107 removed a comment on pull request #4071:
URL: https://github.com/apache/carbondata/pull/4071#issuecomment-762304031


   @Indhumathi27 please find the comparison with master branch and current branch
   
   x-special/nautilus-clipboard
   copy
   file:///home/nihal/Pictures/after1.png
   file:///home/nihal/Pictures/before2.png
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] nihal0107 commented on pull request #4071: [CARBONDATA-4102] Added UT and FT to improve coverage of SI module.

GitBox
In reply to this post by GitBox

nihal0107 commented on pull request #4071:
URL: https://github.com/apache/carbondata/pull/4071#issuecomment-762305509


   ![before2](https://user-images.githubusercontent.com/32429250/104931386-56d60880-59cc-11eb-9914-c38b30f99c72.png)
   ![after1](https://user-images.githubusercontent.com/32429250/104931415-5f2e4380-59cc-11eb-8dde-5c5f2e46feec.png)
   
   
   @Indhumathi27 please find the comparision between master and current branch


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] Indhumathi27 commented on a change in pull request #4071: [CARBONDATA-4102] Added UT and FT to improve coverage of SI module.

GitBox
In reply to this post by GitBox

Indhumathi27 commented on a change in pull request #4071:
URL: https://github.com/apache/carbondata/pull/4071#discussion_r562567398



##########
File path: index/secondary-index/src/test/scala/org/apache/carbondata/spark/testsuite/mergedata/CarbonDataFileMergeTestCaseOnSI.scala
##########
@@ -288,9 +268,7 @@ class CarbonDataFileMergeTestCaseOnSI
       CarbonCommonConstants.CARBON_SI_SEGMENT_MERGE_DEFAULT)
   }
 
-  test("test verify data file merge when exception occurred in rebuild segment") {
-    CarbonProperties.getInstance()
-      .addProperty(CarbonCommonConstants.CARBON_SI_SEGMENT_MERGE, "false")
+  def createTableAndLoadData(): Unit = {

Review comment:
       Still five occurences of 'CREATE TABLE nonindexmerge' can be replaced. GLOBAL_SORT_PARTITIONS can be passed  from test cases




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] Indhumathi27 commented on a change in pull request #4071: [CARBONDATA-4102] Added UT and FT to improve coverage of SI module.

GitBox
In reply to this post by GitBox

Indhumathi27 commented on a change in pull request #4071:
URL: https://github.com/apache/carbondata/pull/4071#discussion_r562601511



##########
File path: index/secondary-index/src/test/scala/org/apache/carbondata/spark/testsuite/secondaryindex/DropTableTest.scala
##########
@@ -88,4 +90,17 @@ class DropTableTest extends QueryTest with BeforeAndAfterAll {
     assert(sql("show indexes on testDrop").collect().isEmpty)
     sql("drop table if exists testDrop")
   }
+
+  test("test index drop when SI table is not deleted while main table is deleted") {
+    sql("drop table if exists testDrop")
+    sql("create table testDrop (a string, b string, c string) STORED AS carbondata")
+    sql("create index index11 on table testDrop (c) AS 'carbondata'")
+    sql("insert into testDrop values('ab', 'cd', 'ef')")
+    val mock = TestSecondaryIndexUtils.mockSIDrop()
+    sql("drop table if exists testDrop")
+    mock.tearDown()
+    assert(Files.exists(Paths.get(warehouse + "/" + "index11")))
+    sql("drop table if exists testDrop")
+    sql("drop database if exists test cascade")

Review comment:
       looks like this database is not created in this testclass.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] Indhumathi27 commented on a change in pull request #4071: [CARBONDATA-4102] Added UT and FT to improve coverage of SI module.

GitBox
In reply to this post by GitBox

Indhumathi27 commented on a change in pull request #4071:
URL: https://github.com/apache/carbondata/pull/4071#discussion_r562601511



##########
File path: index/secondary-index/src/test/scala/org/apache/carbondata/spark/testsuite/secondaryindex/DropTableTest.scala
##########
@@ -88,4 +90,17 @@ class DropTableTest extends QueryTest with BeforeAndAfterAll {
     assert(sql("show indexes on testDrop").collect().isEmpty)
     sql("drop table if exists testDrop")
   }
+
+  test("test index drop when SI table is not deleted while main table is deleted") {
+    sql("drop table if exists testDrop")
+    sql("create table testDrop (a string, b string, c string) STORED AS carbondata")
+    sql("create index index11 on table testDrop (c) AS 'carbondata'")
+    sql("insert into testDrop values('ab', 'cd', 'ef')")
+    val mock = TestSecondaryIndexUtils.mockSIDrop()
+    sql("drop table if exists testDrop")
+    mock.tearDown()
+    assert(Files.exists(Paths.get(warehouse + "/" + "index11")))
+    sql("drop table if exists testDrop")
+    sql("drop database if exists test cascade")

Review comment:
       looks like this database `test` is not created in this testclass.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] Indhumathi27 commented on a change in pull request #4071: [CARBONDATA-4102] Added UT and FT to improve coverage of SI module.

GitBox
In reply to this post by GitBox

Indhumathi27 commented on a change in pull request #4071:
URL: https://github.com/apache/carbondata/pull/4071#discussion_r562603767



##########
File path: index/secondary-index/src/test/scala/org/apache/carbondata/spark/testsuite/secondaryindex/TestCarbonInternalMetastore.scala
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.spark.testsuite.secondaryindex
+
+import org.apache.spark.sql.CarbonEnv
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.index.CarbonIndexUtil
+import org.apache.spark.sql.secondaryindex.hive.CarbonInternalMetastore
+import org.apache.spark.sql.test.util.QueryTest
+import org.scalatest.{BeforeAndAfterAll, BeforeAndAfterEach}
+
+import org.apache.carbondata.core.metadata.schema.table.CarbonTable
+
+class TestCarbonInternalMetastore extends QueryTest with BeforeAndAfterAll with BeforeAndAfterEach {
+
+  val dbName: String = "test"
+  val tableName: String = "table1"
+  val indexName: String = "index1"
+  var tableIdentifier: TableIdentifier = _
+  var parentCarbonTable: CarbonTable = _
+  var indexTable: CarbonTable = _
+
+  override def beforeAll(): Unit = {
+    sql("drop database if exists test cascade");
+  }
+
+  override def beforeEach(): Unit = {
+    sql("drop database if exists test cascade");
+    sql("create database test")
+    sql("use test")
+    sql("drop table if exists table1")
+    sql("create table table1(a string, b string, c string) stored as carbondata")
+    sql("create index index1 on table1(b) as 'carbondata'")
+    sql("insert into table1 values('ab','bc','cd')")
+  }
+
+  def setVariables(indexName: String): Unit = {
+    tableIdentifier = new TableIdentifier(indexName, Some("test"))
+    parentCarbonTable = CarbonEnv.getCarbonTable(Some("test"), "table1")(sqlContext.sparkSession)
+    indexTable = CarbonEnv.getCarbonTable(Some("test"), "index1")(sqlContext.sparkSession)
+  }
+
+  test("test delete index silent") {
+    setVariables("index1")
+    CarbonInternalMetastore.deleteIndexSilent(tableIdentifier, "",
+      parentCarbonTable)(sqlContext.sparkSession)
+    checkExistence(sql("show indexes on table1"), false, "index1")
+  }
+
+  test("test delete index table silently when exception occur") {
+    setVariables("unknown")
+    CarbonInternalMetastore.deleteIndexSilent(tableIdentifier, "",
+      parentCarbonTable)(sqlContext.sparkSession)
+    checkExistence(sql("show indexes on table1"), true, "index1")
+    setVariables("index1")
+    CarbonInternalMetastore.deleteIndexSilent(tableIdentifier, "",
+      indexTable)(sqlContext.sparkSession)

Review comment:
       is it needed? proper use case?

##########
File path: index/secondary-index/src/test/scala/org/apache/carbondata/spark/testsuite/secondaryindex/TestCarbonInternalMetastore.scala
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.spark.testsuite.secondaryindex
+
+import org.apache.spark.sql.CarbonEnv
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.index.CarbonIndexUtil
+import org.apache.spark.sql.secondaryindex.hive.CarbonInternalMetastore
+import org.apache.spark.sql.test.util.QueryTest
+import org.scalatest.{BeforeAndAfterAll, BeforeAndAfterEach}
+
+import org.apache.carbondata.core.metadata.schema.table.CarbonTable
+
+class TestCarbonInternalMetastore extends QueryTest with BeforeAndAfterAll with BeforeAndAfterEach {
+
+  val dbName: String = "test"
+  val tableName: String = "table1"
+  val indexName: String = "index1"
+  var tableIdentifier: TableIdentifier = _
+  var parentCarbonTable: CarbonTable = _
+  var indexTable: CarbonTable = _
+
+  override def beforeAll(): Unit = {
+    sql("drop database if exists test cascade");
+  }
+
+  override def beforeEach(): Unit = {
+    sql("drop database if exists test cascade");
+    sql("create database test")
+    sql("use test")
+    sql("drop table if exists table1")
+    sql("create table table1(a string, b string, c string) stored as carbondata")
+    sql("create index index1 on table1(b) as 'carbondata'")
+    sql("insert into table1 values('ab','bc','cd')")
+  }
+
+  def setVariables(indexName: String): Unit = {
+    tableIdentifier = new TableIdentifier(indexName, Some("test"))
+    parentCarbonTable = CarbonEnv.getCarbonTable(Some("test"), "table1")(sqlContext.sparkSession)
+    indexTable = CarbonEnv.getCarbonTable(Some("test"), "index1")(sqlContext.sparkSession)
+  }
+
+  test("test delete index silent") {
+    setVariables("index1")
+    CarbonInternalMetastore.deleteIndexSilent(tableIdentifier, "",
+      parentCarbonTable)(sqlContext.sparkSession)
+    checkExistence(sql("show indexes on table1"), false, "index1")
+  }
+
+  test("test delete index table silently when exception occur") {
+    setVariables("unknown")
+    CarbonInternalMetastore.deleteIndexSilent(tableIdentifier, "",
+      parentCarbonTable)(sqlContext.sparkSession)
+    checkExistence(sql("show indexes on table1"), true, "index1")
+    setVariables("index1")
+    CarbonInternalMetastore.deleteIndexSilent(tableIdentifier, "",
+      indexTable)(sqlContext.sparkSession)
+    checkExistence(sql("show indexes on table1"), true, "index1")
+  }
+
+  test("test show index when SI were created before the change CARBONDATA-3765") {
+    val mock = TestSecondaryIndexUtils.mockIndexInfo()
+    checkExistence(sql("show indexes on table1"), true, "index1")
+    mock.tearDown()
+  }
+
+  test("test refresh index with different value of isIndexTableExists") {
+    setVariables("index1")
+    sql("create index index2 on table1(b) as 'bloomfilter'")
+    parentCarbonTable = CarbonEnv.getCarbonTable(Some("test"), "table1")(sqlContext.sparkSession)
+    assert(CarbonIndexUtil.isIndexExists(parentCarbonTable).equalsIgnoreCase("true"))
+    CarbonIndexUtil.addOrModifyTableProperty(parentCarbonTable, Map("indexExists" -> "false"))(

Review comment:
       i think here, you need to verify the property indextableexists which table has SI and indexExists for bloom/luecene




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4071: [CARBONDATA-4102] Added UT and FT to improve coverage of SI module.

GitBox
In reply to this post by GitBox

CarbonDataQA2 commented on pull request #4071:
URL: https://github.com/apache/carbondata/pull/4071#issuecomment-766553712


   Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/3295/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4071: [CARBONDATA-4102] Added UT and FT to improve coverage of SI module.

GitBox
In reply to this post by GitBox

CarbonDataQA2 commented on pull request #4071:
URL: https://github.com/apache/carbondata/pull/4071#issuecomment-766567489


   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/5053/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] nihal0107 commented on a change in pull request #4071: [CARBONDATA-4102] Added UT and FT to improve coverage of SI module.

GitBox
In reply to this post by GitBox

nihal0107 commented on a change in pull request #4071:
URL: https://github.com/apache/carbondata/pull/4071#discussion_r563482082



##########
File path: index/secondary-index/src/test/scala/org/apache/carbondata/spark/testsuite/mergedata/CarbonDataFileMergeTestCaseOnSI.scala
##########
@@ -288,9 +268,7 @@ class CarbonDataFileMergeTestCaseOnSI
       CarbonCommonConstants.CARBON_SI_SEGMENT_MERGE_DEFAULT)
   }
 
-  test("test verify data file merge when exception occurred in rebuild segment") {
-    CarbonProperties.getInstance()
-      .addProperty(CarbonCommonConstants.CARBON_SI_SEGMENT_MERGE, "false")
+  def createTableAndLoadData(): Unit = {

Review comment:
       done

##########
File path: index/secondary-index/src/test/scala/org/apache/carbondata/spark/testsuite/secondaryindex/DropTableTest.scala
##########
@@ -88,4 +90,17 @@ class DropTableTest extends QueryTest with BeforeAndAfterAll {
     assert(sql("show indexes on testDrop").collect().isEmpty)
     sql("drop table if exists testDrop")
   }
+
+  test("test index drop when SI table is not deleted while main table is deleted") {
+    sql("drop table if exists testDrop")
+    sql("create table testDrop (a string, b string, c string) STORED AS carbondata")
+    sql("create index index11 on table testDrop (c) AS 'carbondata'")
+    sql("insert into testDrop values('ab', 'cd', 'ef')")
+    val mock = TestSecondaryIndexUtils.mockSIDrop()
+    sql("drop table if exists testDrop")
+    mock.tearDown()
+    assert(Files.exists(Paths.get(warehouse + "/" + "index11")))
+    sql("drop table if exists testDrop")
+    sql("drop database if exists test cascade")

Review comment:
       removed




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] nihal0107 commented on a change in pull request #4071: [CARBONDATA-4102] Added UT and FT to improve coverage of SI module.

GitBox
In reply to this post by GitBox

nihal0107 commented on a change in pull request #4071:
URL: https://github.com/apache/carbondata/pull/4071#discussion_r563482438



##########
File path: index/secondary-index/src/test/scala/org/apache/carbondata/spark/testsuite/secondaryindex/TestCarbonInternalMetastore.scala
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.spark.testsuite.secondaryindex
+
+import org.apache.spark.sql.CarbonEnv
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.index.CarbonIndexUtil
+import org.apache.spark.sql.secondaryindex.hive.CarbonInternalMetastore
+import org.apache.spark.sql.test.util.QueryTest
+import org.scalatest.{BeforeAndAfterAll, BeforeAndAfterEach}
+
+import org.apache.carbondata.core.metadata.schema.table.CarbonTable
+
+class TestCarbonInternalMetastore extends QueryTest with BeforeAndAfterAll with BeforeAndAfterEach {
+
+  val dbName: String = "test"
+  val tableName: String = "table1"
+  val indexName: String = "index1"
+  var tableIdentifier: TableIdentifier = _
+  var parentCarbonTable: CarbonTable = _
+  var indexTable: CarbonTable = _
+
+  override def beforeAll(): Unit = {
+    sql("drop database if exists test cascade");
+  }
+
+  override def beforeEach(): Unit = {
+    sql("drop database if exists test cascade");
+    sql("create database test")
+    sql("use test")
+    sql("drop table if exists table1")
+    sql("create table table1(a string, b string, c string) stored as carbondata")
+    sql("create index index1 on table1(b) as 'carbondata'")
+    sql("insert into table1 values('ab','bc','cd')")
+  }
+
+  def setVariables(indexName: String): Unit = {
+    tableIdentifier = new TableIdentifier(indexName, Some("test"))
+    parentCarbonTable = CarbonEnv.getCarbonTable(Some("test"), "table1")(sqlContext.sparkSession)
+    indexTable = CarbonEnv.getCarbonTable(Some("test"), "index1")(sqlContext.sparkSession)
+  }
+
+  test("test delete index silent") {
+    setVariables("index1")
+    CarbonInternalMetastore.deleteIndexSilent(tableIdentifier, "",
+      parentCarbonTable)(sqlContext.sparkSession)
+    checkExistence(sql("show indexes on table1"), false, "index1")
+  }
+
+  test("test delete index table silently when exception occur") {
+    setVariables("unknown")
+    CarbonInternalMetastore.deleteIndexSilent(tableIdentifier, "",
+      parentCarbonTable)(sqlContext.sparkSession)
+    checkExistence(sql("show indexes on table1"), true, "index1")
+    setVariables("index1")
+    CarbonInternalMetastore.deleteIndexSilent(tableIdentifier, "",
+      indexTable)(sqlContext.sparkSession)

Review comment:
       this is just to check when we are passing index table as parent table then delete will fail. I have added comments and also included true condition.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] nihal0107 commented on a change in pull request #4071: [CARBONDATA-4102] Added UT and FT to improve coverage of SI module.

GitBox
In reply to this post by GitBox

nihal0107 commented on a change in pull request #4071:
URL: https://github.com/apache/carbondata/pull/4071#discussion_r563482533



##########
File path: index/secondary-index/src/test/scala/org/apache/carbondata/spark/testsuite/secondaryindex/TestCarbonInternalMetastore.scala
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.spark.testsuite.secondaryindex
+
+import org.apache.spark.sql.CarbonEnv
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.index.CarbonIndexUtil
+import org.apache.spark.sql.secondaryindex.hive.CarbonInternalMetastore
+import org.apache.spark.sql.test.util.QueryTest
+import org.scalatest.{BeforeAndAfterAll, BeforeAndAfterEach}
+
+import org.apache.carbondata.core.metadata.schema.table.CarbonTable
+
+class TestCarbonInternalMetastore extends QueryTest with BeforeAndAfterAll with BeforeAndAfterEach {
+
+  val dbName: String = "test"
+  val tableName: String = "table1"
+  val indexName: String = "index1"
+  var tableIdentifier: TableIdentifier = _
+  var parentCarbonTable: CarbonTable = _
+  var indexTable: CarbonTable = _
+
+  override def beforeAll(): Unit = {
+    sql("drop database if exists test cascade");
+  }
+
+  override def beforeEach(): Unit = {
+    sql("drop database if exists test cascade");
+    sql("create database test")
+    sql("use test")
+    sql("drop table if exists table1")
+    sql("create table table1(a string, b string, c string) stored as carbondata")
+    sql("create index index1 on table1(b) as 'carbondata'")
+    sql("insert into table1 values('ab','bc','cd')")
+  }
+
+  def setVariables(indexName: String): Unit = {
+    tableIdentifier = new TableIdentifier(indexName, Some("test"))
+    parentCarbonTable = CarbonEnv.getCarbonTable(Some("test"), "table1")(sqlContext.sparkSession)
+    indexTable = CarbonEnv.getCarbonTable(Some("test"), "index1")(sqlContext.sparkSession)
+  }
+
+  test("test delete index silent") {
+    setVariables("index1")
+    CarbonInternalMetastore.deleteIndexSilent(tableIdentifier, "",
+      parentCarbonTable)(sqlContext.sparkSession)
+    checkExistence(sql("show indexes on table1"), false, "index1")
+  }
+
+  test("test delete index table silently when exception occur") {
+    setVariables("unknown")
+    CarbonInternalMetastore.deleteIndexSilent(tableIdentifier, "",
+      parentCarbonTable)(sqlContext.sparkSession)
+    checkExistence(sql("show indexes on table1"), true, "index1")
+    setVariables("index1")
+    CarbonInternalMetastore.deleteIndexSilent(tableIdentifier, "",
+      indexTable)(sqlContext.sparkSession)
+    checkExistence(sql("show indexes on table1"), true, "index1")
+  }
+
+  test("test show index when SI were created before the change CARBONDATA-3765") {
+    val mock = TestSecondaryIndexUtils.mockIndexInfo()
+    checkExistence(sql("show indexes on table1"), true, "index1")
+    mock.tearDown()
+  }
+
+  test("test refresh index with different value of isIndexTableExists") {
+    setVariables("index1")
+    sql("create index index2 on table1(b) as 'bloomfilter'")
+    parentCarbonTable = CarbonEnv.getCarbonTable(Some("test"), "table1")(sqlContext.sparkSession)
+    assert(CarbonIndexUtil.isIndexExists(parentCarbonTable).equalsIgnoreCase("true"))
+    CarbonIndexUtil.addOrModifyTableProperty(parentCarbonTable, Map("indexExists" -> "false"))(

Review comment:
       yeah, made the changes.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4071: [CARBONDATA-4102] Added UT and FT to improve coverage of SI module.

GitBox
In reply to this post by GitBox

CarbonDataQA2 commented on pull request #4071:
URL: https://github.com/apache/carbondata/pull/4071#issuecomment-766617968


   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5344/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


Reply | Threaded
Open this post in threaded view
|

[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4071: [CARBONDATA-4102] Added UT and FT to improve coverage of SI module.

GitBox
In reply to this post by GitBox

CarbonDataQA2 commented on pull request #4071:
URL: https://github.com/apache/carbondata/pull/4071#issuecomment-766622335


   Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3583/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]


1234 ... 6