GitHub user xubo245 opened a pull request:
https://github.com/apache/carbondata/pull/1963 [WIP][CARBONDATA-2153] Failed to update table status for pre-aggregate table when maintable load twice
// TODO: to be fixed
test("test whether all segments are loaded into pre-aggregate table: error in auto merge and input segment") {
CarbonProperties.getInstance()
.addProperty(CarbonCommonConstants.ENABLE_AUTO_LOAD_MERGE, "true")
sql("reset")
sql("DROP TABLE IF EXISTS main_table")
sql(
"""
| CREATE TABLE main_table(
| id INT,
| name STRING,
| city STRING,
| age INT)
| STORED BY 'org.apache.carbondata.format'
""".stripMargin)
sql(s"INSERT INTO main_table VALUES(1, 'xyz', 'bengaluru', 26)")
sql(s"INSERT INTO main_table VALUES(1, 'xyz', 'bengaluru', 26)")
sql(
s"""
| CREATE DATAMAP preagg_sum
| ON TABLE main_table
| USING 'preaggregate'
| AS SELECT id, SUM(age)
| FROM main_table
| GROUP BY id
""".stripMargin)
sql(s"INSERT INTO main_table VALUES(1, 'xyz', 'bengaluru', 26)")
sql("show segments for table main_table_preagg_sum").show()
sql(s"LOAD DATA LOCAL INPATH '$testData' INTO TABLE main_table")
sql(s"LOAD DATA LOCAL INPATH '$testData' INTO TABLE main_table")
sql("show segments for table main_table_preagg_sum").show()
checkExistence(sql("show segments for table main_table_preagg_sum"), true, "Compacted")
CarbonProperties.getInstance()
.addProperty(CarbonCommonConstants.ENABLE_AUTO_LOAD_MERGE, "false")
}
```
committing data maps
java.lang.RuntimeException: Failed to update table status for pre-aggregate table
at scala.sys.package$.error(package.scala:27)
at org.apache.spark.sql.execution.command.preaaggregate.CommitPreAggregateListener$.onEvent(PreAggregateListeners.scala:317)
at org.apache.carbondata.events.OperationListenerBus.fireEvent(OperationListenerBus.java:117)
at org.apache.carbondata.spark.rdd.CarbonTableCompactor.triggerCompaction(CarbonTableCompactor.scala:265)
at org.apache.carbondata.spark.rdd.CarbonTableCompactor.scanSegmentsAndSubmitJob(CarbonTableCompactor.scala:123)
at org.apache.carbondata.spark.rdd.CarbonTableCompactor.executeCompaction(CarbonTableCompactor.scala:74)
at org.apache.carbondata.spark.rdd.CarbonDataRDDFactory$$anon$2.run(CarbonDataRDDFactory.scala:172)
at org.apache.carbondata.spark.rdd.CarbonDataRDDFactory$.startCompactionThreads(CarbonDataRDDFactory.scala:260)
at org.apache.carbondata.spark.rdd.CarbonDataRDDFactory$.handleSegmentMerging(CarbonDataRDDFactory.scala:780)
at org.apache.carbondata.spark.rdd.CarbonDataRDDFactory$.loadCarbonData(CarbonDataRDDFactory.scala:568)
at org.apache.spark.sql.execution.command.management.CarbonLoadDataCommand.loadData(CarbonLoadDataCommand.scala:485)
at org.apache.spark.sql.execution.command.management.CarbonLoadDataCommand.processData(CarbonLoadDataCommand.scala:261)
at org.apache.spark.sql.execution.command.AtomicRunnableCommand.run(package.scala:92)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:58)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:56)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:67)
at org.apache.spark.sql.Dataset.<init>(Dataset.scala:183)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:68)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:632)
at org.apache.spark.sql.test.Spark2TestQueryExecutor.sql(Spark2TestQueryExecutor.scala:35)
at org.apache.spark.sql.test.util.QueryTest.sql(QueryTest.scala:113)
at org.apache.carbondata.integration.spark.testsuite.preaggregate.TestPreAggregateLoad$$anonfun$35.apply$mcV$sp(TestPreAggregateLoad.scala:1196)
```
and auto compaction fail:
```
ScalaTestFailureLocation: org.apache.spark.sql.test.util.QueryTest$$anonfun$checkExistence$1 at (QueryTest.scala:59)
org.scalatest.exceptions.TestFailedException: "3Success2018-02-09 02:10:17.4422018-02-09 02:10:18.146NACOLUMNAR_V32Success2018-02-09 02:10:15.8312018-02-09 02:10:16.756NACOLUMNAR_V31Success2018-02-09 02:10:14.1842018-02-09 02:10:15.175NACOLUMNAR_V30Success2018-02-09 02:10:11.0172018-02-09 02:10:13.773NACOLUMNAR_V3" did not contain "Compacted" Failed for [SegmentSequenceId: string, Status: string ... 4 more fields] (Compacted doesn't exist in result)
at org.scalatest.Assertions$class.newAssertionFailedException(Assertions.scala:500)
```
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
- [ ] Any interfaces changed?
- [ ] Any backward compatibility impacted?
- [ ] Document update required?
- [ ] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
You can merge this pull request into a Git repository by running:
$ git pull
https://github.com/xubo245/carbondata CARBONDATA2153_FailedUpdateTable
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/1963.patchTo close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1963
----
commit 5481032c48fcd3c2b12ac3741fe553cf046efde1
Author: xubo245 <601450868@...>
Date: 2018-02-09T10:09:42Z
[CARBONDATA-2153] Failed to update table status for pre-aggregate table when maintable load twice
----
---
Locked
