Apache CarbonData Dev Mailing List archive › Apache CarbonData JIRA issues

[jira] [Commented] (CARBONDATA-1422) Major and Minor Compaction Failing

classic Classic list List threaded Threaded
♦
♦
 1 message Options
Akash R Nilugal (Jira)
Reply | Threaded
Open this post in threaded view
♦
♦
|

[jira] [Commented] (CARBONDATA-1422) Major and Minor Compaction Failing

Akash R Nilugal (Jira)

    [ https://issues.apache.org/jira/browse/CARBONDATA-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16146975#comment-16146975 ]

Pallavi Singh commented on CARBONDATA-1422:
-------------------------------------------

Hi Zhichao Zhang,

Here is the stack-trace of the error:

17/08/30 15:12:30 INFO CatalystSqlParser: Parsing command: array<string>
17/08/30 15:12:30 AUDIT CarbonDataRDDFactory$: [pallavi][hduser][Thread-136]Compaction request received for table 30aug_5.uniqdata
17/08/30 15:12:30 INFO HdfsFileLock: pool-24-thread-7 HDFS lock path:hdfs://localhost:54310/user/hive/warehouse/carbon.store/30aug_5/uniqdata/compaction.lock
17/08/30 15:12:30 INFO CarbonDataRDDFactory$: pool-24-thread-7 Acquired the compaction lock for table 30aug_5.uniqdata
17/08/30 15:12:30 INFO DataManagementFunc$: pool-24-thread-7 loads identified for merge is 0
17/08/30 15:12:30 INFO DataManagementFunc$: pool-24-thread-7 loads identified for merge is 1
17/08/30 15:12:30 INFO DataManagementFunc$: pool-24-thread-7 loads identified for merge is 2
17/08/30 15:12:30 INFO DataManagementFunc$: pool-24-thread-7 loads identified for merge is 3
17/08/30 15:12:30 INFO DataManagementFunc$: pool-24-thread-7 loads identified for merge is 4
17/08/30 15:12:30 INFO DataManagementFunc$: pool-24-thread-7 loads identified for merge is 5
17/08/30 15:12:30 INFO DataManagementFunc$: pool-24-thread-7 loads identified for merge is 6
17/08/30 15:12:30 INFO DataManagementFunc$: pool-24-thread-7 loads identified for merge is 7
17/08/30 15:12:30 INFO TableInfo: pool-28-thread-1 Table block size not specified for 30aug_5_uniqdata. Therefore considering the default value 1024 MB
17/08/30 15:12:30 INFO DistributionUtil$: pool-28-thread-1 Requesting total executors: 1
17/08/30 15:12:30 INFO CarbonMergerRDD: No.of Executors required=1 , spark.executor.instances=1, no.of.nodes where data present=1
17/08/30 15:12:30 INFO CarbonMergerRDD: Time taken to wait for executor allocation is =0millis
17/08/30 15:12:30 INFO CarbonMergerRDD: for the node pallavi
17/08/30 15:12:30 INFO CarbonMergerRDD: Task ID is 0no. of blocks is 8
17/08/30 15:12:30 INFO CarbonMergerRDD: Identified  no.of.Blocks: 8,parallelism: 4 , no.of.nodes: 1, no.of.tasks: 1
17/08/30 15:12:30 INFO CarbonMergerRDD: Time taken to identify Blocks to scan : 177
17/08/30 15:12:30 INFO CarbonMergerRDD: Node: pallavi, No.Of Blocks: 8
17/08/30 15:12:30 INFO SparkContext: Starting job: collect at Compactor.scala:100
17/08/30 15:12:30 INFO DAGScheduler: Got job 4 (collect at Compactor.scala:100) with 1 output partitions
17/08/30 15:12:30 INFO DAGScheduler: Final stage: ResultStage 4 (collect at Compactor.scala:100)
17/08/30 15:12:30 INFO DAGScheduler: Parents of final stage: List()
17/08/30 15:12:30 INFO DAGScheduler: Missing parents: List()
17/08/30 15:12:30 INFO DAGScheduler: Submitting ResultStage 4 (CarbonMergerRDD[11] at RDD at CarbonRDD.scala:35), which has no missing parents
17/08/30 15:12:30 INFO MemoryStore: Block broadcast_6 stored as values in memory (estimated size 14.2 KB, free 366.2 MB)
17/08/30 15:12:30 INFO MemoryStore: Block broadcast_6_piece0 stored as bytes in memory (estimated size 6.7 KB, free 366.2 MB)
17/08/30 15:12:30 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on 192.168.2.188:41587 (size: 6.7 KB, free: 366.3 MB)
17/08/30 15:12:30 INFO SparkContext: Created broadcast 6 from broadcast at DAGScheduler.scala:996
17/08/30 15:12:30 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 4 (CarbonMergerRDD[11] at RDD at CarbonRDD.scala:35)
17/08/30 15:12:30 INFO TaskSchedulerImpl: Adding task set 4.0 with 1 tasks
17/08/30 15:12:30 INFO TaskSetManager: Starting task 0.0 in stage 4.0 (TID 4, 192.168.2.188, executor 0, partition 0, ANY, 9296 bytes)
17/08/30 15:12:30 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on 192.168.2.188:38477 (size: 6.7 KB, free: 366.3 MB)
17/08/30 15:12:31 INFO TaskSetManager: Finished task 0.0 in stage 4.0 (TID 4) in 266 ms on 192.168.2.188 (executor 0) (1/1)
17/08/30 15:12:31 INFO TaskSchedulerImpl: Removed TaskSet 4.0, whose tasks have all completed, from pool
17/08/30 15:12:31 INFO DAGScheduler: ResultStage 4 (collect at Compactor.scala:100) finished in 0.267 s
17/08/30 15:12:31 INFO DAGScheduler: Job 4 finished: collect at Compactor.scala:100, took 0.276120 s
17/08/30 15:12:31 AUDIT Compactor$: [pallavi][hduser][Thread-143]Compaction request failed for table 30aug_5.uniqdata
17/08/30 15:12:31 ERROR Compactor$: pool-28-thread-1 Compaction request failed for table 30aug_5.uniqdata
17/08/30 15:12:31 ERROR DataManagementFunc$: pool-24-thread-7 Exception in compaction thread java.lang.Exception: Compaction Failure in Merger Rdd.
java.util.concurrent.ExecutionException: java.lang.Exception: Compaction Failure in Merger Rdd.
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:192)
        at org.apache.carbondata.spark.rdd.DataManagementFunc$$anonfun$executeCompaction$1.apply(DataManagementFunc.scala:193)
        at org.apache.carbondata.spark.rdd.DataManagementFunc$$anonfun$executeCompaction$1.apply(DataManagementFunc.scala:192)
        at scala.collection.Iterator$class.foreach(Iterator.scala:893)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
        at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
        at org.apache.carbondata.spark.rdd.DataManagementFunc$.executeCompaction(DataManagementFunc.scala:192)
        at org.apache.carbondata.spark.rdd.CarbonDataRDDFactory$$anon$2.run(CarbonDataRDDFactory.scala:273)
        at org.apache.carbondata.spark.rdd.CarbonDataRDDFactory$.startCompactionThreads(CarbonDataRDDFactory.scala:364)
        at org.apache.carbondata.spark.rdd.CarbonDataRDDFactory$.alterTableForCompaction(CarbonDataRDDFactory.scala:157)
        at org.apache.spark.sql.execution.command.AlterTableCompaction.processData(carbonTableSchema.scala:175)
        at org.apache.spark.sql.execution.command.AlterTableCompaction.run(carbonTableSchema.scala:139)
        at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:58)
        at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:56)
        at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:74)
        at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114)
        at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114)
        at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:135)
        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
        at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:132)
        at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:113)
        at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:87)
        at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:87)
        at org.apache.spark.sql.Dataset.<init>(Dataset.scala:185)
        at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:64)
        at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:592)
        at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:699)
        at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute(SparkExecuteStatementOperation.scala:220)
        at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$1$$anon$2.run(SparkExecuteStatementOperation.scala:163)
        at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$1$$anon$2.run(SparkExecuteStatementOperation.scala:160)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
        at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$1.run(SparkExecuteStatementOperation.scala:173)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.Exception: Compaction Failure in Merger Rdd.
        at org.apache.carbondata.spark.rdd.Compactor$.triggerCompaction(Compactor.scala:143)
        at org.apache.carbondata.spark.rdd.Compactor.triggerCompaction(Compactor.scala)
        at org.apache.carbondata.spark.compaction.CompactionCallable.call(CompactionCallable.java:40)
        at org.apache.carbondata.spark.compaction.CompactionCallable.call(CompactionCallable.java:29)
        ... 4 more
17/08/30 15:12:31 ERROR CarbonDataRDDFactory$: pool-24-thread-7 Exception in compaction thread java.lang.Exception: Compaction Failure in Merger Rdd.
17/08/30 15:12:31 INFO HdfsFileLock: pool-24-thread-7 Deleted the lock file hdfs://localhost:54310/user/hive/warehouse/carbon.store/30aug_5/uniqdata/compaction.lock
17/08/30 15:12:31 ERROR CarbonDataRDDFactory$: pool-24-thread-7 Exception in start compaction thread. Exception in compaction java.lang.Exception: Compaction Failure in Merger Rdd.
17/08/30 15:12:31 ERROR AbstractDFSCarbonFile: pool-24-thread-7 Exception occurred:File does not exist: hdfs://localhost:54310/user/hive/warehouse/carbon.store/30aug_5/uniqdata/compaction.lock
17/08/30 15:12:31 ERROR HdfsFileLock: pool-24-thread-7 Not able to delete the lock file because it is not existed in location hdfs://localhost:54310/user/hive/warehouse/carbon.store/30aug_5/uniqdata/compaction.lock
17/08/30 15:12:31 ERROR SparkExecuteStatementOperation: Error executing query, currentState RUNNING,
java.lang.RuntimeException: Compaction failed. Please check logs for more info. Exception in compaction java.lang.Exception: Compaction Failure in Merger Rdd.
        at scala.sys.package$.error(package.scala:27)
        at org.apache.spark.sql.execution.command.AlterTableCompaction.processData(carbonTableSchema.scala:184)
        at org.apache.spark.sql.execution.command.AlterTableCompaction.run(carbonTableSchema.scala:139)
        at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:58)
        at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:56)
        at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:74)
        at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114)
        at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114)
        at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:135)
        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
        at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:132)
        at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:113)
        at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:87)
        at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:87)
        at org.apache.spark.sql.Dataset.<init>(Dataset.scala:185)
        at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:64)
        at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:592)
        at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:699)
        at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute(SparkExecuteStatementOperation.scala:220)
        at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$1$$anon$2.run(SparkExecuteStatementOperation.scala:163)
        at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$1$$anon$2.run(SparkExecuteStatementOperation.scala:160)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
        at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$1.run(SparkExecuteStatementOperation.scala:173)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
17/08/30 15:12:31 ERROR SparkExecuteStatementOperation: Error running hive query:
org.apache.hive.service.cli.HiveSQLException: java.lang.RuntimeException: Compaction failed. Please check logs for more info. Exception in compaction java.lang.Exception: Compaction Failure in Merger Rdd.
        at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute(SparkExecuteStatementOperation.scala:258)
        at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$1$$anon$2.run(SparkExecuteStatementOperation.scala:163)
        at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$1$$anon$2.run(SparkExecuteStatementOperation.scala:160)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
        at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$1.run(SparkExecuteStatementOperation.scala:173)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)





> Major and Minor Compaction Failing
> ----------------------------------
>
>                 Key: CARBONDATA-1422
>                 URL: https://issues.apache.org/jira/browse/CARBONDATA-1422
>             Project: CarbonData
>          Issue Type: Bug
>    Affects Versions: 1.2.0
>            Reporter: Pallavi Singh
>             Fix For: 1.2.0
>
>
> The major and minor compaction is failing.
> The Compaction is failing in the default scenario where the table property dictionary_include is not specified which is the default behaviour.
> Please find the error logs below :
> 0: jdbc:hive2://localhost:10000> show segments for table uniqdata;
> +--------------------+----------+--------------------------+--------------------------+--+
> | SegmentSequenceId  |  Status  |     Load Start Time      |      Load End Time       |
> +--------------------+----------+--------------------------+--------------------------+--+
> | 9                  | Success  | 2017-08-29 11:17:29.927  | 2017-08-29 11:17:30.555  |
> | 8                  | Success  | 2017-08-29 11:17:27.572  | 2017-08-29 11:17:28.363  |
> | 7                  | Success  | 2017-08-29 11:17:23.583  | 2017-08-29 11:17:25.112  |
> | 6                  | Success  | 2017-08-29 11:17:07.966  | 2017-08-29 11:17:09.322  |
> | 5                  | Success  | 2017-08-29 10:38:15.727  | 2017-08-29 10:38:16.548  |
> | 4                  | Success  | 2017-08-29 10:37:13.053  | 2017-08-29 10:37:13.888  |
> | 3                  | Success  | 2017-08-29 10:36:57.851  | 2017-08-29 10:36:59.08   |
> | 2                  | Success  | 2017-08-29 10:36:49.439  | 2017-08-29 10:36:50.373  |
> | 1                  | Success  | 2017-08-29 10:36:37.365  | 2017-08-29 10:36:38.768  |
> | 0                  | Success  | 2017-08-29 10:36:21.011  | 2017-08-29 10:36:26.1    |
> +--------------------+----------+--------------------------+--------------------------+--+
> 10 rows selected (0.081 seconds)
> 0: jdbc:hive2://localhost:10000> ALTER TABLE uniqdata COMPACT 'MINOR';
> Error: java.lang.RuntimeException: Compaction failed. Please check logs for more info. Exception in compaction java.lang.Exception: Compaction Failure in Merger Rdd. (state=,code=0)
> 0: jdbc:hive2://localhost:10000> ALTER TABLE uniqdata COMPACT 'MAJOR';
> Error: java.lang.RuntimeException: Compaction failed. Please check logs for more info. Exception in compaction java.lang.Exception: Compaction Failure in Merger Rdd. (state=,code=0)
> 0: jdbc:hive2://localhost:10000>



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
Free forum by Nabble Edit this page