query err 'NullPointerException' but fine after table cached in memory
Locked
 
|
Hello,
use carbondata 0.2.0, following is the problem: Only one column 'store_id' throws NullPointerException when query, but it works fine when some value or table is cached in memory. store_id's type is int, cardinality is 200 Thousand, is configured about dictionary and inverted index. sql: select order_code,saletype,checkout_date,cashier_code,item_cont,invoice_price,giveamt,saleamt from store.sale where store_id=299998 error: ERROR 12-01 10:40:16,861 - [Executor task launch worker-0][partitionID:sale;queryID:1438806645368420_0] java.lang.NullPointerException at org.apache.carbondata.scan.result.iterator.AbstractDetailQueryResultIterator.intialiseInfos(AbstractDetailQueryResultIterator.java:117) at org.apache.carbondata.scan.result.iterator.AbstractDetailQueryResultIterator.<init>(AbstractDetailQueryResultIterator.java:107) at org.apache.carbondata.scan.result.iterator.DetailQueryResultIterator.<init>(DetailQueryResultIterator.java:43) at org.apache.carbondata.scan.executor.impl.DetailQueryExecutor.execute(DetailQueryExecutor.java:39) at org.apache.carbondata.spark.rdd.CarbonScanRDD$$anon$1.<init>(CarbonScanRDD.scala:216) at org.apache.carbondata.spark.rdd.CarbonScanRDD.compute(CarbonScanRDD.scala:192) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) at org.apache.spark.scheduler.Task.run(Task.scala:89) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) ERROR 12-01 10:40:16,874 - Exception in task 0.1 in stage 0.0 (TID 1) java.lang.RuntimeException: Exception occurred in query execution.Please check logs. at scala.sys.package$.error(package.scala:27) at org.apache.carbondata.spark.rdd.CarbonScanRDD$$anon$1.<init>(CarbonScanRDD.scala:226) at org.apache.carbondata.spark.rdd.CarbonScanRDD.compute(CarbonScanRDD.scala:192) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) at org.apache.spark.scheduler.Task.run(Task.scala:89) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) ------------------------------------------------------------------------------------------------------------------------------- Same err 'NullPointerException' in following sql: select * from store.sale where store_id=100000 select * from store.sale where store_id=100001 select * from store.sale where store_id=100002 select * from store.sale where store_id=100006 select * from store.sale where store_id=100011 select * from store.sale where store_id=299999 But fine and can return results in following sql: select * from store.sale where store_id=100008 select * from store.sale where store_id=100009 select * from store.sale where store_id=100010 select * from store.sale where store_id=100013 select * from store.sale where store_id=100027 select * from store.sale limit 10 select count(*) from store.sale select * from store.sale where store_id=100005 select count(*) from store.sale where store_id=100005 select distinct(store_id) from store.sale order by store_id But all work fine when table is cached to memory, not throwing 'NullPointerException'. |
Re: query err 'NullPointerException' but fine after table cached in memory
|
|
Hi,
Can u please share executor log. -Regards Kumar Vishal On Thu, Jan 12, 2017 at 1:59 PM, Li Peng <[hidden email]> wrote: > Hello, > > use carbondata 0.2.0, following is the problem: > > Only one column 'store_id' throws NullPointerException when query, but it > works fine when some value or table is cached in memory. > > store_id's type is int, cardinality is 200 Thousand, is configured about > dictionary and inverted index. > > sql: > select > order_code,saletype,checkout_date,cashier_code,item_cont, > invoice_price,giveamt,saleamt > from store.sale where store_id=299998 > > error: > ERROR 12-01 10:40:16,861 - [Executor task launch > worker-0][partitionID:sale;queryID:1438806645368420_0] > java.lang.NullPointerException > at > org.apache.carbondata.scan.result.iterator.AbstractDetailQueryResultItera > tor.intialiseInfos(AbstractDetailQueryResultIterator.java:117) > at > org.apache.carbondata.scan.result.iterator.AbstractDetailQueryResultItera > tor.<init>(AbstractDetailQueryResultIterator.java:107) > at > org.apache.carbondata.scan.result.iterator.DetailQueryResultIterator.< > init>(DetailQueryResultIterator.java:43) > at > org.apache.carbondata.scan.executor.impl.DetailQueryExecutor.execute( > DetailQueryExecutor.java:39) > at > org.apache.carbondata.spark.rdd.CarbonScanRDD$$anon$1.< > init>(CarbonScanRDD.scala:216) > at > org.apache.carbondata.spark.rdd.CarbonScanRDD.compute( > CarbonScanRDD.scala:192) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) > at org.apache.spark.rdd.MapPartitionsRDD.compute( > MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) > at org.apache.spark.rdd.MapPartitionsRDD.compute( > MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) > at org.apache.spark.rdd.MapPartitionsRDD.compute( > MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) > at org.apache.spark.rdd.MapPartitionsRDD.compute( > MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask. > scala:66) > at org.apache.spark.scheduler.Task.run(Task.scala:89) > at org.apache.spark.executor.Executor$TaskRunner.run( > Executor.scala:227) > at > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > ERROR 12-01 10:40:16,874 - Exception in task 0.1 in stage 0.0 (TID 1) > java.lang.RuntimeException: Exception occurred in query execution.Please > check logs. > at scala.sys.package$.error(package.scala:27) > at > org.apache.carbondata.spark.rdd.CarbonScanRDD$$anon$1.< > init>(CarbonScanRDD.scala:226) > at > org.apache.carbondata.spark.rdd.CarbonScanRDD.compute( > CarbonScanRDD.scala:192) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) > at org.apache.spark.rdd.MapPartitionsRDD.compute( > MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) > at org.apache.spark.rdd.MapPartitionsRDD.compute( > MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) > at org.apache.spark.rdd.MapPartitionsRDD.compute( > MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) > at org.apache.spark.rdd.MapPartitionsRDD.compute( > MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask. > scala:66) > at org.apache.spark.scheduler.Task.run(Task.scala:89) > at org.apache.spark.executor.Executor$TaskRunner.run( > Executor.scala:227) > at > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > ------------------------------------------------------------ > ------------------------------------------------------------------- > Same err 'NullPointerException' in following sql: > > select * from store.sale where store_id=100000 > select * from store.sale where store_id=100001 > select * from store.sale where store_id=100002 > select * from store.sale where store_id=100006 > select * from store.sale where store_id=100011 > select * from store.sale where store_id=299999 > > But fine and can return results in following sql: > > select * from store.sale where store_id=100008 > select * from store.sale where store_id=100009 > select * from store.sale where store_id=100010 > select * from store.sale where store_id=100013 > select * from store.sale where store_id=100027 > select * from store.sale limit 10 > select count(*) from store.sale > select * from store.sale where store_id=100005 > select count(*) from store.sale where store_id=100005 > select distinct(store_id) from store.sale order by store_id > > > But all work fine when table is cached to memory, not throwing > 'NullPointerException'. > > > > > -- > View this message in context: http://apache-carbondata- > mailing-list-archive.1130556.n5.nabble.com/query-err- > NullPointerException-but-fine-after-table-cached-in-memory-tp6032.html > Sent from the Apache CarbonData Mailing List archive mailing list archive > at Nabble.com. >
kumar vishal
|
|
|
Hi,
Executor log as follows: INFO 12-01 15:45:58,574 - Registered signal handlers for [TERM, HUP, INT] INFO 12-01 15:45:59,380 - Changing view acls to: yarn,spark INFO 12-01 15:45:59,387 - Changing modify acls to: yarn,spark INFO 12-01 15:45:59,387 - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, spark); users with modify permissions: Set(yarn, spark) INFO 12-01 15:46:00,422 - Changing view acls to: yarn,spark INFO 12-01 15:46:00,422 - Changing modify acls to: yarn,spark INFO 12-01 15:46:00,422 - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, spark); users with modify permissions: Set(yarn, spark) INFO 12-01 15:46:00,825 - Slf4jLogger started INFO 12-01 15:46:00,873 - Starting remoting INFO 12-01 15:46:01,046 - Remoting started; listening on addresses :[akka.tcp://sparkExecutorActorSystem@dpnode03:42230] INFO 12-01 15:46:01,051 - Successfully started service 'sparkExecutorActorSystem' on port 42230. INFO 12-01 15:46:01,117 - Created local directory at /data02/hadoop/yarn/local/usercache/spark/appcache/application_1484187732071_0003/blockmgr-f1549267-39fd-4f92-a793-db3975c05272 INFO 12-01 15:46:01,117 - Created local directory at /data03/hadoop/yarn/local/usercache/spark/appcache/application_1484187732071_0003/blockmgr-e1f964d4-877c-45e2-88a0-3d9d14ebf47e INFO 12-01 15:46:01,117 - Created local directory at /data04/hadoop/yarn/local/usercache/spark/appcache/application_1484187732071_0003/blockmgr-63faa7b7-248b-424a-98bb-ab85d9aef0c3 INFO 12-01 15:46:01,118 - Created local directory at /data05/hadoop/yarn/local/usercache/spark/appcache/application_1484187732071_0003/blockmgr-e1ac73f4-54e0-4239-933f-2f16f6997a49 INFO 12-01 15:46:01,118 - Created local directory at /data06/hadoop/yarn/local/usercache/spark/appcache/application_1484187732071_0003/blockmgr-5b8bf328-594a-419b-af4d-d34a8a350dce INFO 12-01 15:46:01,118 - Created local directory at /data07/hadoop/yarn/local/usercache/spark/appcache/application_1484187732071_0003/blockmgr-6eed5405-1b71-4d70-931a-3c68a8bff14a INFO 12-01 15:46:01,119 - Created local directory at /data08/hadoop/yarn/local/usercache/spark/appcache/application_1484187732071_0003/blockmgr-5dfcf5a3-d06c-4ae7-a2ec-1a41eb709a26 INFO 12-01 15:46:01,119 - Created local directory at /data09/hadoop/yarn/local/usercache/spark/appcache/application_1484187732071_0003/blockmgr-8ed5f4f3-6c92-4f46-adea-362324bed778 INFO 12-01 15:46:01,119 - Created local directory at /data10/hadoop/yarn/local/usercache/spark/appcache/application_1484187732071_0003/blockmgr-c7367242-4a68-4b45-9a13-cffd4793b46e INFO 12-01 15:46:01,119 - Created local directory at /data11/hadoop/yarn/local/usercache/spark/appcache/application_1484187732071_0003/blockmgr-2e25390e-835e-4f1e-bca5-96b6ecdc7247 INFO 12-01 15:46:01,120 - Created local directory at /data12/hadoop/yarn/local/usercache/spark/appcache/application_1484187732071_0003/blockmgr-c0911b1e-9c85-4393-bdab-db35a772b4fb INFO 12-01 15:46:01,141 - MemoryStore started with capacity 5.5 GB INFO 12-01 15:46:01,632 - Connecting to driver: spark://CoarseGrainedScheduler@192.168.50.8:39206 INFO 12-01 15:46:01,681 - Successfully registered with driver INFO 12-01 15:46:01,684 - Starting executor ID 2 on host dpnode03 INFO 12-01 15:46:01,722 - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 47403. INFO 12-01 15:46:01,723 - Server created on 47403 INFO 12-01 15:46:01,728 - external shuffle service port = 7337 INFO 12-01 15:46:01,728 - Trying to register BlockManager INFO 12-01 15:46:01,743 - Registered BlockManager INFO 12-01 15:46:01,747 - Registering executor with local external shuffle service. INFO 12-01 15:46:01,778 - Using REPL class URI: http://192.168.50.8:44119 INFO 12-01 15:49:19,059 - Got assigned task 0 INFO 12-01 15:49:19,066 - Running task 0.0 in stage 0.0 (TID 0) INFO 12-01 15:49:19,270 - Started reading broadcast variable 0 INFO 12-01 15:49:19,442 - Block broadcast_0_piece0 stored as bytes in memory (estimated size 18.8 KB, free 18.8 KB) INFO 12-01 15:49:19,468 - Reading broadcast variable 0 took 198 ms INFO 12-01 15:49:19,616 - Block broadcast_0 stored as values in memory (estimated size 35.2 KB, free 54.0 KB) INFO 12-01 15:49:21,375 - *************************/usr/hdp/2.5.0.0-1245/spark/conf/carbon.properties INFO 12-01 15:49:21,376 - [Executor task launch worker-0][partitionID:sale;queryID:1457355617520266_0] Query will be executed on table: sale INFO 12-01 15:49:21,379 - [Executor task launch worker-0][partitionID:sale;queryID:1457355617520266_0] Property file path: /usr/hdp/2.5.0.0-1245/spark/conf/carbon.properties INFO 12-01 15:49:21,379 - [Executor task launch worker-0][partitionID:sale;queryID:1457355617520266_0] ------Using Carbon.properties -------- INFO 12-01 15:49:21,379 - [Executor task launch worker-0][partitionID:sale;queryID:1457355617520266_0] {carbon.number.of.cores.while.loading=6, carbon.number.of.cores.while.compacting=4, carbon.sort.file.buffer.size=20, carbon.inmemory.record.size=120000, carbon.sort.size=500000, carbon.graph.rowset.size=100000, carbon.ddl.base.hdfs.url=/user/spark, carbon.compaction.level.threshold=8,6, carbon.number.of.cores=4, carbon.kettle.home=/usr/hdp/2.5.0.0-1245/spark/carbonlib/carbonplugins, carbon.storelocation=hdfs://julong/carbondata/carbonstore, carbon.enable.auto.load.merge=true, carbon.enableXXHash=true, carbon.sort.intermediate.files.limit=100, carbon.major.compaction.size=1024, carbon.badRecords.location=/opt/Carbon/Spark/badrecords, carbon.use.local.dir=true, carbon.enable.quick.filter=false} INFO 12-01 15:49:23,999 - pool-19-thread-1 ****************************Total Number Rows In BTREE: 10380500 INFO 12-01 15:49:24,005 - [Executor task launch worker-0][partitionID:sale;queryID:1457355617520266_0] Column cache size not configured. Therefore default behavior will be considered and no LRU based eviction of columns will be done ERROR 12-01 15:49:24,565 - [Executor task launch worker-0][partitionID:sale;queryID:1457355617520266_0] java.lang.NullPointerException at org.apache.carbondata.scan.result.iterator.AbstractDetailQueryResultIterator.intialiseInfos(AbstractDetailQueryResultIterator.java:117) at org.apache.carbondata.scan.result.iterator.AbstractDetailQueryResultIterator.<init>(AbstractDetailQueryResultIterator.java:107) at org.apache.carbondata.scan.result.iterator.DetailQueryResultIterator.<init>(DetailQueryResultIterator.java:43) at org.apache.carbondata.scan.executor.impl.DetailQueryExecutor.execute(DetailQueryExecutor.java:39) at org.apache.carbondata.spark.rdd.CarbonScanRDD$$anon$1.<init>(CarbonScanRDD.scala:216) at org.apache.carbondata.spark.rdd.CarbonScanRDD.compute(CarbonScanRDD.scala:192) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) at org.apache.spark.scheduler.Task.run(Task.scala:89) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) ERROR 12-01 15:49:24,577 - Exception in task 0.0 in stage 0.0 (TID 0) java.lang.RuntimeException: Exception occurred in query execution.Please check logs. at scala.sys.package$.error(package.scala:27) at org.apache.carbondata.spark.rdd.CarbonScanRDD$$anon$1.<init>(CarbonScanRDD.scala:226) at org.apache.carbondata.spark.rdd.CarbonScanRDD.compute(CarbonScanRDD.scala:192) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) at org.apache.spark.scheduler.Task.run(Task.scala:89) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) INFO 12-01 15:49:34,727 - Got assigned task 3 INFO 12-01 15:49:34,727 - Running task 0.3 in stage 0.0 (TID 3) INFO 12-01 15:49:34,799 - *************************/usr/hdp/2.5.0.0-1245/spark/conf/carbon.properties INFO 12-01 15:49:34,799 - [Executor task launch worker-0][partitionID:sale;queryID:1457355617520266_0] Query will be executed on table: sale ERROR 12-01 15:49:34,813 - [Executor task launch worker-0][partitionID:sale;queryID:1457355617520266_0] java.lang.NullPointerException at org.apache.carbondata.scan.result.iterator.AbstractDetailQueryResultIterator.intialiseInfos(AbstractDetailQueryResultIterator.java:117) at org.apache.carbondata.scan.result.iterator.AbstractDetailQueryResultIterator.<init>(AbstractDetailQueryResultIterator.java:107) at org.apache.carbondata.scan.result.iterator.DetailQueryResultIterator.<init>(DetailQueryResultIterator.java:43) at org.apache.carbondata.scan.executor.impl.DetailQueryExecutor.execute(DetailQueryExecutor.java:39) at org.apache.carbondata.spark.rdd.CarbonScanRDD$$anon$1.<init>(CarbonScanRDD.scala:216) at org.apache.carbondata.spark.rdd.CarbonScanRDD.compute(CarbonScanRDD.scala:192) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) at org.apache.spark.scheduler.Task.run(Task.scala:89) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) ERROR 12-01 15:49:34,816 - Exception in task 0.3 in stage 0.0 (TID 3) java.lang.RuntimeException: Exception occurred in query execution.Please check logs. at scala.sys.package$.error(package.scala:27) at org.apache.carbondata.spark.rdd.CarbonScanRDD$$anon$1.<init>(CarbonScanRDD.scala:226) at org.apache.carbondata.spark.rdd.CarbonScanRDD.compute(CarbonScanRDD.scala:192) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) at org.apache.spark.scheduler.Task.run(Task.scala:89) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Thanks.
|
Re: query err 'NullPointerException' but fine after table cached in memory
|
|
Hi Le Peng,
This is an issue which got fixed after 0.2 release. Please take latest 0.2 branch from git and compile and try. -Regards Kumar Vishal On Thu, Jan 12, 2017 at 4:14 PM, Li Peng <[hidden email]> wrote: > Hi, > Executor log as follows: > > INFO 12-01 15:45:58,574 - Registered signal handlers for [TERM, HUP, INT] > INFO 12-01 15:45:59,380 - Changing view acls to: yarn,spark > INFO 12-01 15:45:59,387 - Changing modify acls to: yarn,spark > INFO 12-01 15:45:59,387 - SecurityManager: authentication disabled; ui > acls > disabled; users with view permissions: Set(yarn, spark); users with modify > permissions: Set(yarn, spark) > INFO 12-01 15:46:00,422 - Changing view acls to: yarn,spark > INFO 12-01 15:46:00,422 - Changing modify acls to: yarn,spark > INFO 12-01 15:46:00,422 - SecurityManager: authentication disabled; ui > acls > disabled; users with view permissions: Set(yarn, spark); users with modify > permissions: Set(yarn, spark) > INFO 12-01 15:46:00,825 - Slf4jLogger started > INFO 12-01 15:46:00,873 - Starting remoting > INFO 12-01 15:46:01,046 - Remoting started; listening on addresses > :[akka.tcp://sparkExecutorActorSystem@dpnode03:42230] > INFO 12-01 15:46:01,051 - Successfully started service > 'sparkExecutorActorSystem' on port 42230. > INFO 12-01 15:46:01,117 - Created local directory at > /data02/hadoop/yarn/local/usercache/spark/appcache/ > application_1484187732071_0003/blockmgr-f1549267-39fd- > 4f92-a793-db3975c05272 > INFO 12-01 15:46:01,117 - Created local directory at > /data03/hadoop/yarn/local/usercache/spark/appcache/ > application_1484187732071_0003/blockmgr-e1f964d4-877c- > 45e2-88a0-3d9d14ebf47e > INFO 12-01 15:46:01,117 - Created local directory at > /data04/hadoop/yarn/local/usercache/spark/appcache/ > application_1484187732071_0003/blockmgr-63faa7b7-248b- > 424a-98bb-ab85d9aef0c3 > INFO 12-01 15:46:01,118 - Created local directory at > /data05/hadoop/yarn/local/usercache/spark/appcache/ > application_1484187732071_0003/blockmgr-e1ac73f4-54e0- > 4239-933f-2f16f6997a49 > INFO 12-01 15:46:01,118 - Created local directory at > /data06/hadoop/yarn/local/usercache/spark/appcache/ > application_1484187732071_0003/blockmgr-5b8bf328-594a- > 419b-af4d-d34a8a350dce > INFO 12-01 15:46:01,118 - Created local directory at > /data07/hadoop/yarn/local/usercache/spark/appcache/ > application_1484187732071_0003/blockmgr-6eed5405-1b71- > 4d70-931a-3c68a8bff14a > INFO 12-01 15:46:01,119 - Created local directory at > /data08/hadoop/yarn/local/usercache/spark/appcache/ > application_1484187732071_0003/blockmgr-5dfcf5a3-d06c- > 4ae7-a2ec-1a41eb709a26 > INFO 12-01 15:46:01,119 - Created local directory at > /data09/hadoop/yarn/local/usercache/spark/appcache/ > application_1484187732071_0003/blockmgr-8ed5f4f3-6c92- > 4f46-adea-362324bed778 > INFO 12-01 15:46:01,119 - Created local directory at > /data10/hadoop/yarn/local/usercache/spark/appcache/ > application_1484187732071_0003/blockmgr-c7367242-4a68- > 4b45-9a13-cffd4793b46e > INFO 12-01 15:46:01,119 - Created local directory at > /data11/hadoop/yarn/local/usercache/spark/appcache/ > application_1484187732071_0003/blockmgr-2e25390e-835e- > 4f1e-bca5-96b6ecdc7247 > INFO 12-01 15:46:01,120 - Created local directory at > /data12/hadoop/yarn/local/usercache/spark/appcache/ > application_1484187732071_0003/blockmgr-c0911b1e-9c85- > 4393-bdab-db35a772b4fb > INFO 12-01 15:46:01,141 - MemoryStore started with capacity 5.5 GB > INFO 12-01 15:46:01,632 - Connecting to driver: > spark://CoarseGrainedScheduler@192.168.50.8:39206 > INFO 12-01 15:46:01,681 - Successfully registered with driver > INFO 12-01 15:46:01,684 - Starting executor ID 2 on host dpnode03 > INFO 12-01 15:46:01,722 - Successfully started service > 'org.apache.spark.network.netty.NettyBlockTransferService' on port 47403. > INFO 12-01 15:46:01,723 - Server created on 47403 > INFO 12-01 15:46:01,728 - external shuffle service port = 7337 > INFO 12-01 15:46:01,728 - Trying to register BlockManager > INFO 12-01 15:46:01,743 - Registered BlockManager > INFO 12-01 15:46:01,747 - Registering executor with local external shuffle > service. > INFO 12-01 15:46:01,778 - Using REPL class URI: http://192.168.50.8:44119 > INFO 12-01 15:49:19,059 - Got assigned task 0 > INFO 12-01 15:49:19,066 - Running task 0.0 in stage 0.0 (TID 0) > INFO 12-01 15:49:19,270 - Started reading broadcast variable 0 > INFO 12-01 15:49:19,442 - Block broadcast_0_piece0 stored as bytes in > memory (estimated size 18.8 KB, free 18.8 KB) > INFO 12-01 15:49:19,468 - Reading broadcast variable 0 took 198 ms > INFO 12-01 15:49:19,616 - Block broadcast_0 stored as values in memory > (estimated size 35.2 KB, free 54.0 KB) > INFO 12-01 15:49:21,375 - > *************************/usr/hdp/2.5.0.0-1245/spark/conf/ > carbon.properties > INFO 12-01 15:49:21,376 - [Executor task launch > worker-0][partitionID:sale;queryID:1457355617520266_0] Query will be > executed on table: sale > INFO 12-01 15:49:21,379 - [Executor task launch > worker-0][partitionID:sale;queryID:1457355617520266_0] Property file path: > /usr/hdp/2.5.0.0-1245/spark/conf/carbon.properties > INFO 12-01 15:49:21,379 - [Executor task launch > worker-0][partitionID:sale;queryID:1457355617520266_0] ------Using > Carbon.properties -------- > INFO 12-01 15:49:21,379 - [Executor task launch > worker-0][partitionID:sale;queryID:1457355617520266_0] > {carbon.number.of.cores.while.loading=6, > carbon.number.of.cores.while.compacting=4, carbon.sort.file.buffer.size= > 20, > carbon.inmemory.record.size=120000, carbon.sort.size=500000, > carbon.graph.rowset.size=100000, carbon.ddl.base.hdfs.url=/user/spark, > carbon.compaction.level.threshold=8,6, carbon.number.of.cores=4, > carbon.kettle.home=/usr/hdp/2.5.0.0-1245/spark/carbonlib/carbonplugins, > carbon.storelocation=hdfs://julong/carbondata/carbonstore, > carbon.enable.auto.load.merge=true, carbon.enableXXHash=true, > carbon.sort.intermediate.files.limit=100, carbon.major.compaction.size= > 1024, > carbon.badRecords.location=/opt/Carbon/Spark/badrecords, > carbon.use.local.dir=true, carbon.enable.quick.filter=false} > INFO 12-01 15:49:23,999 - pool-19-thread-1 > ****************************Total Number Rows In BTREE: 10380500 > INFO 12-01 15:49:24,005 - [Executor task launch > worker-0][partitionID:sale;queryID:1457355617520266_0] Column cache size > not > configured. Therefore default behavior will be considered and no LRU based > eviction of columns will be done > ERROR 12-01 15:49:24,565 - [Executor task launch > worker-0][partitionID:sale;queryID:1457355617520266_0] > java.lang.NullPointerException > at > org.apache.carbondata.scan.result.iterator.AbstractDetailQueryResultItera > tor.intialiseInfos(AbstractDetailQueryResultIterator.java:117) > at > org.apache.carbondata.scan.result.iterator.AbstractDetailQueryResultItera > tor.<init>(AbstractDetailQueryResultIterator.java:107) > at > org.apache.carbondata.scan.result.iterator.DetailQueryResultIterator.< > init>(DetailQueryResultIterator.java:43) > at > org.apache.carbondata.scan.executor.impl.DetailQueryExecutor.execute( > DetailQueryExecutor.java:39) > at > org.apache.carbondata.spark.rdd.CarbonScanRDD$$anon$1.< > init>(CarbonScanRDD.scala:216) > at > org.apache.carbondata.spark.rdd.CarbonScanRDD.compute( > CarbonScanRDD.scala:192) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) > at org.apache.spark.rdd.MapPartitionsRDD.compute( > MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) > at org.apache.spark.rdd.MapPartitionsRDD.compute( > MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) > at org.apache.spark.rdd.MapPartitionsRDD.compute( > MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) > at org.apache.spark.rdd.MapPartitionsRDD.compute( > MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) > at org.apache.spark.rdd.MapPartitionsRDD.compute( > MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask. > scala:66) > at org.apache.spark.scheduler.Task.run(Task.scala:89) > at org.apache.spark.executor.Executor$TaskRunner.run( > Executor.scala:227) > at > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > ERROR 12-01 15:49:24,577 - Exception in task 0.0 in stage 0.0 (TID 0) > java.lang.RuntimeException: Exception occurred in query execution.Please > check logs. > at scala.sys.package$.error(package.scala:27) > at > org.apache.carbondata.spark.rdd.CarbonScanRDD$$anon$1.< > init>(CarbonScanRDD.scala:226) > at > org.apache.carbondata.spark.rdd.CarbonScanRDD.compute( > CarbonScanRDD.scala:192) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) > at org.apache.spark.rdd.MapPartitionsRDD.compute( > MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) > at org.apache.spark.rdd.MapPartitionsRDD.compute( > MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) > at org.apache.spark.rdd.MapPartitionsRDD.compute( > MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) > at org.apache.spark.rdd.MapPartitionsRDD.compute( > MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) > at org.apache.spark.rdd.MapPartitionsRDD.compute( > MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask. > scala:66) > at org.apache.spark.scheduler.Task.run(Task.scala:89) > at org.apache.spark.executor.Executor$TaskRunner.run( > Executor.scala:227) > at > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > INFO 12-01 15:49:34,727 - Got assigned task 3 > INFO 12-01 15:49:34,727 - Running task 0.3 in stage 0.0 (TID 3) > INFO 12-01 15:49:34,799 - > *************************/usr/hdp/2.5.0.0-1245/spark/conf/ > carbon.properties > INFO 12-01 15:49:34,799 - [Executor task launch > worker-0][partitionID:sale;queryID:1457355617520266_0] Query will be > executed on table: sale > ERROR 12-01 15:49:34,813 - [Executor task launch > worker-0][partitionID:sale;queryID:1457355617520266_0] > java.lang.NullPointerException > at > org.apache.carbondata.scan.result.iterator.AbstractDetailQueryResultItera > tor.intialiseInfos(AbstractDetailQueryResultIterator.java:117) > at > org.apache.carbondata.scan.result.iterator.AbstractDetailQueryResultItera > tor.<init>(AbstractDetailQueryResultIterator.java:107) > at > org.apache.carbondata.scan.result.iterator.DetailQueryResultIterator.< > init>(DetailQueryResultIterator.java:43) > at > org.apache.carbondata.scan.executor.impl.DetailQueryExecutor.execute( > DetailQueryExecutor.java:39) > at > org.apache.carbondata.spark.rdd.CarbonScanRDD$$anon$1.< > init>(CarbonScanRDD.scala:216) > at > org.apache.carbondata.spark.rdd.CarbonScanRDD.compute( > CarbonScanRDD.scala:192) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) > at org.apache.spark.rdd.MapPartitionsRDD.compute( > MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) > at org.apache.spark.rdd.MapPartitionsRDD.compute( > MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) > at org.apache.spark.rdd.MapPartitionsRDD.compute( > MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) > at org.apache.spark.rdd.MapPartitionsRDD.compute( > MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) > at org.apache.spark.rdd.MapPartitionsRDD.compute( > MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask. > scala:66) > at org.apache.spark.scheduler.Task.run(Task.scala:89) > at org.apache.spark.executor.Executor$TaskRunner.run( > Executor.scala:227) > at > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > ERROR 12-01 15:49:34,816 - Exception in task 0.3 in stage 0.0 (TID 3) > java.lang.RuntimeException: Exception occurred in query execution.Please > check logs. > at scala.sys.package$.error(package.scala:27) > at > org.apache.carbondata.spark.rdd.CarbonScanRDD$$anon$1.< > init>(CarbonScanRDD.scala:226) > at > org.apache.carbondata.spark.rdd.CarbonScanRDD.compute( > CarbonScanRDD.scala:192) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) > at org.apache.spark.rdd.MapPartitionsRDD.compute( > MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) > at org.apache.spark.rdd.MapPartitionsRDD.compute( > MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) > at org.apache.spark.rdd.MapPartitionsRDD.compute( > MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) > at org.apache.spark.rdd.MapPartitionsRDD.compute( > MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) > at org.apache.spark.rdd.MapPartitionsRDD.compute( > MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask. > scala:66) > at org.apache.spark.scheduler.Task.run(Task.scala:89) > at org.apache.spark.executor.Executor$TaskRunner.run( > Executor.scala:227) > at > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > > Thanks. > > > > > > kumarvishal09 wrote > > Hi, > > Can u please share executor log. > > > > -Regards > > Kumar Vishal > > > > On Thu, Jan 12, 2017 at 1:59 PM, Li Peng < > > > pengli0606@ > > > > wrote: > > > >> Hello, > >> > >> use carbondata 0.2.0, following is the problem: > >> > >> Only one column 'store_id' throws NullPointerException when query, but > >> it > >> works fine when some value or table is cached in memory. > >> > >> store_id's type is int, cardinality is 200 Thousand, is configured > about > >> dictionary and inverted index. > >> > >> sql: > >> select > >> order_code,saletype,checkout_date,cashier_code,item_cont, > >> invoice_price,giveamt,saleamt > >> from store.sale where store_id=299998 > >> > >> error: > >> ERROR 12-01 10:40:16,861 - [Executor task launch > >> worker-0][partitionID:sale;queryID:1438806645368420_0] > >> java.lang.NullPointerException > >> at > >> org.apache.carbondata.scan.result.iterator. > AbstractDetailQueryResultItera > >> tor.intialiseInfos(AbstractDetailQueryResultIterator.java:117) > >> at > >> org.apache.carbondata.scan.result.iterator. > AbstractDetailQueryResultItera > >> tor. > > <init> > > (AbstractDetailQueryResultIterator.java:107) > >> at > >> org.apache.carbondata.scan.result.iterator.DetailQueryResultIterator.< > >> init>(DetailQueryResultIterator.java:43) > >> at > >> org.apache.carbondata.scan.executor.impl.DetailQueryExecutor.execute( > >> DetailQueryExecutor.java:39) > >> at > >> org.apache.carbondata.spark.rdd.CarbonScanRDD$$anon$1.< > >> init>(CarbonScanRDD.scala:216) > >> at > >> org.apache.carbondata.spark.rdd.CarbonScanRDD.compute( > >> CarbonScanRDD.scala:192) > >> at > >> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) > >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) > >> at org.apache.spark.rdd.MapPartitionsRDD.compute( > >> MapPartitionsRDD.scala:38) > >> at > >> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) > >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) > >> at org.apache.spark.rdd.MapPartitionsRDD.compute( > >> MapPartitionsRDD.scala:38) > >> at > >> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) > >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) > >> at org.apache.spark.rdd.MapPartitionsRDD.compute( > >> MapPartitionsRDD.scala:38) > >> at > >> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) > >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) > >> at org.apache.spark.rdd.MapPartitionsRDD.compute( > >> MapPartitionsRDD.scala:38) > >> at > >> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) > >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) > >> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask. > >> scala:66) > >> at org.apache.spark.scheduler.Task.run(Task.scala:89) > >> at org.apache.spark.executor.Executor$TaskRunner.run( > >> Executor.scala:227) > >> at > >> java.util.concurrent.ThreadPoolExecutor.runWorker( > >> ThreadPoolExecutor.java:1145) > >> at > >> java.util.concurrent.ThreadPoolExecutor$Worker.run( > >> ThreadPoolExecutor.java:615) > >> at java.lang.Thread.run(Thread.java:745) > >> ERROR 12-01 10:40:16,874 - Exception in task 0.1 in stage 0.0 (TID 1) > >> java.lang.RuntimeException: Exception occurred in query execution.Please > >> check logs. > >> at scala.sys.package$.error(package.scala:27) > >> at > >> org.apache.carbondata.spark.rdd.CarbonScanRDD$$anon$1.< > >> init>(CarbonScanRDD.scala:226) > >> at > >> org.apache.carbondata.spark.rdd.CarbonScanRDD.compute( > >> CarbonScanRDD.scala:192) > >> at > >> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) > >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) > >> at org.apache.spark.rdd.MapPartitionsRDD.compute( > >> MapPartitionsRDD.scala:38) > >> at > >> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) > >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) > >> at org.apache.spark.rdd.MapPartitionsRDD.compute( > >> MapPartitionsRDD.scala:38) > >> at > >> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) > >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) > >> at org.apache.spark.rdd.MapPartitionsRDD.compute( > >> MapPartitionsRDD.scala:38) > >> at > >> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) > >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) > >> at org.apache.spark.rdd.MapPartitionsRDD.compute( > >> MapPartitionsRDD.scala:38) > >> at > >> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313) > >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:277) > >> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask. > >> scala:66) > >> at org.apache.spark.scheduler.Task.run(Task.scala:89) > >> at org.apache.spark.executor.Executor$TaskRunner.run( > >> Executor.scala:227) > >> at > >> java.util.concurrent.ThreadPoolExecutor.runWorker( > >> ThreadPoolExecutor.java:1145) > >> at > >> java.util.concurrent.ThreadPoolExecutor$Worker.run( > >> ThreadPoolExecutor.java:615) > >> at java.lang.Thread.run(Thread.java:745) > >> ------------------------------------------------------------ > >> ------------------------------------------------------------------- > >> Same err 'NullPointerException' in following sql: > >> > >> select * from store.sale where store_id=100000 > >> select * from store.sale where store_id=100001 > >> select * from store.sale where store_id=100002 > >> select * from store.sale where store_id=100006 > >> select * from store.sale where store_id=100011 > >> select * from store.sale where store_id=299999 > >> > >> But fine and can return results in following sql: > >> > >> select * from store.sale where store_id=100008 > >> select * from store.sale where store_id=100009 > >> select * from store.sale where store_id=100010 > >> select * from store.sale where store_id=100013 > >> select * from store.sale where store_id=100027 > >> select * from store.sale limit 10 > >> select count(*) from store.sale > >> select * from store.sale where store_id=100005 > >> select count(*) from store.sale where store_id=100005 > >> select distinct(store_id) from store.sale order by store_id > >> > >> > >> But all work fine when table is cached to memory, not throwing > >> 'NullPointerException'. > >> > >> > >> > >> > >> -- > >> View this message in context: http://apache-carbondata- > >> mailing-list-archive.1130556.n5.nabble.com/query-err- > >> NullPointerException-but-fine-after-table-cached-in-memory-tp6032.html > >> Sent from the Apache CarbonData Mailing List archive mailing list > archive > >> at Nabble.com. > >> > > > > > > -- > View this message in context: http://apache-carbondata- > mailing-list-archive.1130556.n5.nabble.com/query-err- > NullPointerException-but-fine-after-table-cached-in-memory- > tp6032p6035.html > Sent from the Apache CarbonData Mailing List archive mailing list archive > at Nabble.com. >
kumar vishal
|
|
|
OK. Thanks.
|
«
Return to Apache CarbonData Dev Mailing List archive
|
1 view|%1 views
| Free forum by Nabble | Edit this page |