ERROR sortdata.SortTempFileChunkHolder when loading data

Posted by dylan on Jan 02, 2018; 9:46am
URL: http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/ERROR-sortdata-SortTempFileChunkHolder-when-loading-data-tp33516.html

 hello all:
  when i loading data on carbondata1.2,spark1.6 on yarn,
  i get this error:
18/01/02 17:27:25 INFO sortdata.SortDataRows: pool-24-thread-1 Time taken to sort and write sort temp file /hadoop7/yarn/nm/usercache/dba/appcache/application_1513058773052_20157/container_e38_1513058773052_20157_01_000002/tmp/7020573636328672_0/test/table1/Fact/Part0/Segment_0/0/sortrowtmp/mobeye_o2o_lbs_frequency_daily_20171229_228_1e8bcd325f9343348172b7e6fb10b3f77020814710024274.sorttemp is: 347
18/01/02 17:27:28 INFO sortdata.SortDataRows: pool-24-thread-1 Time taken to sort and write sort temp file /hadoop7/yarn/nm/usercache/dba/appcache/application_1513058773052_20157/container_e38_1513058773052_20157_01_000002/tmp/7020573636328672_0/test/table1/Fact/Part0/Segment_0/0/sortrowtmp/mobeye_o2o_lbs_frequency_daily_20171229_228_1e8bcd325f9343348172b7e6fb10b3f77020818030809514.sorttemp is: 334
18/01/02 17:27:32 INFO sortdata.SortDataRows: pool-24-thread-1 Time taken to sort and write sort temp file /hadoop7/yarn/nm/usercache/dba/appcache/application_1513058773052_20157/container_e38_1513058773052_20157_01_000002/tmp/7020573636328672_0/test/table1/Fact/Part0/Segment_0/0/sortrowtmp/mobeye_o2o_lbs_frequency_daily_20171229_228_1e8bcd325f9343348172b7e6fb10b3f77020821312442170.sorttemp is: 348
18/01/02 17:27:34 INFO newflow.AbstractDataLoadProcessorStep: Thread-359 Rows processed in step Sort Processor : 7072000
18/01/02 17:27:34 INFO newflow.AbstractDataLoadProcessorStep: Thread-358 Rows processed in step Data Writer : 0
18/01/02 17:27:34 INFO newflow.AbstractDataLoadProcessorStep: Thread-360 Rows processed in step Data Converter : 7072000
18/01/02 17:27:34 INFO newflow.AbstractDataLoadProcessorStep: Thread-361 Rows processed in step Input Processor : 7073000
18/01/02 17:27:35 INFO sortdata.SortDataRows: pool-24-thread-1 Time taken to sort and write sort temp file /hadoop7/yarn/nm/usercache/dba/appcache/application_1513058773052_20157/container_e38_1513058773052_20157_01_000002/tmp/7020573636328672_0/test/table1/Fact/Part0/Segment_0/0/sortrowtmp/mobeye_o2o_lbs_frequency_daily_20171229_228_1e8bcd325f9343348172b7e6fb10b3f77020824678303996.sorttemp is: 355
18/01/02 17:27:41 INFO sortdata.SortDataRows: pool-24-thread-1 Time taken to sort and write sort temp file /hadoop7/yarn/nm/usercache/dba/appcache/application_1513058773052_20157/container_e38_1513058773052_20157_01_000002/tmp/7020573636328672_0/test/table1/Fact/Part0/Segment_0/0/sortrowtmp/mobeye_o2o_lbs_frequency_daily_20171229_228_1e8bcd325f9343348172b7e6fb10b3f77020830413379712.sorttemp is: 369
18/01/02 17:27:42 INFO sortdata.SortDataRows: Executor task launch worker-3 File based sorting will be used
18/01/02 17:27:42 INFO impl.ParallelReadMergeSorterImpl: Executor task launch worker-3 Record Processed For table: table1
18/01/02 17:27:42 INFO store.SingleThreadFinalSortFilesMerger: Executor task launch worker-3 Number of temp file: 16
18/01/02 17:27:42 INFO store.SingleThreadFinalSortFilesMerger: Executor task launch worker-3 File Buffer Size: 655360
18/01/02 17:27:42 INFO store.SingleThreadFinalSortFilesMerger: Executor task launch worker-3 Started adding first record from each file
18/01/02 17:27:42 ERROR sortdata.SortTempFileChunkHolder: pool-86-thread-2
java.lang.NegativeArraySizeException
        at org.apache.carbondata.processing.sortandgroupby.sortdata.SortTempFileChunkHolder.getRowFromStream(SortTempFileChunkHolder.java:334)
        at org.apache.carbondata.processing.sortandgroupby.sortdata.SortTempFileChunkHolder.prefetchRecordsFromFile(SortTempFileChunkHolder.java:518)
        at org.apache.carbondata.processing.sortandgroupby.sortdata.SortTempFileChunkHolder.access$500(SortTempFileChunkHolder.java:42)
        at org.apache.carbondata.processing.sortandgroupby.sortdata.SortTempFileChunkHolder$DataFetcher.call(SortTempFileChunkHolder.java:497)
        at org.apache.carbondata.processing.sortandgroupby.sortdata.SortTempFileChunkHolder.initialise(SortTempFileChunkHolder.java:227)
        at org.apache.carbondata.processing.sortandgroupby.sortdata.SortTempFileChunkHolder.initialize(SortTempFileChunkHolder.java:215)
        at org.apache.carbondata.processing.store.SingleThreadFinalSortFilesMerger$2.run(SingleThreadFinalSortFilesMerger.java:210)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
18/01/02 17:27:42 ERROR sortdata.SortTempFileChunkHolder: pool-86-thread-2
java.lang.NullPointerException
        at org.apache.carbondata.processing.sortandgroupby.sortdata.SortTempFileChunkHolder.initialise(SortTempFileChunkHolder.java:228)
        at org.apache.carbondata.processing.sortandgroupby.sortdata.SortTempFileChunkHolder.initialize(SortTempFileChunkHolder.java:215)
        at org.apache.carbondata.processing.store.SingleThreadFinalSortFilesMerger$2.run(SingleThreadFinalSortFilesMerger.java:210)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
18/01/02 17:27:42 ERROR store.SingleThreadFinalSortFilesMerger: pool-86-thread-2
org.apache.carbondata.processing.sortandgroupby.exception.CarbonSortKeyAndGroupByException: /hadoop7/yarn/nm/usercache/dba/appcache/application_1513058773052_20157/container_e38_1513058773052_20157_01_000002/tmp/7020573636328672_0/test/table1/Fact/Part0/Segment_0/0/sortrowtmp/mobeye_o2o_lbs_frequency_daily_20171229_228_1e8bcd325f9343348172b7e6fb10b3f77020721749691375.merge Problem while reading
        at org.apache.carbondata.processing.sortandgroupby.sortdata.SortTempFileChunkHolder.initialise(SortTempFileChunkHolder.java:246)
        at org.apache.carbondata.processing.sortandgroupby.sortdata.SortTempFileChunkHolder.initialize(SortTempFileChunkHolder.java:215)
        at org.apache.carbondata.processing.store.SingleThreadFinalSortFilesMerger$2.run(SingleThreadFinalSortFilesMerger.java:210)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
        at org.apache.carbondata.processing.sortandgroupby.sortdata.SortTempFileChunkHolder.initialise(SortTempFileChunkHolder.java:228)
        ... 5 more