Apache CarbonData Dev Mailing List archive

ERROR sortdata.SortTempFileChunkHolder when loading data

classic Classic list List threaded Threaded
♦
♦
 2 messages Options
dylan
Reply | Threaded
Open this post in threaded view
♦
♦
|

ERROR sortdata.SortTempFileChunkHolder when loading data

dylan
This post was updated on .
 hello all:
  when i loading data on carbondata1.2,spark1.6 on yarn,
  i get this error:
18/01/02 17:27:25 INFO sortdata.SortDataRows: pool-24-thread-1 Time taken to sort and write sort temp file /hadoop7/yarn/nm/usercache/dba/appcache/application_1513058773052_20157/container_e38_1513058773052_20157_01_000002/tmp/7020573636328672_0/test/table1/Fact/Part0/Segment_0/0/sortrowtmp/mobeye_o2o_lbs_frequency_daily_20171229_228_1e8bcd325f9343348172b7e6fb10b3f77020814710024274.sorttemp is: 347
18/01/02 17:27:28 INFO sortdata.SortDataRows: pool-24-thread-1 Time taken to sort and write sort temp file /hadoop7/yarn/nm/usercache/dba/appcache/application_1513058773052_20157/container_e38_1513058773052_20157_01_000002/tmp/7020573636328672_0/test/table1/Fact/Part0/Segment_0/0/sortrowtmp/mobeye_o2o_lbs_frequency_daily_20171229_228_1e8bcd325f9343348172b7e6fb10b3f77020818030809514.sorttemp is: 334
18/01/02 17:27:32 INFO sortdata.SortDataRows: pool-24-thread-1 Time taken to sort and write sort temp file /hadoop7/yarn/nm/usercache/dba/appcache/application_1513058773052_20157/container_e38_1513058773052_20157_01_000002/tmp/7020573636328672_0/test/table1/Fact/Part0/Segment_0/0/sortrowtmp/mobeye_o2o_lbs_frequency_daily_20171229_228_1e8bcd325f9343348172b7e6fb10b3f77020821312442170.sorttemp is: 348
18/01/02 17:27:34 INFO newflow.AbstractDataLoadProcessorStep: Thread-359 Rows processed in step Sort Processor : 7072000
18/01/02 17:27:34 INFO newflow.AbstractDataLoadProcessorStep: Thread-358 Rows processed in step Data Writer : 0
18/01/02 17:27:34 INFO newflow.AbstractDataLoadProcessorStep: Thread-360 Rows processed in step Data Converter : 7072000
18/01/02 17:27:34 INFO newflow.AbstractDataLoadProcessorStep: Thread-361 Rows processed in step Input Processor : 7073000
18/01/02 17:27:35 INFO sortdata.SortDataRows: pool-24-thread-1 Time taken to sort and write sort temp file /hadoop7/yarn/nm/usercache/dba/appcache/application_1513058773052_20157/container_e38_1513058773052_20157_01_000002/tmp/7020573636328672_0/test/table1/Fact/Part0/Segment_0/0/sortrowtmp/mobeye_o2o_lbs_frequency_daily_20171229_228_1e8bcd325f9343348172b7e6fb10b3f77020824678303996.sorttemp is: 355
18/01/02 17:27:41 INFO sortdata.SortDataRows: pool-24-thread-1 Time taken to sort and write sort temp file /hadoop7/yarn/nm/usercache/dba/appcache/application_1513058773052_20157/container_e38_1513058773052_20157_01_000002/tmp/7020573636328672_0/test/table1/Fact/Part0/Segment_0/0/sortrowtmp/mobeye_o2o_lbs_frequency_daily_20171229_228_1e8bcd325f9343348172b7e6fb10b3f77020830413379712.sorttemp is: 369
18/01/02 17:27:42 INFO sortdata.SortDataRows: Executor task launch worker-3 File based sorting will be used
18/01/02 17:27:42 INFO impl.ParallelReadMergeSorterImpl: Executor task launch worker-3 Record Processed For table: table1
18/01/02 17:27:42 INFO store.SingleThreadFinalSortFilesMerger: Executor task launch worker-3 Number of temp file: 16
18/01/02 17:27:42 INFO store.SingleThreadFinalSortFilesMerger: Executor task launch worker-3 File Buffer Size: 655360
18/01/02 17:27:42 INFO store.SingleThreadFinalSortFilesMerger: Executor task launch worker-3 Started adding first record from each file
18/01/02 17:27:42 ERROR sortdata.SortTempFileChunkHolder: pool-86-thread-2
java.lang.NegativeArraySizeException
        at org.apache.carbondata.processing.sortandgroupby.sortdata.SortTempFileChunkHolder.getRowFromStream(SortTempFileChunkHolder.java:334)
        at org.apache.carbondata.processing.sortandgroupby.sortdata.SortTempFileChunkHolder.prefetchRecordsFromFile(SortTempFileChunkHolder.java:518)
        at org.apache.carbondata.processing.sortandgroupby.sortdata.SortTempFileChunkHolder.access$500(SortTempFileChunkHolder.java:42)
        at org.apache.carbondata.processing.sortandgroupby.sortdata.SortTempFileChunkHolder$DataFetcher.call(SortTempFileChunkHolder.java:497)
        at org.apache.carbondata.processing.sortandgroupby.sortdata.SortTempFileChunkHolder.initialise(SortTempFileChunkHolder.java:227)
        at org.apache.carbondata.processing.sortandgroupby.sortdata.SortTempFileChunkHolder.initialize(SortTempFileChunkHolder.java:215)
        at org.apache.carbondata.processing.store.SingleThreadFinalSortFilesMerger$2.run(SingleThreadFinalSortFilesMerger.java:210)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
18/01/02 17:27:42 ERROR sortdata.SortTempFileChunkHolder: pool-86-thread-2
java.lang.NullPointerException
        at org.apache.carbondata.processing.sortandgroupby.sortdata.SortTempFileChunkHolder.initialise(SortTempFileChunkHolder.java:228)
        at org.apache.carbondata.processing.sortandgroupby.sortdata.SortTempFileChunkHolder.initialize(SortTempFileChunkHolder.java:215)
        at org.apache.carbondata.processing.store.SingleThreadFinalSortFilesMerger$2.run(SingleThreadFinalSortFilesMerger.java:210)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
18/01/02 17:27:42 ERROR store.SingleThreadFinalSortFilesMerger: pool-86-thread-2
org.apache.carbondata.processing.sortandgroupby.exception.CarbonSortKeyAndGroupByException: /hadoop7/yarn/nm/usercache/dba/appcache/application_1513058773052_20157/container_e38_1513058773052_20157_01_000002/tmp/7020573636328672_0/test/table1/Fact/Part0/Segment_0/0/sortrowtmp/mobeye_o2o_lbs_frequency_daily_20171229_228_1e8bcd325f9343348172b7e6fb10b3f77020721749691375.merge Problem while reading
        at org.apache.carbondata.processing.sortandgroupby.sortdata.SortTempFileChunkHolder.initialise(SortTempFileChunkHolder.java:246)
        at org.apache.carbondata.processing.sortandgroupby.sortdata.SortTempFileChunkHolder.initialize(SortTempFileChunkHolder.java:215)
        at org.apache.carbondata.processing.store.SingleThreadFinalSortFilesMerger$2.run(SingleThreadFinalSortFilesMerger.java:210)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
        at org.apache.carbondata.processing.sortandgroupby.sortdata.SortTempFileChunkHolder.initialise(SortTempFileChunkHolder.java:228)
        ... 5 more
sounak
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: ERROR sortdata.SortTempFileChunkHolder when loading data

sounak
Hi,

Can you please try your test case in spark-2.1 or spark-2.2? In case it is
getting reproduce in these versions then please provide us the test case to
reproduce. Also specify if there is any specific configuration change you
have made.

On Tue, Jan 2, 2018 at 3:16 PM, dylan <[hidden email]> wrote:

>  hello all:
>   when i loading data on carbondata1.2,spark1.6 on yarn,
>   i get this error:
> 18/01/02 17:27:25 INFO sortdata.SortDataRows: pool-24-thread-1 Time taken
> to
> sort and write sort temp file
> /hadoop7/yarn/nm/usercache/dba/appcache/application_
> 1513058773052_20157/container_e38_1513058773052_20157_01_
> 000002/tmp/7020573636328672_0/test/table1/Fact/Part0/
> Segment_0/0/sortrowtmp/mobeye_o2o_lbs_frequency_daily_20171229_228_
> 1e8bcd325f9343348172b7e6fb10b3f77020814710024274.sorttemp
> is: 347
> 18/01/02 17:27:28 INFO sortdata.SortDataRows: pool-24-thread-1 Time taken
> to
> sort and write sort temp file
> /hadoop7/yarn/nm/usercache/dba/appcache/application_
> 1513058773052_20157/container_e38_1513058773052_20157_01_
> 000002/tmp/7020573636328672_0/test/table1/Fact/Part0/
> Segment_0/0/sortrowtmp/mobeye_o2o_lbs_frequency_daily_20171229_228_
> 1e8bcd325f9343348172b7e6fb10b3f77020818030809514.sorttemp
> is: 334
> 18/01/02 17:27:32 INFO sortdata.SortDataRows: pool-24-thread-1 Time taken
> to
> sort and write sort temp file
> /hadoop7/yarn/nm/usercache/dba/appcache/application_
> 1513058773052_20157/container_e38_1513058773052_20157_01_
> 000002/tmp/7020573636328672_0/test/table1/Fact/Part0/
> Segment_0/0/sortrowtmp/mobeye_o2o_lbs_frequency_daily_20171229_228_
> 1e8bcd325f9343348172b7e6fb10b3f77020821312442170.sorttemp
> is: 348
> 18/01/02 17:27:34 INFO newflow.AbstractDataLoadProcessorStep: Thread-359
> Rows processed in step Sort Processor : 7072000
> 18/01/02 17:27:34 INFO newflow.AbstractDataLoadProcessorStep: Thread-358
> Rows processed in step Data Writer : 0
> 18/01/02 17:27:34 INFO newflow.AbstractDataLoadProcessorStep: Thread-360
> Rows processed in step Data Converter : 7072000
> 18/01/02 17:27:34 INFO newflow.AbstractDataLoadProcessorStep: Thread-361
> Rows processed in step Input Processor : 7073000
> 18/01/02 17:27:35 INFO sortdata.SortDataRows: pool-24-thread-1 Time taken
> to
> sort and write sort temp file
> /hadoop7/yarn/nm/usercache/dba/appcache/application_
> 1513058773052_20157/container_e38_1513058773052_20157_01_
> 000002/tmp/7020573636328672_0/test/table1/Fact/Part0/
> Segment_0/0/sortrowtmp/mobeye_o2o_lbs_frequency_daily_20171229_228_
> 1e8bcd325f9343348172b7e6fb10b3f77020824678303996.sorttemp
> is: 355
> 18/01/02 17:27:41 INFO sortdata.SortDataRows: pool-24-thread-1 Time taken
> to
> sort and write sort temp file
> /hadoop7/yarn/nm/usercache/dba/appcache/application_
> 1513058773052_20157/container_e38_1513058773052_20157_01_
> 000002/tmp/7020573636328672_0/test/table1/Fact/Part0/
> Segment_0/0/sortrowtmp/mobeye_o2o_lbs_frequency_daily_20171229_228_
> 1e8bcd325f9343348172b7e6fb10b3f77020830413379712.sorttemp
> is: 369
> 18/01/02 17:27:42 INFO sortdata.SortDataRows: Executor task launch worker-3
> File based sorting will be used
> 18/01/02 17:27:42 INFO impl.ParallelReadMergeSorterImpl: Executor task
> launch worker-3 Record Processed For table:
> mobeye_o2o_lbs_frequency_daily_20171229_228_1e8bcd325f9343348172b7e6fb10b3
> f7
> 18/01/02 17:27:42 INFO store.SingleThreadFinalSortFilesMerger: Executor
> task
> launch worker-3 Number of temp file: 16
> 18/01/02 17:27:42 INFO store.SingleThreadFinalSortFilesMerger: Executor
> task
> launch worker-3 File Buffer Size: 655360
> 18/01/02 17:27:42 INFO store.SingleThreadFinalSortFilesMerger: Executor
> task
> launch worker-3 Started adding first record from each file
> 18/01/02 17:27:42 ERROR sortdata.SortTempFileChunkHolder: pool-86-thread-2
> java.lang.NegativeArraySizeException
>         at
> org.apache.carbondata.processing.sortandgroupby.sortdata.
> SortTempFileChunkHolder.getRowFromStream(SortTempFileChunkHolder.java:334)
>         at
> org.apache.carbondata.processing.sortandgroupby.sortdata.
> SortTempFileChunkHolder.prefetchRecordsFromFile(
> SortTempFileChunkHolder.java:518)
>         at
> org.apache.carbondata.processing.sortandgroupby.sortdata.
> SortTempFileChunkHolder.access$500(SortTempFileChunkHolder.java:42)
>         at
> org.apache.carbondata.processing.sortandgroupby.sortdata.
> SortTempFileChunkHolder$DataFetcher.call(SortTempFileChunkHolder.java:497)
>         at
> org.apache.carbondata.processing.sortandgroupby.sortdata.
> SortTempFileChunkHolder.initialise(SortTempFileChunkHolder.java:227)
>         at
> org.apache.carbondata.processing.sortandgroupby.sortdata.
> SortTempFileChunkHolder.initialize(SortTempFileChunkHolder.java:215)
>         at
> org.apache.carbondata.processing.store.SingleThreadFinalSortFilesMerg
> er$2.run(SingleThreadFinalSortFilesMerger.java:210)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
> 18/01/02 17:27:42 ERROR sortdata.SortTempFileChunkHolder: pool-86-thread-2
> java.lang.NullPointerException
>         at
> org.apache.carbondata.processing.sortandgroupby.sortdata.
> SortTempFileChunkHolder.initialise(SortTempFileChunkHolder.java:228)
>         at
> org.apache.carbondata.processing.sortandgroupby.sortdata.
> SortTempFileChunkHolder.initialize(SortTempFileChunkHolder.java:215)
>         at
> org.apache.carbondata.processing.store.SingleThreadFinalSortFilesMerg
> er$2.run(SingleThreadFinalSortFilesMerger.java:210)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
> 18/01/02 17:27:42 ERROR store.SingleThreadFinalSortFilesMerger:
> pool-86-thread-2
> org.apache.carbondata.processing.sortandgroupby.exception.
> CarbonSortKeyAndGroupByException:
> /hadoop7/yarn/nm/usercache/dba/appcache/application_
> 1513058773052_20157/container_e38_1513058773052_20157_01_
> 000002/tmp/7020573636328672_0/test/table1/Fact/Part0/
> Segment_0/0/sortrowtmp/mobeye_o2o_lbs_frequency_daily_20171229_228_
> 1e8bcd325f9343348172b7e6fb10b3f77020721749691375.merge
> Problem while reading
>         at
> org.apache.carbondata.processing.sortandgroupby.sortdata.
> SortTempFileChunkHolder.initialise(SortTempFileChunkHolder.java:246)
>         at
> org.apache.carbondata.processing.sortandgroupby.sortdata.
> SortTempFileChunkHolder.initialize(SortTempFileChunkHolder.java:215)
>         at
> org.apache.carbondata.processing.store.SingleThreadFinalSortFilesMerg
> er$2.run(SingleThreadFinalSortFilesMerger.java:210)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
>         at
> org.apache.carbondata.processing.sortandgroupby.sortdata.
> SortTempFileChunkHolder.initialise(SortTempFileChunkHolder.java:228)
>         ... 5 more
>
>
>
> --
> Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.
> n5.nabble.com/
>



--
Thanks
Sounak
Free forum by Nabble Edit this page