[
https://issues.apache.org/jira/browse/CARBONDATA-2848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pawan Malwal updated CARBONDATA-2848:
-------------------------------------
Description:
Steps :
Huge data load performed. (table has 3.5 billion records)
Actual Issue :
One of the task failed during first load and restarted again in the same executor and took double the time to load first segment, rest of the 5 load ran properly.
Due to this failure and task rerun Segment_0 has got double the number of .carbondata files and even the number of records loaded twice.
Expected :
Task should not fail in 1st load. Even after failed task restarts carbondata files and records loaded should not double.
was:
Steps :
Huge data load performed. (table has 3.5 billion records)
Actual Issue :
One of the task failed during first load and restarted again in the same executor and took double the time to load first segment, rest of the 5 load ran properly.
Due to this failure and task rerun Segment_0 has got double the number of .carbondata files and even the number of records loaded twice.
Expected :
Task should not fail in 1st load.
> Task failed during 1st load and restarted on same executor
> ----------------------------------------------------------
>
> Key: CARBONDATA-2848
> URL:
https://issues.apache.org/jira/browse/CARBONDATA-2848> Project: CarbonData
> Issue Type: Bug
> Components: data-load
> Affects Versions: 1.4.1
> Environment: Spark 2.1
> Reporter: Pawan Malwal
> Priority: Minor
>
> Steps :
> Huge data load performed. (table has 3.5 billion records)
>
> Actual Issue :
> One of the task failed during first load and restarted again in the same executor and took double the time to load first segment, rest of the 5 load ran properly.
> Due to this failure and task rerun Segment_0 has got double the number of .carbondata files and even the number of records loaded twice.
>
> Expected :
> Task should not fail in 1st load. Even after failed task restarts carbondata files and records loaded should not double.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Locked
