GitHub user manishgupta88 opened a pull request:
https://github.com/apache/carbondata/pull/2286 [CARBONDATA-2417] [SDK Writer] SDK writer goes to infinite wait when consumer thread goes dead
Problem:
SDK Writer going into infinte loop in case of multi-threaded scenario
Analysis:
In multi-threaded scenarios where multiple instances of writer thread are trying to add a row to the RowBatch, addition to given
batch size cannot be ensured as addition process is not synchronized and it can lead to ArrayIndexOutOfBound Exception or data loss/mismatch issues
If multiple writer threads are adding the data to RowBatch and immediately after launching all the threads closeWriter is called, in that case we
don't know when all the data is finished writing by all the threads but we are returning immediately from close writer after setting the close flag
to true. This does not ensure complete processing of data.
Solution:
Make the row addition logic synchronized and modify the code in closeWriter to ensure data completeness.
- [ ] Any interfaces changed?
No
- [ ] Any backward compatibility impacted?
No
- [ ] Document update required?
No
- [ ] Testing done
Manually verified
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
NA
You can merge this pull request into a Git repository by running:
$ git pull
https://github.com/manishgupta88/carbondata sdk_writer_infinte_loop_fix
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/2286.patchTo close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2286
----
commit dc48cb81745d752e54268b169419fbcef1085369
Author: manishgupta88 <tomanishgupta18@...>
Date: 2018-05-09T09:38:05Z
Problem:
SDK Writer going into infinte loop in case of multi-threaded scenario
Analysis:
In multi-threaded scenarios where multiple instances of writer thread are trying to add a row to the RowBatch, addition to given
batch size cannot be ensured as addition process is not synchronized and it can lead to ArrayIndexOutOfBound Exception or data loss/mismatch issues
If multiple writer threads are adding the data to RowBatch and immediately after launching all the threads closeWriter is called, in that case we
don't know when all the data is finished writing by all the threads but we are returning immediately from close writer after setting the close flag
to true. This does not ensure complete processing of data.
Solution:
Make the row addition logic synchronized and modify the code in closeWriter to ensure data completeness.
----
---
Locked
