GitHub user ajantha-bhat opened a pull request:
https://github.com/apache/carbondata/pull/2251 [CARBONDATA-2417] SDK writer goes to infinite wait when consumer thread is dead
[CARBONDATA-2417]
problem: SDK writer goes to infinite wait when consumer thread is dead
root cause: due to bad record when exception happens at consumer thread
during write, this message is not reached producer (sdk writer).
So, SDK keeps writing data assuming consumer will consume it. But as
consumer is dead. Queue becomes full and queue.put() will be blocked
forever.
Solution: If cannot be added to queue, check whether consumer is alive
or not after every 10 seconds.
If not alive throw exception, if alive try again
- [ ] Any interfaces changed? no
- [ ] Any backward compatibility impacted? no
- [ ] Document update required? no
- [ ] Testing done
updated the testcase in TestNonTransactionalCarbonTable.scala
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. NA
You can merge this pull request into a Git repository by running:
$ git pull
https://github.com/ajantha-bhat/carbondata branch3
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/2251.patchTo close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2251
----
commit 7f19372fc32a6978ac5aa87f20f346e75a9b1e9d
Author: ajantha-bhat <ajanthabhat@...>
Date: 2018-04-28T12:47:35Z
[CARBONDATA-2417]
problem: SDK writer goes to infinite wait when cosumer thread is dead
root cause: due to bad record when exception happens at consumer thread
during write, this message is not reached producer (sdk writer).
So, SDK keeps writing data assuming consumer will consume it. But as
consumer is dead. Queue becomes full and queue.put() will be blocked
forever.
Solution: If cannot be added to queue, check whether consumer is alive
or not after every 10 seconds.
If not alive throw exception, if alive try again
----
---
Locked
