GitHub user BJangir opened a pull request:
https://github.com/apache/carbondata/pull/2508 [CARBONDATA-2744]Streaming lock is not released even Batch processingâ¦
Issue Detail :- if Streaming Application is running , DDLs like finish streaming ,close streaming are blocked.
ideally DDLs like finish streaming ,close streaming should be blocked if Batch Processing is running. if Batch processing is not happening then DDL's should be allowed from JDBCServer/Beeline.
Root Cause :- Streaming lock is taken on application start and it is released onQueryTerminate event of CarbonStreamingQueryListener ,this event will be called when stop() is called on StreamingQuery which means streaming Lock will be released on Either Streaming Query should be terminated Or complete Streaming Application should be terminated till then all stream lock DDL's are blocked.
Solution :- on AddBatch take streaming lock and once Batch is processed ,release the streaming lock.
Note:-
a. If close streaming is called OR streaming Table is updated with 'streaming'='false' and on Trigger time to AddBatch, addBatch will throw Exception and StreamingQuery should be start again.
b. if DDLs like finish streaming ,close streaming started 1st and addBatch started 2nd. addBatch will throw "can not acquire lock" Exception and StreamingQuery should be start again.
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
- [ ] Any interfaces changed?
NO
- [ ] Any backward compatibility impacted?
NO
- [ ] Document update required?
NO
- [ ] Testing done
Manually verified below scenarios
a. Call StreamingFinished when addBatch is Done.
b. Whether New Batch works after Streaming Finish DDL success. it creates new Streaming Segment
c. New Batch while Streaming Finnish DDL was running
d. Call Streaming Finish when Add Batch was Running
Verified above scenarios for other DDL' also (close streaming,set SET TBLPROPERTIES('streaming'='false')
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
NO
You can merge this pull request into a Git repository by running:
$ git pull
https://github.com/BJangir/incubator-carbondata streaming_lock
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/2508.patchTo close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2508
----
commit 2ff88bad23a74ea9b0167574955c4e14c65ca755
Author: BJangir <babulaljangir111@...>
Date: 2018-07-15T16:14:02Z
[CARBONDATA-2744]Streaming lock is not released even Batch processing is not happening
----
---
Locked
