GitHub user manishgupta88 opened a pull request:
https://github.com/apache/carbondata/pull/2086 [CARBONDATA-2181] Thread Leak during compaction processing on restructured table
**Problem**
Thread leak in compaction operation
**Analysis**
Compaction uses both query and data loading processes. During data laod operation during compaction new threads are spawned in sorting, merger and
data writer step using executor service. These threads are not getting closed in case comapciton fails or the operation is killed from spark UI
as observed by taking a thread dump after compaction failure.
**Fix**
Add a task completion listener in each compaction task which will close all the executor service instances as well as clean any other system
resources to prevent thread leak
- [ ] Any interfaces changed?
No
- [ ] Any backward compatibility impacted?
No
- [ ] Document update required?
No
- [ ] Testing done
Manually Verified in cluster testing
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
NA
You can merge this pull request into a Git repository by running:
$ git pull
https://github.com/manishgupta88/carbondata compaction_thread_leak_fix
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/2086.patchTo close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2086
----
commit ca8e8b1ec81c1b2dda10ff2a24bede17aebb112e
Author: manishgupta88 <tomanishgupta18@...>
Date: 2018-03-21T06:38:01Z
Problem
Thread leak in compaction operation
Analysis
Compaction uses both query and data loading processes. During data laod operation during compaction new threads are spawned in sorting, merger and
data writer step using executor service. These threads are not getting closed in case comapciton fails or the operation is killed from spark UI
as observed by taking a thread dump after compaction failure.
Fix
Add a task completion listener in each compaction task which will close all the executor service instances as well as clean any other system
resources to prevent thread leak
----
---
Locked
