bpo-39360: Ensure all workers exit when finalizing a multiprocessing Pool (GH-19009)
When the pull is not used via the context manager or terminate() is called, there is a system in multiprocessing.util that handles finalization of all pools via an atexit handler (the Finalize) class. This class registers the _terminate_pool handler in the registry of finalizers of the module, and that registry is called on interpreter exit via _exit_function. The problem is that the "happy" path with the context manager or manual call to finalize() does some extra steps that _terminate_pool does not. The step that is not executed when the atexit() handler calls _terminate_pool is pinging the _change_notifier queue to unblock the maintenance threads.
This commit moves the notification to the _terminate_pool function so is called from both code paths.
Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
(cherry picked from commit ac10e0c93218627d1a639db0b7b41714c5f6a883)
Co-authored-by: Batuhan Taşkaya <47358913+isidentical@users.noreply.github.com>
diff --git a/Lib/multiprocessing/pool.py b/Lib/multiprocessing/pool.py
index b223d6a..41dd923 100644
--- a/Lib/multiprocessing/pool.py
+++ b/Lib/multiprocessing/pool.py
@@ -651,8 +651,6 @@
def terminate(self):
util.debug('terminating pool')
self._state = TERMINATE
- self._worker_handler._state = TERMINATE
- self._change_notifier.put(None)
self._terminate()
def join(self):
@@ -682,7 +680,12 @@
# this is guaranteed to only be called once
util.debug('finalizing pool')
+ # Notify that the worker_handler state has been changed so the
+ # _handle_workers loop can be unblocked (and exited) in order to
+ # send the finalization sentinel all the workers.
worker_handler._state = TERMINATE
+ change_notifier.put(None)
+
task_handler._state = TERMINATE
util.debug('helping task handler/workers to finish')