bpo-35621: Support running subprocesses in asyncio when loop is executed in non-main thread (GH-14344)
diff --git a/Doc/library/asyncio-policy.rst b/Doc/library/asyncio-policy.rst
index 6212df8..aa8f8f1 100644
--- a/Doc/library/asyncio-policy.rst
+++ b/Doc/library/asyncio-policy.rst
@@ -117,6 +117,7 @@
.. availability:: Windows.
+.. _asyncio-watchers:
Process Watchers
================
@@ -129,10 +130,11 @@
:func:`create_subprocess_exec` and :meth:`loop.subprocess_exec`
functions.
-asyncio defines the :class:`AbstractChildWatcher` abstract base class,
-which child watchers should implement, and has two different
-implementations: :class:`SafeChildWatcher` (configured to be used
-by default) and :class:`FastChildWatcher`.
+asyncio defines the :class:`AbstractChildWatcher` abstract base class, which child
+watchers should implement, and has four different implementations:
+:class:`ThreadedChildWatcher` (configured to be used by default),
+:class:`MultiLoopChildWatcher`, :class:`SafeChildWatcher`, and
+:class:`FastChildWatcher`.
See also the :ref:`Subprocess and Threads <asyncio-subprocess-threads>`
section.
@@ -184,6 +186,15 @@
Note: loop may be ``None``.
+ .. method:: is_active()
+
+ Return ``True`` if the watcher is ready to use.
+
+ Spawning a subprocess with *inactive* current child watcher raises
+ :exc:`RuntimeError`.
+
+ .. versionadded:: 3.8
+
.. method:: close()
Close the watcher.
@@ -191,16 +202,48 @@
This method has to be called to ensure that underlying
resources are cleaned-up.
-.. class:: SafeChildWatcher
+.. class:: ThreadedChildWatcher
- This implementation avoids disrupting other code spawning processes
+ This implementation starts a new waiting thread for every subprocess spawn.
+
+ It works reliably even when the asyncio event loop is run in a non-main OS thread.
+
+ There is no noticeable overhead when handling a big number of children (*O(1)* each
+ time a child terminates), but stating a thread per process requires extra memory.
+
+ This watcher is used by default.
+
+ .. versionadded:: 3.8
+
+.. class:: MultiLoopChildWatcher
+
+ This implementation registers a :py:data:`SIGCHLD` signal handler on
+ instantiation. That can break third-party code that installs a custom handler for
+ `SIGCHLD`. signal).
+
+ The watcher avoids disrupting other code spawning processes
by polling every process explicitly on a :py:data:`SIGCHLD` signal.
- This is a safe solution but it has a significant overhead when
+ There is no limitation for running subprocesses from different threads once the
+ watcher is installed.
+
+ The solution is safe but it has a significant overhead when
handling a big number of processes (*O(n)* each time a
:py:data:`SIGCHLD` is received).
- asyncio uses this safe implementation by default.
+ .. versionadded:: 3.8
+
+.. class:: SafeChildWatcher
+
+ This implementation uses active event loop from the main thread to handle
+ :py:data:`SIGCHLD` signal. If the main thread has no running event loop another
+ thread cannot spawn a subprocess (:exc:`RuntimeError` is raised).
+
+ The watcher avoids disrupting other code spawning processes
+ by polling every process explicitly on a :py:data:`SIGCHLD` signal.
+
+ This solution is as safe as :class:`MultiLoopChildWatcher` and has the same *O(N)*
+ complexity but requires a running event loop in the main thread to work.
.. class:: FastChildWatcher
@@ -211,6 +254,9 @@
There is no noticeable overhead when handling a big number of
children (*O(1)* each time a child terminates).
+ This solution requires a running event loop in the main thread to work, as
+ :class:`SafeChildWatcher`.
+
Custom Policies
===============
diff --git a/Doc/library/asyncio-subprocess.rst b/Doc/library/asyncio-subprocess.rst
index 00dc66c..444fb63 100644
--- a/Doc/library/asyncio-subprocess.rst
+++ b/Doc/library/asyncio-subprocess.rst
@@ -293,18 +293,26 @@
Subprocess and Threads
----------------------
-Standard asyncio event loop supports running subprocesses from
-different threads, but there are limitations:
+Standard asyncio event loop supports running subprocesses from different threads by
+default.
-* An event loop must run in the main thread.
+On Windows subprocesses are provided by :class:`ProactorEventLoop` only (default),
+:class:`SelectorEventLoop` has no subprocess support.
-* The child watcher must be instantiated in the main thread
- before executing subprocesses from other threads. Call the
- :func:`get_child_watcher` function in the main thread to instantiate
- the child watcher.
+On UNIX *child watchers* are used for subprocess finish waiting, see
+:ref:`asyncio-watchers` for more info.
-Note that alternative event loop implementations might not share
-the above limitations; please refer to their documentation.
+
+.. versionchanged:: 3.8
+
+ UNIX switched to use :class:`ThreadedChildWatcher` for spawning subprocesses from
+ different threads without any limitation.
+
+ Spawning a subprocess with *inactive* current child watcher raises
+ :exc:`RuntimeError`.
+
+Note that alternative event loop implementations might have own limitations;
+please refer to their documentation.
.. seealso::