bpo-33649: Fix asyncio-dev (GH-9324)
diff --git a/Doc/library/asyncio-api-index.rst b/Doc/library/asyncio-api-index.rst
index a4d0e1b..dc69100 100644
--- a/Doc/library/asyncio-api-index.rst
+++ b/Doc/library/asyncio-api-index.rst
@@ -15,7 +15,7 @@
await on multiple things with timeouts.
.. list-table::
- :widths: 30 70
+ :widths: 50 50
* - :func:`run`
- Create event loop, run a coroutine, close the loop.
@@ -71,7 +71,7 @@
.. list-table::
- :widths: 30 70
+ :widths: 50 50
* - :class:`Queue`
- A FIFO queue.
@@ -97,7 +97,7 @@
Utilities to spawn subprocesses and run shell commands.
.. list-table::
- :widths: 30 70
+ :widths: 50 50
* - ``await`` :func:`create_subprocess_exec`
- Create a subprocess.
@@ -120,7 +120,7 @@
High-level APIs to work with network IO.
.. list-table::
- :widths: 30 70
+ :widths: 50 50
* - ``await`` :func:`open_connection`
- Establish a TCP connection.
@@ -155,7 +155,7 @@
Threading-like synchronization primitives that can be used in Tasks.
.. list-table::
- :widths: 30 70
+ :widths: 50 50
* - :class:`Lock`
- A mutex lock.
@@ -185,7 +185,7 @@
==========
.. list-table::
- :widths: 30 70
+ :widths: 50 50
* - :exc:`asyncio.TimeoutError`
diff --git a/Doc/library/asyncio-dev.rst b/Doc/library/asyncio-dev.rst
index 87abe92..e815318 100644
--- a/Doc/library/asyncio-dev.rst
+++ b/Doc/library/asyncio-dev.rst
@@ -2,415 +2,237 @@
.. _asyncio-dev:
-Develop with asyncio
-====================
+=======================
+Developing with asyncio
+=======================
-Asynchronous programming is different than classical "sequential" programming.
-This page lists common traps and explains how to avoid them.
+Asynchronous programming is different from classical "sequential"
+programming.
+
+This page lists common mistakes and traps and explains how
+to avoid them.
.. _asyncio-debug-mode:
-Debug mode of asyncio
----------------------
+Debug Mode
+==========
-The implementation of :mod:`asyncio` has been written for performance.
-In order to ease the development of asynchronous code, you may wish to
-enable *debug mode*.
+By default asyncio runs in production mode. In order to ease
+the development asyncio has a *debug mode*.
-To enable all debug checks for an application:
+To enable debugging for an application:
-* Enable the asyncio debug mode globally by setting the environment variable
- :envvar:`PYTHONASYNCIODEBUG` to ``1``, using ``-X dev`` command line option
- (see the :option:`-X` option), or by calling
- :meth:`loop.set_debug`.
-* Set the log level of the :ref:`asyncio logger <asyncio-logger>` to
- :py:data:`logging.DEBUG`. For example, call
- ``logging.basicConfig(level=logging.DEBUG)`` at startup.
-* Configure the :mod:`warnings` module to display :exc:`ResourceWarning`
- warnings. For example, use the ``-Wdefault`` command line option of Python to
- display them.
+* Enable the debug mode globally by setting the environment variable
+ :envvar:`PYTHONASYNCIODEBUG` to ``1``.
-Examples debug checks:
+* Alternatively, the debug mode can be enabled by using the ``-X dev``
+ command line option for Python (see the :option:`-X` option).
-* Log :ref:`coroutines defined but never "yielded from"
- <asyncio-coroutine-not-scheduled>`
-* :meth:`loop.call_soon` and :meth:`loop.call_at` methods
- raise an exception if they are called from the wrong thread.
-* Log the execution time of the selector
-* Log callbacks taking more than 100 ms to be executed. The
+* Yet another way to enable the debug mode is by calling
+ :meth:`loop.set_debug` or by passing ``debug=True`` to
+ :func:`asyncio.run`.
+
+In addition to enabling debug mode, consider also:
+
+* setting the log level of the :ref:`asyncio logger <asyncio-logger>` to
+ :py:data:`logging.DEBUG`, for example the following snippet of code
+ can be run at startup of the application::
+
+ logging.basicConfig(level=logging.DEBUG)
+
+* configuring the :mod:`warnings` module to display
+ :exc:`ResourceWarning` warnings. One way of doing that is by
+ using the ``-Wdefault`` command line option.
+
+
+In asyncio debug mode the following checks are performed:
+
+* Log :ref:`coroutines that were not awaited
+ <asyncio-coroutine-not-scheduled>`; this mitigates the "forgotten
+ await" pitfall.
+
+* Many non-treadsafe asyncio APIs (such as :meth:`loop.call_soon` and
+ :meth:`loop.call_at` methods) raise an exception if they are called
+ from a wrong thread.
+
+* Log the execution time of the IO selector if it takes too long to
+ perform an IO operation.
+
+* Log callbacks taking longer than 100 ms to be executed. The
:attr:`loop.slow_callback_duration` attribute is the minimum
duration in seconds of "slow" callbacks.
-* :exc:`ResourceWarning` warnings are emitted when transports and event loops
- are :ref:`not closed explicitly <asyncio-close-transports>`.
-
-.. versionchanged:: 3.7
-
- The new ``-X dev`` command line option can now also be used to enable
- the debug mode.
-
-.. seealso::
-
- The :meth:`loop.set_debug` method and the :ref:`asyncio logger
- <asyncio-logger>`.
-
-
-Cancellation
-------------
-
-Cancellation of tasks is not common in classic programming. In asynchronous
-programming, not only is it something common, but you have to prepare your
-code to handle it.
-
-Futures and tasks can be cancelled explicitly with their :meth:`Future.cancel`
-method. The :func:`wait_for` function cancels the waited task when the timeout
-occurs. There are many other cases where a task can be cancelled indirectly.
-
-Don't call :meth:`~Future.set_result` or :meth:`~Future.set_exception` method
-of :class:`Future` if the future is cancelled: it would fail with an exception.
-For example, write::
-
- if not fut.cancelled():
- fut.set_result('done')
-
-Don't schedule directly a call to the :meth:`~Future.set_result` or the
-:meth:`~Future.set_exception` method of a future with
-:meth:`loop.call_soon`: the future can be cancelled before its method
-is called.
-
-If you wait for a future, you should check early if the future was cancelled to
-avoid useless operations. Example::
-
- async def slow_operation(fut):
- if fut.cancelled():
- return
- # ... slow computation ...
- await fut
- # ...
-
-The :func:`shield` function can also be used to ignore cancellation.
.. _asyncio-multithreading:
-Concurrency and multithreading
-------------------------------
+Concurrency and Multithreading
+==============================
An event loop runs in a thread (typically the main thread) and executes
-all callbacks and tasks in its thread. While a task is running in the
-event loop, no other tasks may run in the same thread. When a task
-executes an ``await`` expression, the running task gets suspended, and the
-event loop executes the next task. Prior to suspending the task, the awaiting
-chain is checked, and if the chain ends with a future, the running task is
-not suspended.
+all callbacks and Tasks in its thread. While a Task is running in the
+event loop, no other Tasks can run in the same thread. When a Task
+executes an ``await`` expression, the running Task gets suspended, and
+the event loop executes the next Task.
-To schedule a callback from a different thread, the
+To schedule a callback from a different OS thread, the
:meth:`loop.call_soon_threadsafe` method should be used. Example::
loop.call_soon_threadsafe(callback, *args)
-Most asyncio objects are not thread safe. You should only worry if you access
-objects outside the event loop. For example, to cancel a future, don't call
-directly its :meth:`Future.cancel` method, but::
+Almost all asyncio objects are not thread safe, which is typically
+not a problem unless there is code that works with them from outside
+of a Task or a callback. If there's a need for such code to call a
+low-level asyncio API, the :meth:`loop.call_soon_threadsafe` method
+should be used, e.g.::
loop.call_soon_threadsafe(fut.cancel)
-To handle signals and to execute subprocesses, the event loop must be run in
-the main thread.
-
-To schedule a coroutine object from a different thread, the
+To schedule a coroutine object from a different OS thread, the
:func:`run_coroutine_threadsafe` function should be used. It returns a
:class:`concurrent.futures.Future` to access the result::
+ async def coro_func():
+ return await asyncio.sleep(1, 42)
+
+ # Later in another OS thread:
+
future = asyncio.run_coroutine_threadsafe(coro_func(), loop)
- result = future.result(timeout) # Wait for the result with a timeout
+ # Wait for the result:
+ result = future.result()
+
+To handle signals and to execute subprocesses, the event loop must be
+run in the main thread.
The :meth:`loop.run_in_executor` method can be used with a
-:class:`concurrent.futures.ThreadPoolExecutor` to execute a callback in
-different thread so as not to block the event loop's main thread.
-
-.. seealso::
-
- The :ref:`Synchronization primitives <asyncio-sync>` section describes ways
- to synchronize tasks.
-
- The :ref:`Subprocess and threads <asyncio-subprocess-threads>` section lists
- asyncio limitations to run subprocesses from different threads.
-
-
+:class:`concurrent.futures.ThreadPoolExecutor` to execute
+blocking code in a different OS thread without blocking the OS thread
+that the event loop runs in.
.. _asyncio-handle-blocking:
-Handle blocking functions correctly
------------------------------------
+Running Blocking Code
+=====================
-Blocking functions should not be called directly. For example, if a function
-blocks for 1 second, other tasks are delayed by 1 second which can have an
-important impact on reactivity.
+Blocking (CPU-bound) code should not be called directly. For example,
+if a function performs a CPU-intensive calculation for 1 second,
+all concurrent asyncio Tasks and IO operations would be delayed
+by 1 second.
-For networking and subprocesses, the :mod:`asyncio` module provides high-level
-APIs like :ref:`protocols <asyncio-protocol>`.
-
-An executor can be used to run a task in a different thread or even in a
-different process, to not block the thread of the event loop. See the
-:meth:`loop.run_in_executor` method.
-
-.. seealso::
-
- The :ref:`Delayed calls <asyncio-delayed-calls>` section details how the
- event loop handles time.
+An executor can be used to run a task in a different thread or even in
+a different process to avoid blocking block the OS thread with the
+event loop. See the :meth:`loop.run_in_executor` method for more
+details.
.. _asyncio-logger:
Logging
--------
+=======
-The :mod:`asyncio` module logs information with the :mod:`logging` module in
-the logger ``'asyncio'``.
+asyncio uses the :mod:`logging` module and all logging is performed
+via the ``"asyncio"`` logger.
-The default log level for the :mod:`asyncio` module is :py:data:`logging.INFO`.
-For those not wanting such verbosity from :mod:`asyncio` the log level can
-be changed. For example, to change the level to :py:data:`logging.WARNING`:
+The default log level is :py:data:`logging.INFO`, which can easily be
+adjusted::
-.. code-block:: none
-
- logging.getLogger('asyncio').setLevel(logging.WARNING)
+ logging.getLogger("asyncio").setLevel(logging.WARNING)
.. _asyncio-coroutine-not-scheduled:
-Detect coroutine objects never scheduled
-----------------------------------------
+Detect never awaited coroutines
+===============================
-When a coroutine function is called and its result is not passed to
-:meth:`asyncio.create_task` the execution of the coroutine object will
-never be scheduled which is probably a bug. Using ``asyncio.create_task`` is
-preferred to the low level :func:`ensure_future` and :meth:`loop.create_task`
-methods. :ref:`Enable the debug mode of asyncio <asyncio-debug-mode>`
-to :ref:`log a warning <asyncio-logger>` to detect it.
-
-Example with the bug::
+When a coroutine is called (e.g. ``coro()`` instead of ``await coro()``)
+the call is not wrapped with :meth:`asyncio.create_task`, the execution
+of the coroutine object will never be scheduled. For example::
import asyncio
async def test():
print("never scheduled")
+ async def main():
+ test()
+
+ asyncio.run(main())
+
+Output::
+
+ test.py:7: RuntimeWarning: coroutine 'test' was never awaited
test()
Output in debug mode::
- Coroutine test() at test.py:3 was never yielded from
- Coroutine object created at (most recent call last):
- File "test.py", line 7, in <module>
- test()
+ test.py:7: RuntimeWarning: coroutine 'test' was never awaited
+ Coroutine created at (most recent call last)
+ File "../t.py", line 9, in <module>
+ asyncio.run(main(), debug=True)
-The fix is to call the :meth:`asyncio.create_task` function. Using
-``asyncio.create_task`` is preferred to the low level :func:`ensure_future` and
-:meth:`loop.create_task` methods.
+ < .. >
-.. seealso::
+ File "../t.py", line 7, in main
+ test()
+ test()
- :ref:`Pending task destroyed <asyncio-pending-task-destroyed>`.
+The usual fix is to either await the coroutine or call the
+:meth:`asyncio.create_task` function::
+
+ async def main():
+ await test()
-Detect exceptions never consumed
---------------------------------
+Detect never consumed exceptions
+================================
-Python usually calls :func:`sys.excepthook` on unhandled exceptions. If
-:meth:`Future.set_exception` is called, but the exception is never consumed,
-:func:`sys.excepthook` is not called. Instead, :ref:`a log is emitted
-<asyncio-logger>` when the future is deleted by the garbage collector, with the
-traceback where the exception was raised.
+If a :meth:`Future.set_exception` is called but the Future object is
+never awaited on, the exception would never be propagated to the
+user code. In this case, asyncio would emit a log message when the
+Future object is garbage collected.
-Example of unhandled exception::
+Example of an unhandled exception::
import asyncio
- @asyncio.coroutine
- def bug():
+ async def bug():
raise Exception("not consumed")
- loop = asyncio.get_event_loop()
- asyncio.ensure_future(bug())
- loop.run_forever()
- loop.close()
+ async def main():
+ asyncio.create_task(bug())
+
+ asyncio.run(main())
Output::
Task exception was never retrieved
- future: <Task finished coro=<coro() done, defined at asyncio/coroutines.py:139> exception=Exception('not consumed',)>
+ future: <Task finished coro=<bug() done, defined at test.py:3>
+ exception=Exception('not consumed')>
+
Traceback (most recent call last):
- File "asyncio/tasks.py", line 237, in _step
- result = next(coro)
- File "asyncio/coroutines.py", line 141, in coro
- res = func(*args, **kw)
- File "test.py", line 5, in bug
+ File "test.py", line 4, in bug
raise Exception("not consumed")
Exception: not consumed
-:ref:`Enable the debug mode of asyncio <asyncio-debug-mode>` to get the
-traceback where the task was created. Output in debug mode::
+:ref:`Enable the debug mode <asyncio-debug-mode>` to get the
+traceback where the task was created::
+
+ asyncio.run(main(), debug=True)
+
+Output in debug mode::
Task exception was never retrieved
- future: <Task finished coro=<bug() done, defined at test.py:3> exception=Exception('not consumed',) created at test.py:8>
+ future: <Task finished coro=<bug() done, defined at test.py:3>
+ exception=Exception('not consumed') created at asyncio/tasks.py:321>
+
source_traceback: Object created at (most recent call last):
- File "test.py", line 8, in <module>
- asyncio.ensure_future(bug())
+ File "../t.py", line 9, in <module>
+ asyncio.run(main(), debug=True)
+
+ < .. >
+
Traceback (most recent call last):
- File "asyncio/tasks.py", line 237, in _step
- result = next(coro)
- File "asyncio/coroutines.py", line 79, in __next__
- return next(self.gen)
- File "asyncio/coroutines.py", line 141, in coro
- res = func(*args, **kw)
- File "test.py", line 5, in bug
+ File "../t.py", line 4, in bug
raise Exception("not consumed")
Exception: not consumed
-
-There are different options to fix this issue. The first option is to chain the
-coroutine in another coroutine and use classic try/except::
-
- async def handle_exception():
- try:
- await bug()
- except Exception:
- print("exception consumed")
-
- loop = asyncio.get_event_loop()
- asyncio.ensure_future(handle_exception())
- loop.run_forever()
- loop.close()
-
-Another option is to use the :meth:`asyncio.run` function::
-
- asyncio.run(bug())
-
-.. seealso::
-
- The :meth:`Future.exception` method.
-
-
-Chain coroutines correctly
---------------------------
-
-When a coroutine function calls other coroutine functions and tasks, they
-should be chained explicitly with ``await``. Otherwise, the execution is
-not guaranteed to be sequential.
-
-Example with different bugs using :func:`asyncio.sleep` to simulate slow
-operations::
-
- import asyncio
-
- async def create():
- await asyncio.sleep(3.0)
- print("(1) create file")
-
- async def write():
- await asyncio.sleep(1.0)
- print("(2) write into file")
-
- async def close():
- print("(3) close file")
-
- async def test():
- asyncio.ensure_future(create())
- asyncio.ensure_future(write())
- asyncio.ensure_future(close())
- await asyncio.sleep(2.0)
- loop.stop()
-
- loop = asyncio.get_event_loop()
- asyncio.ensure_future(test())
- loop.run_forever()
- print("Pending tasks at exit: %s" % asyncio.Task.all_tasks(loop))
- loop.close()
-
-Expected output:
-
-.. code-block:: none
-
- (1) create file
- (2) write into file
- (3) close file
- Pending tasks at exit: set()
-
-Actual output:
-
-.. code-block:: none
-
- (3) close file
- (2) write into file
- Pending tasks at exit: {<Task pending create() at test.py:7 wait_for=<Future pending cb=[Task._wakeup()]>>}
- Task was destroyed but it is pending!
- task: <Task pending create() done at test.py:5 wait_for=<Future pending cb=[Task._wakeup()]>>
-
-The loop stopped before the ``create()`` finished, ``close()`` has been called
-before ``write()``, whereas coroutine functions were called in this order:
-``create()``, ``write()``, ``close()``.
-
-To fix the example, tasks must be marked with ``await``::
-
- async def test():
- await asyncio.ensure_future(create())
- await asyncio.ensure_future(write())
- await asyncio.ensure_future(close())
- await asyncio.sleep(2.0)
- loop.stop()
-
-Or without ``asyncio.ensure_future()``::
-
- async def test():
- await create()
- await write()
- await close()
- await asyncio.sleep(2.0)
- loop.stop()
-
-
-.. _asyncio-pending-task-destroyed:
-
-Pending task destroyed
-----------------------
-
-If a pending task is destroyed, the execution of its wrapped :ref:`coroutine
-<coroutine>` did not complete. It is probably a bug and so a warning is logged.
-
-Example of log:
-
-.. code-block:: none
-
- Task was destroyed but it is pending!
- task: <Task pending coro=<kill_me() done, defined at test.py:5> wait_for=<Future pending cb=[Task._wakeup()]>>
-
-:ref:`Enable the debug mode of asyncio <asyncio-debug-mode>` to get the
-traceback where the task was created. Example of log in debug mode:
-
-.. code-block:: none
-
- Task was destroyed but it is pending!
- source_traceback: Object created at (most recent call last):
- File "test.py", line 15, in <module>
- task = asyncio.ensure_future(coro, loop=loop)
- task: <Task pending coro=<kill_me() done, defined at test.py:5> wait_for=<Future pending cb=[Task._wakeup()] created at test.py:7> created at test.py:15>
-
-
-.. seealso::
-
- :ref:`Detect coroutine objects never scheduled <asyncio-coroutine-not-scheduled>`.
-
-.. _asyncio-close-transports:
-
-Close transports and event loops
---------------------------------
-
-When a transport is no longer needed, call its ``close()`` method to release
-resources. Event loops must also be closed explicitly.
-
-If a transport or an event loop is not closed explicitly, a
-:exc:`ResourceWarning` warning will be emitted in its destructor. By default,
-:exc:`ResourceWarning` warnings are ignored. The :ref:`Debug mode of asyncio
-<asyncio-debug-mode>` section explains how to display them.
diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst
index 19a0acd..718c322 100644
--- a/Doc/library/asyncio-eventloop.rst
+++ b/Doc/library/asyncio-eventloop.rst
@@ -1075,6 +1075,11 @@
Set the debug mode of the event loop.
+ .. versionchanged:: 3.7
+
+ The new ``-X dev`` command line option can now also be used
+ to enable the debug mode.
+
.. seealso::
The :ref:`debug mode of asyncio <asyncio-debug-mode>`.
diff --git a/Doc/library/asyncio-future.rst b/Doc/library/asyncio-future.rst
index ff04339..19bf8a6 100644
--- a/Doc/library/asyncio-future.rst
+++ b/Doc/library/asyncio-future.rst
@@ -121,6 +121,16 @@
or an exception set with :meth:`set_result` or
:meth:`set_exception` calls.
+ .. method:: cancelled()
+
+ Return ``True`` if the Future was *cancelled*.
+
+ The method is usually used to check if a Future is not
+ *cancelled* before setting a result or an exception for it::
+
+ if not fut.cancelled():
+ fut.set_result(42)
+
.. method:: add_done_callback(callback, *, context=None)
Add a callback to be run when the Future is *done*.
@@ -180,10 +190,6 @@
.. versionadded:: 3.7
- .. method:: cancelled()
-
- Return ``True`` if the Future was *cancelled*.
-
This example creates a Future object, creates and schedules an
asynchronous Task to set result for the Future, and waits until
diff --git a/Doc/library/asyncio.rst b/Doc/library/asyncio.rst
index 25daeb6..9f45f74 100644
--- a/Doc/library/asyncio.rst
+++ b/Doc/library/asyncio.rst
@@ -32,8 +32,9 @@
as well as **low-level** APIs for *library and framework developers* to:
* create and manage :ref:`event loops <asyncio-event-loop>`, which
- provide asynchronous APIs for networking, subprocesses, OS signals,
- etc;
+ provide asynchronous APIs for :meth:`networking <loop.create_server>`,
+ running :meth:`subprocesses <loop.subprocess_exec>`,
+ handling :meth:`OS signals <loop.add_signal_handler>`, etc;
* implement efficient protocols using
:ref:`transports <asyncio-transports-protocols>`;
@@ -75,11 +76,3 @@
asyncio-api-index.rst
asyncio-dev.rst
-
-
-.. seealso::
-
- The :mod:`asyncio` module was proposed in :PEP:`3156`.
- Since the acceptance of the PEP many new APIs were added and many
- original APIs were altered. The PEP should be treated as a
- historical document.
diff --git a/Doc/library/ipc.rst b/Doc/library/ipc.rst
index 8f5b3b2..b88a174 100644
--- a/Doc/library/ipc.rst
+++ b/Doc/library/ipc.rst
@@ -1,11 +1,11 @@
.. _ipc:
*****************************************
-Interprocess Communication and Networking
+Networking and Interprocess Communication
*****************************************
-The modules described in this chapter provide mechanisms for different processes
-to communicate.
+The modules described in this chapter provide mechanisms for
+networking and inter-processes communication.
Some modules only work for two processes that are on the same machine, e.g.
:mod:`signal` and :mod:`mmap`. Other modules support networking protocols
@@ -15,6 +15,7 @@
.. toctree::
+ :maxdepth: 1
asyncio.rst
socket.rst