Guido van Rossum | 7a46564 | 2013-11-22 11:47:22 -0800 | [diff] [blame] | 1 | :mod:`asyncio` -- Asynchronous I/O, event loop, coroutines and tasks |
| 2 | ==================================================================== |
| 3 | |
| 4 | .. module:: asyncio |
| 5 | :synopsis: Asynchronous I/O, event loop, coroutines and tasks. |
| 6 | |
| 7 | .. versionadded:: 3.4 |
| 8 | |
Guido van Rossum | f8d0ff9 | 2013-11-22 16:53:25 -0800 | [diff] [blame^] | 9 | **Source code:** :source:`Lib/asyncio/` |
| 10 | |
| 11 | -------------- |
Guido van Rossum | 7a46564 | 2013-11-22 11:47:22 -0800 | [diff] [blame] | 12 | |
Antoine Pitrou | bba8682 | 2013-11-23 00:34:26 +0100 | [diff] [blame] | 13 | This module provides infrastructure for writing single-threaded concurrent |
| 14 | code using coroutines, multiplexing I/O access over sockets and other |
| 15 | resources, running network clients and servers, and other related primitives. |
Guido van Rossum | 7a46564 | 2013-11-22 11:47:22 -0800 | [diff] [blame] | 16 | |
Antoine Pitrou | bba8682 | 2013-11-23 00:34:26 +0100 | [diff] [blame] | 17 | Here is a more detailed list of the package contents: |
| 18 | |
| 19 | * a pluggable :ref:`event loop <event-loop>` with various system-specific |
| 20 | implementations; |
| 21 | |
| 22 | * :ref:`transport <transport>` and :ref:`protocol <protocol>` abstractions |
| 23 | (similar to those in `Twisted <http://twistedmatrix.com/>`_); |
| 24 | |
| 25 | * concrete support for TCP, UDP, SSL, subprocess pipes, delayed calls, and |
| 26 | others (some may be system-dependent); |
| 27 | |
| 28 | * a Future class that mimicks the one in the :mod:`concurrent.futures` module, |
| 29 | but adapted for use with the event loop; |
| 30 | |
| 31 | * coroutines and tasks based on ``yield from`` (:PEP:`380`), to help write |
| 32 | concurrent code in a sequential fashion; |
| 33 | |
| 34 | * cancellation support for Futures and coroutines; |
| 35 | |
| 36 | * :ref:`synchronization primitives <sync>` for use between coroutines in |
| 37 | a single thread, mimicking those in the :mod:`threading` module; |
| 38 | |
Guido van Rossum | f0f5d38 | 2013-11-22 15:45:02 -0800 | [diff] [blame] | 39 | * an interface for passing work off to a threadpool, for times when |
| 40 | you absolutely, positively have to use a library that makes blocking |
| 41 | I/O calls. |
| 42 | |
Antoine Pitrou | bba8682 | 2013-11-23 00:34:26 +0100 | [diff] [blame] | 43 | |
| 44 | Disclaimer |
| 45 | ---------- |
Guido van Rossum | 7a46564 | 2013-11-22 11:47:22 -0800 | [diff] [blame] | 46 | |
| 47 | Full documentation is not yet ready; we hope to have it written |
| 48 | before Python 3.4 leaves beta. Until then, the best reference is |
| 49 | :PEP:`3156`. For a motivational primer on transports and protocols, |
| 50 | see :PEP:`3153`. |
Antoine Pitrou | bba8682 | 2013-11-23 00:34:26 +0100 | [diff] [blame] | 51 | |
| 52 | |
| 53 | .. XXX should the asyncio documentation come in several pages, as for logging? |
| 54 | |
| 55 | |
| 56 | .. _event-loop: |
| 57 | |
| 58 | Event loops |
| 59 | ----------- |
| 60 | |
| 61 | |
| 62 | .. _protocol: |
| 63 | |
| 64 | Protocols |
| 65 | --------- |
| 66 | |
Antoine Pitrou | a035e1b | 2013-11-23 01:08:43 +0100 | [diff] [blame] | 67 | :mod:`asyncio` provides base classes that you can subclass to implement |
| 68 | your network protocols. Those classes are used in conjunction with |
| 69 | :ref:`transports <transport>` (see below): the protocol parses incoming |
| 70 | data and asks for the writing of outgoing data, while the transport is |
| 71 | responsible for the actual I/O and buffering. |
| 72 | |
| 73 | When subclassing a protocol class, it is recommended you override certain |
| 74 | methods. Those methods are callbacks: they will be called by the transport |
| 75 | on certain events (for example when some data is received); you shouldn't |
| 76 | call them yourself, unless you are implementing a transport. |
| 77 | |
Antoine Pitrou | 74193af | 2013-11-23 01:21:11 +0100 | [diff] [blame] | 78 | .. note:: |
| 79 | All callbacks have default implementations, which are empty. Therefore, |
| 80 | you only need to implement the callbacks for the events in which you |
| 81 | are interested. |
| 82 | |
Antoine Pitrou | a035e1b | 2013-11-23 01:08:43 +0100 | [diff] [blame] | 83 | |
| 84 | Protocol classes |
| 85 | ^^^^^^^^^^^^^^^^ |
| 86 | |
| 87 | .. class:: Protocol |
| 88 | |
| 89 | The base class for implementing streaming protocols (for use with |
| 90 | e.g. TCP and SSL transports). |
| 91 | |
| 92 | .. class:: DatagramProtocol |
| 93 | |
| 94 | The base class for implementing datagram protocols (for use with |
| 95 | e.g. UDP transports). |
| 96 | |
| 97 | .. class:: SubprocessProtocol |
| 98 | |
Antoine Pitrou | 74193af | 2013-11-23 01:21:11 +0100 | [diff] [blame] | 99 | The base class for implementing protocols communicating with child |
| 100 | processes (through a set of unidirectional pipes). |
Antoine Pitrou | a035e1b | 2013-11-23 01:08:43 +0100 | [diff] [blame] | 101 | |
| 102 | |
| 103 | Connection callbacks |
| 104 | ^^^^^^^^^^^^^^^^^^^^ |
| 105 | |
| 106 | These callbacks may be called on :class:`Protocol` and |
Antoine Pitrou | 74193af | 2013-11-23 01:21:11 +0100 | [diff] [blame] | 107 | :class:`SubprocessProtocol` instances: |
Antoine Pitrou | a035e1b | 2013-11-23 01:08:43 +0100 | [diff] [blame] | 108 | |
| 109 | .. method:: connection_made(transport) |
| 110 | |
| 111 | Called when a connection is made. |
| 112 | |
| 113 | The *transport* argument is the transport representing the |
| 114 | connection. You are responsible for storing it somewhere |
| 115 | (e.g. as an attribute) if you need to. |
| 116 | |
| 117 | .. method:: connection_lost(exc) |
| 118 | |
| 119 | Called when the connection is lost or closed. |
| 120 | |
| 121 | The argument is either an exception object or :const:`None`. |
| 122 | The latter means a regular EOF is received, or the connection was |
| 123 | aborted or closed by this side of the connection. |
| 124 | |
| 125 | :meth:`connection_made` and :meth:`connection_lost` are called exactly once |
| 126 | per successful connection. All other callbacks will be called between those |
| 127 | two methods, which allows for easier resource management in your protocol |
| 128 | implementation. |
| 129 | |
Antoine Pitrou | 74193af | 2013-11-23 01:21:11 +0100 | [diff] [blame] | 130 | The following callbacks may be called only on :class:`SubprocessProtocol` |
| 131 | instances: |
| 132 | |
| 133 | .. method:: pipe_data_received(fd, data) |
| 134 | |
| 135 | Called when the child process writes data into its stdout or stderr pipe. |
| 136 | *fd* is the integer file descriptor of the pipe. *data* is a non-empty |
| 137 | bytes object containing the data. |
| 138 | |
| 139 | .. method:: pipe_connection_lost(fd, exc) |
| 140 | |
| 141 | Called when one of the pipes communicating with the child process |
| 142 | is closed. *fd* is the integer file descriptor that was closed. |
| 143 | |
| 144 | .. method:: process_exited() |
| 145 | |
| 146 | Called when the child process has exited. |
| 147 | |
Antoine Pitrou | a035e1b | 2013-11-23 01:08:43 +0100 | [diff] [blame] | 148 | |
| 149 | Data reception callbacks |
| 150 | ^^^^^^^^^^^^^^^^^^^^^^^^ |
| 151 | |
Antoine Pitrou | 74193af | 2013-11-23 01:21:11 +0100 | [diff] [blame] | 152 | Streaming protocols |
| 153 | """"""""""""""""""" |
| 154 | |
| 155 | The following callbacks are called on :class:`Protocol` instances: |
Antoine Pitrou | a035e1b | 2013-11-23 01:08:43 +0100 | [diff] [blame] | 156 | |
| 157 | .. method:: data_received(data) |
| 158 | |
| 159 | Called when some data is received. *data* is a non-empty bytes object |
| 160 | containing the incoming data. |
| 161 | |
| 162 | .. note:: |
| 163 | Whether the data is buffered, chunked or reassembled depends on |
| 164 | the transport. In general, you shouldn't rely on specific semantics |
Antoine Pitrou | 74193af | 2013-11-23 01:21:11 +0100 | [diff] [blame] | 165 | and instead make your parsing generic and flexible enough. However, |
| 166 | data is always received in the correct order. |
Antoine Pitrou | a035e1b | 2013-11-23 01:08:43 +0100 | [diff] [blame] | 167 | |
| 168 | .. method:: eof_received() |
| 169 | |
| 170 | Calls when the other end signals it won't send any more data |
| 171 | (for example by calling :meth:`write_eof`, if the other end also uses |
| 172 | asyncio). |
| 173 | |
| 174 | This method may return a false value (including None), in which case |
| 175 | the transport will close itself. Conversely, if this method returns a |
| 176 | true value, closing the transport is up to the protocol. Since the |
| 177 | default implementation returns None, it implicitly closes the connection. |
| 178 | |
| 179 | .. note:: |
| 180 | Some transports such as SSL don't support half-closed connections, |
| 181 | in which case returning true from this method will not prevent closing |
| 182 | the connection. |
| 183 | |
Antoine Pitrou | a035e1b | 2013-11-23 01:08:43 +0100 | [diff] [blame] | 184 | :meth:`data_received` can be called an arbitrary number of times during |
| 185 | a connection. However, :meth:`eof_received` is called at most once |
| 186 | and, if called, :meth:`data_received` won't be called after it. |
| 187 | |
Antoine Pitrou | 74193af | 2013-11-23 01:21:11 +0100 | [diff] [blame] | 188 | Datagram protocols |
| 189 | """""""""""""""""" |
| 190 | |
| 191 | The following callbacks are called on :class:`DatagramProtocol` instances. |
| 192 | |
| 193 | .. method:: datagram_received(data, addr) |
| 194 | |
| 195 | Called when a datagram is received. *data* is a bytes object containing |
| 196 | the incoming data. *addr* is the address of the peer sending the data; |
| 197 | the exact format depends on the transport. |
| 198 | |
| 199 | .. method:: error_received(exc) |
| 200 | |
| 201 | Called when a previous send or receive operation raises an |
| 202 | :class:`OSError`. *exc* is the :class:`OSError` instance. |
| 203 | |
| 204 | This method is called in rare conditions, when the transport (e.g. UDP) |
| 205 | detects that a datagram couldn't be delivered to its recipient. |
| 206 | In many conditions though, undeliverable datagrams will be silently |
| 207 | dropped. |
| 208 | |
Antoine Pitrou | a035e1b | 2013-11-23 01:08:43 +0100 | [diff] [blame] | 209 | |
| 210 | Flow control callbacks |
| 211 | ^^^^^^^^^^^^^^^^^^^^^^ |
| 212 | |
| 213 | These callbacks may be called on :class:`Protocol` and |
Antoine Pitrou | 74193af | 2013-11-23 01:21:11 +0100 | [diff] [blame] | 214 | :class:`SubprocessProtocol` instances: |
Antoine Pitrou | a035e1b | 2013-11-23 01:08:43 +0100 | [diff] [blame] | 215 | |
| 216 | .. method:: pause_writing() |
| 217 | |
| 218 | Called when the transport's buffer goes over the high-water mark. |
| 219 | |
| 220 | .. method:: resume_writing() |
| 221 | |
| 222 | Called when the transport's buffer drains below the low-water mark. |
| 223 | |
| 224 | |
| 225 | :meth:`pause_writing` and :meth:`resume_writing` calls are paired -- |
| 226 | :meth:`pause_writing` is called once when the buffer goes strictly over |
| 227 | the high-water mark (even if subsequent writes increases the buffer size |
| 228 | even more), and eventually :meth:`resume_writing` is called once when the |
| 229 | buffer size reaches the low-water mark. |
| 230 | |
| 231 | .. note:: |
| 232 | If the buffer size equals the high-water mark, |
| 233 | :meth:`pause_writing` is not called -- it must go strictly over. |
| 234 | Conversely, :meth:`resume_writing` is called when the buffer size is |
| 235 | equal or lower than the low-water mark. These end conditions |
| 236 | are important to ensure that things go as expected when either |
| 237 | mark is zero. |
| 238 | |
Antoine Pitrou | bba8682 | 2013-11-23 00:34:26 +0100 | [diff] [blame] | 239 | |
| 240 | .. _transport: |
| 241 | |
| 242 | Transports |
| 243 | ---------- |
| 244 | |
| 245 | |
| 246 | .. _sync: |
| 247 | |
| 248 | Synchronization primitives |
| 249 | -------------------------- |
| 250 | |
| 251 | |
| 252 | Examples |
| 253 | -------- |
| 254 | |
| 255 | A :class:`Protocol` implementing an echo server:: |
| 256 | |
| 257 | class EchoServer(asyncio.Protocol): |
| 258 | |
| 259 | TIMEOUT = 5.0 |
| 260 | |
| 261 | def timeout(self): |
| 262 | print('connection timeout, closing.') |
| 263 | self.transport.close() |
| 264 | |
| 265 | def connection_made(self, transport): |
| 266 | print('connection made') |
| 267 | self.transport = transport |
| 268 | |
| 269 | # start 5 seconds timeout timer |
| 270 | self.h_timeout = asyncio.get_event_loop().call_later( |
| 271 | self.TIMEOUT, self.timeout) |
| 272 | |
| 273 | def data_received(self, data): |
| 274 | print('data received: ', data.decode()) |
| 275 | self.transport.write(b'Re: ' + data) |
| 276 | |
| 277 | # restart timeout timer |
| 278 | self.h_timeout.cancel() |
| 279 | self.h_timeout = asyncio.get_event_loop().call_later( |
| 280 | self.TIMEOUT, self.timeout) |
| 281 | |
| 282 | def eof_received(self): |
| 283 | pass |
| 284 | |
| 285 | def connection_lost(self, exc): |
| 286 | print('connection lost:', exc) |
| 287 | self.h_timeout.cancel() |
| 288 | |