blob: e42f5a93281663574a28de6b2ea844e8818b917d [file] [log] [blame]
Serhiy Storchaka2b57c432018-12-19 08:09:46 +02001:mod:`!contextlib` --- Utilities for :keyword:`!with`\ -statement contexts
2==========================================================================
Georg Brandl116aa622007-08-15 14:28:22 +00003
4.. module:: contextlib
5 :synopsis: Utilities for with-statement contexts.
6
Raymond Hettinger10480942011-01-10 03:26:08 +00007**Source code:** :source:`Lib/contextlib.py`
Georg Brandl116aa622007-08-15 14:28:22 +00008
Raymond Hettinger4f707fd2011-01-10 19:54:11 +00009--------------
10
Georg Brandl116aa622007-08-15 14:28:22 +000011This module provides utilities for common tasks involving the :keyword:`with`
12statement. For more information see also :ref:`typecontextmanager` and
13:ref:`context-managers`.
14
Georg Brandl116aa622007-08-15 14:28:22 +000015
Nick Coghlan3267a302012-05-21 22:54:43 +100016Utilities
17---------
18
19Functions and classes provided:
Georg Brandl116aa622007-08-15 14:28:22 +000020
Brett Cannon9e080e02016-04-08 12:15:27 -070021.. class:: AbstractContextManager
22
Brett Cannon516f5462016-06-09 15:55:52 -070023 An :term:`abstract base class` for classes that implement
Brett Cannon9e080e02016-04-08 12:15:27 -070024 :meth:`object.__enter__` and :meth:`object.__exit__`. A default
25 implementation for :meth:`object.__enter__` is provided which returns
26 ``self`` while :meth:`object.__exit__` is an abstract method which by default
27 returns ``None``. See also the definition of :ref:`typecontextmanager`.
28
29 .. versionadded:: 3.6
30
31
Jelle Zijlstra176baa32017-12-13 17:19:17 -080032.. class:: AbstractAsyncContextManager
33
34 An :term:`abstract base class` for classes that implement
35 :meth:`object.__aenter__` and :meth:`object.__aexit__`. A default
36 implementation for :meth:`object.__aenter__` is provided which returns
37 ``self`` while :meth:`object.__aexit__` is an abstract method which by default
38 returns ``None``. See also the definition of
39 :ref:`async-context-managers`.
40
41 .. versionadded:: 3.7
42
Brett Cannon9e080e02016-04-08 12:15:27 -070043
Georg Brandl8a1caa22010-07-29 16:01:11 +000044.. decorator:: contextmanager
Georg Brandl116aa622007-08-15 14:28:22 +000045
Christian Heimesd8654cf2007-12-02 15:22:16 +000046 This function is a :term:`decorator` that can be used to define a factory
47 function for :keyword:`with` statement context managers, without needing to
48 create a class or separate :meth:`__enter__` and :meth:`__exit__` methods.
Georg Brandl116aa622007-08-15 14:28:22 +000049
Matthias Bussonnierbde782b2018-07-23 14:10:56 -070050 While many objects natively support use in with statements, sometimes a
51 resource needs to be managed that isn't a context manager in its own right,
52 and doesn't implement a ``close()`` method for use with ``contextlib.closing``
53
54 An abstract example would be the following to ensure correct resource
55 management::
Georg Brandl116aa622007-08-15 14:28:22 +000056
Georg Brandl116aa622007-08-15 14:28:22 +000057 from contextlib import contextmanager
58
59 @contextmanager
Matthias Bussonnierbde782b2018-07-23 14:10:56 -070060 def managed_resource(*args, **kwds):
61 # Code to acquire resource, e.g.:
62 resource = acquire_resource(*args, **kwds)
63 try:
64 yield resource
65 finally:
66 # Code to release resource, e.g.:
67 release_resource(resource)
Georg Brandl116aa622007-08-15 14:28:22 +000068
Matthias Bussonnierbde782b2018-07-23 14:10:56 -070069 >>> with managed_resource(timeout=3600) as resource:
70 ... # Resource is released at the end of this block,
71 ... # even if code in the block raises an exception
Georg Brandl116aa622007-08-15 14:28:22 +000072
Georg Brandl9afde1c2007-11-01 20:32:30 +000073 The function being decorated must return a :term:`generator`-iterator when
74 called. This iterator must yield exactly one value, which will be bound to
Serhiy Storchaka2b57c432018-12-19 08:09:46 +020075 the targets in the :keyword:`with` statement's :keyword:`!as` clause, if any.
Georg Brandl116aa622007-08-15 14:28:22 +000076
77 At the point where the generator yields, the block nested in the :keyword:`with`
78 statement is executed. The generator is then resumed after the block is exited.
79 If an unhandled exception occurs in the block, it is reraised inside the
80 generator at the point where the yield occurred. Thus, you can use a
81 :keyword:`try`...\ :keyword:`except`...\ :keyword:`finally` statement to trap
82 the error (if any), or ensure that some cleanup takes place. If an exception is
83 trapped merely in order to log it or to perform some action (rather than to
84 suppress it entirely), the generator must reraise that exception. Otherwise the
Serhiy Storchaka2b57c432018-12-19 08:09:46 +020085 generator context manager will indicate to the :keyword:`!with` statement that
Georg Brandl116aa622007-08-15 14:28:22 +000086 the exception has been handled, and execution will resume with the statement
Serhiy Storchaka2b57c432018-12-19 08:09:46 +020087 immediately following the :keyword:`!with` statement.
Georg Brandl116aa622007-08-15 14:28:22 +000088
Nick Coghlan0ded3e32011-05-05 23:49:25 +100089 :func:`contextmanager` uses :class:`ContextDecorator` so the context managers
90 it creates can be used as decorators as well as in :keyword:`with` statements.
91 When used as a decorator, a new generator instance is implicitly created on
92 each function call (this allows the otherwise "one-shot" context managers
93 created by :func:`contextmanager` to meet the requirement that context
94 managers support multiple invocations in order to be used as decorators).
Michael Foordb3a89842010-06-30 12:17:50 +000095
96 .. versionchanged:: 3.2
97 Use of :class:`ContextDecorator`.
Georg Brandl116aa622007-08-15 14:28:22 +000098
Georg Brandl86e78d12010-07-18 13:43:32 +000099
Jelle Zijlstra2e624692017-04-30 18:25:58 -0700100.. decorator:: asynccontextmanager
101
102 Similar to :func:`~contextlib.contextmanager`, but creates an
103 :ref:`asynchronous context manager <async-context-managers>`.
104
105 This function is a :term:`decorator` that can be used to define a factory
106 function for :keyword:`async with` statement asynchronous context managers,
107 without needing to create a class or separate :meth:`__aenter__` and
108 :meth:`__aexit__` methods. It must be applied to an :term:`asynchronous
109 generator` function.
110
111 A simple example::
112
113 from contextlib import asynccontextmanager
114
115 @asynccontextmanager
116 async def get_connection():
117 conn = await acquire_db_connection()
118 try:
Alexander Vasin416cbce2018-08-25 05:38:11 +0300119 yield conn
Jelle Zijlstra2e624692017-04-30 18:25:58 -0700120 finally:
121 await release_db_connection(conn)
122
123 async def get_all_users():
124 async with get_connection() as conn:
125 return conn.query('SELECT ...')
126
127 .. versionadded:: 3.7
128
129
Georg Brandl116aa622007-08-15 14:28:22 +0000130.. function:: closing(thing)
131
132 Return a context manager that closes *thing* upon completion of the block. This
133 is basically equivalent to::
134
135 from contextlib import contextmanager
136
137 @contextmanager
138 def closing(thing):
139 try:
140 yield thing
141 finally:
142 thing.close()
143
144 And lets you write code like this::
145
Georg Brandl116aa622007-08-15 14:28:22 +0000146 from contextlib import closing
Georg Brandl0f7ede42008-06-23 11:23:31 +0000147 from urllib.request import urlopen
Georg Brandl116aa622007-08-15 14:28:22 +0000148
Georg Brandl0f7ede42008-06-23 11:23:31 +0000149 with closing(urlopen('http://www.python.org')) as page:
Georg Brandl116aa622007-08-15 14:28:22 +0000150 for line in page:
Georg Brandl6911e3c2007-09-04 07:15:32 +0000151 print(line)
Georg Brandl116aa622007-08-15 14:28:22 +0000152
153 without needing to explicitly close ``page``. Even if an error occurs,
154 ``page.close()`` will be called when the :keyword:`with` block is exited.
155
Georg Brandla7c17e52013-10-13 22:25:10 +0200156
Joongi Kim6e8dcda2020-11-02 17:02:48 +0900157.. class:: aclosing(thing)
158
159 Return an async context manager that calls the ``aclose()`` method of *thing*
160 upon completion of the block. This is basically equivalent to::
161
162 from contextlib import asynccontextmanager
163
164 @asynccontextmanager
165 async def aclosing(thing):
166 try:
167 yield thing
168 finally:
169 await thing.aclose()
170
171 Significantly, ``aclosing()`` supports deterministic cleanup of async
172 generators when they happen to exit early by :keyword:`break` or an
173 exception. For example::
174
175 from contextlib import aclosing
176
177 async with aclosing(my_generator()) as values:
178 async for value in values:
179 if value == 42:
180 break
181
182 This pattern ensures that the generator's async exit code is executed in
183 the same context as its iterations (so that exceptions and context
184 variables work as expected, and the exit code isn't run after the
185 lifetime of some task it depends on).
186
187 .. versionadded:: 3.10
188
189
Jesse-Bakker0784a2e2017-11-23 01:23:28 +0100190.. _simplifying-support-for-single-optional-context-managers:
191
192.. function:: nullcontext(enter_result=None)
193
Daniel Porteousc2875452018-07-09 06:49:29 -0700194 Return a context manager that returns *enter_result* from ``__enter__``, but
Jesse-Bakker0784a2e2017-11-23 01:23:28 +0100195 otherwise does nothing. It is intended to be used as a stand-in for an
196 optional context manager, for example::
197
Daniel Porteousc2875452018-07-09 06:49:29 -0700198 def myfunction(arg, ignore_exceptions=False):
199 if ignore_exceptions:
200 # Use suppress to ignore all exceptions.
201 cm = contextlib.suppress(Exception)
202 else:
203 # Do not ignore any exceptions, cm has no effect.
204 cm = contextlib.nullcontext()
205 with cm:
206 # Do something
207
208 An example using *enter_result*::
209
Jesse-Bakker0784a2e2017-11-23 01:23:28 +0100210 def process_file(file_or_path):
211 if isinstance(file_or_path, str):
212 # If string, open file
213 cm = open(file_or_path)
214 else:
215 # Caller is responsible for closing file
216 cm = nullcontext(file_or_path)
217
218 with cm as file:
219 # Perform processing on the file
220
221 .. versionadded:: 3.7
222
223
Nick Coghlan240f86d2013-10-17 23:40:57 +1000224.. function:: suppress(*exceptions)
Raymond Hettingere318a882013-03-10 22:26:51 -0700225
Nick Coghlan240f86d2013-10-17 23:40:57 +1000226 Return a context manager that suppresses any of the specified exceptions
227 if they occur in the body of a with statement and then resumes execution
228 with the first statement following the end of the with statement.
Raymond Hettingere318a882013-03-10 22:26:51 -0700229
Nick Coghlan240f86d2013-10-17 23:40:57 +1000230 As with any other mechanism that completely suppresses exceptions, this
231 context manager should be used only to cover very specific errors where
232 silently continuing with program execution is known to be the right
233 thing to do.
Nick Coghlanb4534ae2013-10-13 23:23:08 +1000234
Raymond Hettingere318a882013-03-10 22:26:51 -0700235 For example::
236
Nick Coghlan240f86d2013-10-17 23:40:57 +1000237 from contextlib import suppress
Raymond Hettingere318a882013-03-10 22:26:51 -0700238
Nick Coghlan240f86d2013-10-17 23:40:57 +1000239 with suppress(FileNotFoundError):
Raymond Hettingere318a882013-03-10 22:26:51 -0700240 os.remove('somefile.tmp')
241
Nick Coghlan240f86d2013-10-17 23:40:57 +1000242 with suppress(FileNotFoundError):
243 os.remove('someotherfile.tmp')
244
Raymond Hettingere318a882013-03-10 22:26:51 -0700245 This code is equivalent to::
246
247 try:
248 os.remove('somefile.tmp')
Nick Coghlanb4534ae2013-10-13 23:23:08 +1000249 except FileNotFoundError:
Raymond Hettingere318a882013-03-10 22:26:51 -0700250 pass
251
Nick Coghlan240f86d2013-10-17 23:40:57 +1000252 try:
253 os.remove('someotherfile.tmp')
254 except FileNotFoundError:
255 pass
256
Nick Coghlan8608d262013-10-20 00:30:51 +1000257 This context manager is :ref:`reentrant <reentrant-cms>`.
258
Raymond Hettingere318a882013-03-10 22:26:51 -0700259 .. versionadded:: 3.4
Georg Brandl116aa622007-08-15 14:28:22 +0000260
Nick Coghlanb4534ae2013-10-13 23:23:08 +1000261
Raymond Hettinger088cbf22013-10-10 00:46:57 -0700262.. function:: redirect_stdout(new_target)
263
264 Context manager for temporarily redirecting :data:`sys.stdout` to
265 another file or file-like object.
266
267 This tool adds flexibility to existing functions or classes whose output
268 is hardwired to stdout.
269
270 For example, the output of :func:`help` normally is sent to *sys.stdout*.
Serhiy Storchakad65c9492015-11-02 14:10:23 +0200271 You can capture that output in a string by redirecting the output to an
Raymond Hettinger088cbf22013-10-10 00:46:57 -0700272 :class:`io.StringIO` object::
273
274 f = io.StringIO()
275 with redirect_stdout(f):
276 help(pow)
277 s = f.getvalue()
278
279 To send the output of :func:`help` to a file on disk, redirect the output
280 to a regular file::
281
282 with open('help.txt', 'w') as f:
283 with redirect_stdout(f):
284 help(pow)
285
286 To send the output of :func:`help` to *sys.stderr*::
287
288 with redirect_stdout(sys.stderr):
289 help(pow)
290
Nick Coghlanb4534ae2013-10-13 23:23:08 +1000291 Note that the global side effect on :data:`sys.stdout` means that this
292 context manager is not suitable for use in library code and most threaded
293 applications. It also has no effect on the output of subprocesses.
294 However, it is still a useful approach for many utility scripts.
295
Nick Coghlan36d8ef92014-10-12 10:25:00 +1000296 This context manager is :ref:`reentrant <reentrant-cms>`.
Nick Coghlan8608d262013-10-20 00:30:51 +1000297
Raymond Hettinger088cbf22013-10-10 00:46:57 -0700298 .. versionadded:: 3.4
299
Georg Brandla7c17e52013-10-13 22:25:10 +0200300
Berker Peksagbb44fe02014-11-28 23:28:06 +0200301.. function:: redirect_stderr(new_target)
302
303 Similar to :func:`~contextlib.redirect_stdout` but redirecting
304 :data:`sys.stderr` to another file or file-like object.
305
306 This context manager is :ref:`reentrant <reentrant-cms>`.
307
308 .. versionadded:: 3.5
309
310
Michael Foordb3a89842010-06-30 12:17:50 +0000311.. class:: ContextDecorator()
312
313 A base class that enables a context manager to also be used as a decorator.
314
315 Context managers inheriting from ``ContextDecorator`` have to implement
316 ``__enter__`` and ``__exit__`` as normal. ``__exit__`` retains its optional
317 exception handling even when used as a decorator.
318
Georg Brandl86e78d12010-07-18 13:43:32 +0000319 ``ContextDecorator`` is used by :func:`contextmanager`, so you get this
320 functionality automatically.
321
322 Example of ``ContextDecorator``::
Michael Foordb3a89842010-06-30 12:17:50 +0000323
324 from contextlib import ContextDecorator
325
326 class mycontext(ContextDecorator):
Georg Brandl86e78d12010-07-18 13:43:32 +0000327 def __enter__(self):
328 print('Starting')
329 return self
Michael Foordb3a89842010-06-30 12:17:50 +0000330
Georg Brandl86e78d12010-07-18 13:43:32 +0000331 def __exit__(self, *exc):
332 print('Finishing')
333 return False
Michael Foordb3a89842010-06-30 12:17:50 +0000334
335 >>> @mycontext()
336 ... def function():
Georg Brandl86e78d12010-07-18 13:43:32 +0000337 ... print('The bit in the middle')
Michael Foordb3a89842010-06-30 12:17:50 +0000338 ...
339 >>> function()
340 Starting
341 The bit in the middle
342 Finishing
343
344 >>> with mycontext():
Georg Brandl86e78d12010-07-18 13:43:32 +0000345 ... print('The bit in the middle')
Michael Foordb3a89842010-06-30 12:17:50 +0000346 ...
347 Starting
348 The bit in the middle
349 Finishing
350
Georg Brandl86e78d12010-07-18 13:43:32 +0000351 This change is just syntactic sugar for any construct of the following form::
352
353 def f():
354 with cm():
355 # Do stuff
356
357 ``ContextDecorator`` lets you instead write::
358
359 @cm()
360 def f():
361 # Do stuff
362
363 It makes it clear that the ``cm`` applies to the whole function, rather than
364 just a piece of it (and saving an indentation level is nice, too).
365
Michael Foordb3a89842010-06-30 12:17:50 +0000366 Existing context managers that already have a base class can be extended by
367 using ``ContextDecorator`` as a mixin class::
368
369 from contextlib import ContextDecorator
370
371 class mycontext(ContextBaseClass, ContextDecorator):
Georg Brandl86e78d12010-07-18 13:43:32 +0000372 def __enter__(self):
373 return self
Michael Foordb3a89842010-06-30 12:17:50 +0000374
Georg Brandl86e78d12010-07-18 13:43:32 +0000375 def __exit__(self, *exc):
376 return False
Michael Foordb3a89842010-06-30 12:17:50 +0000377
Nick Coghlan0ded3e32011-05-05 23:49:25 +1000378 .. note::
379 As the decorated function must be able to be called multiple times, the
380 underlying context manager must support use in multiple :keyword:`with`
381 statements. If this is not the case, then the original construct with the
Serhiy Storchaka2b57c432018-12-19 08:09:46 +0200382 explicit :keyword:`!with` statement inside the function should be used.
Nick Coghlan0ded3e32011-05-05 23:49:25 +1000383
Michael Foordb3a89842010-06-30 12:17:50 +0000384 .. versionadded:: 3.2
385
386
Nick Coghlan3267a302012-05-21 22:54:43 +1000387.. class:: ExitStack()
388
389 A context manager that is designed to make it easy to programmatically
390 combine other context managers and cleanup functions, especially those
391 that are optional or otherwise driven by input data.
392
393 For example, a set of files may easily be handled in a single with
394 statement as follows::
395
396 with ExitStack() as stack:
397 files = [stack.enter_context(open(fname)) for fname in filenames]
398 # All opened files will automatically be closed at the end of
399 # the with statement, even if attempts to open files later
Andrew Svetlov5b898402012-12-18 21:26:36 +0200400 # in the list raise an exception
Nick Coghlan3267a302012-05-21 22:54:43 +1000401
402 Each instance maintains a stack of registered callbacks that are called in
403 reverse order when the instance is closed (either explicitly or implicitly
Nick Coghlan27228272012-05-31 22:17:08 +1000404 at the end of a :keyword:`with` statement). Note that callbacks are *not*
405 invoked implicitly when the context stack instance is garbage collected.
Nick Coghlan3267a302012-05-21 22:54:43 +1000406
407 This stack model is used so that context managers that acquire their
408 resources in their ``__init__`` method (such as file objects) can be
409 handled correctly.
410
411 Since registered callbacks are invoked in the reverse order of
Nick Coghlan27228272012-05-31 22:17:08 +1000412 registration, this ends up behaving as if multiple nested :keyword:`with`
Nick Coghlan3267a302012-05-21 22:54:43 +1000413 statements had been used with the registered set of callbacks. This even
414 extends to exception handling - if an inner callback suppresses or replaces
415 an exception, then outer callbacks will be passed arguments based on that
416 updated state.
417
418 This is a relatively low level API that takes care of the details of
419 correctly unwinding the stack of exit callbacks. It provides a suitable
420 foundation for higher level context managers that manipulate the exit
421 stack in application specific ways.
422
Nick Coghlana497b442012-05-22 23:02:00 +1000423 .. versionadded:: 3.3
424
Nick Coghlan3267a302012-05-21 22:54:43 +1000425 .. method:: enter_context(cm)
426
427 Enters a new context manager and adds its :meth:`__exit__` method to
428 the callback stack. The return value is the result of the context
429 manager's own :meth:`__enter__` method.
430
431 These context managers may suppress exceptions just as they normally
Nick Coghlan27228272012-05-31 22:17:08 +1000432 would if used directly as part of a :keyword:`with` statement.
Nick Coghlan3267a302012-05-21 22:54:43 +1000433
434 .. method:: push(exit)
435
436 Adds a context manager's :meth:`__exit__` method to the callback stack.
437
438 As ``__enter__`` is *not* invoked, this method can be used to cover
439 part of an :meth:`__enter__` implementation with a context manager's own
440 :meth:`__exit__` method.
441
442 If passed an object that is not a context manager, this method assumes
443 it is a callback with the same signature as a context manager's
444 :meth:`__exit__` method and adds it directly to the callback stack.
445
446 By returning true values, these callbacks can suppress exceptions the
447 same way context manager :meth:`__exit__` methods can.
448
449 The passed in object is returned from the function, allowing this
Nick Coghlan27228272012-05-31 22:17:08 +1000450 method to be used as a function decorator.
Nick Coghlan3267a302012-05-21 22:54:43 +1000451
Serhiy Storchaka142566c2019-06-05 18:22:31 +0300452 .. method:: callback(callback, /, *args, **kwds)
Nick Coghlan3267a302012-05-21 22:54:43 +1000453
454 Accepts an arbitrary callback function and arguments and adds it to
455 the callback stack.
456
457 Unlike the other methods, callbacks added this way cannot suppress
458 exceptions (as they are never passed the exception details).
459
460 The passed in callback is returned from the function, allowing this
Nick Coghlan27228272012-05-31 22:17:08 +1000461 method to be used as a function decorator.
Nick Coghlan3267a302012-05-21 22:54:43 +1000462
463 .. method:: pop_all()
464
465 Transfers the callback stack to a fresh :class:`ExitStack` instance
466 and returns it. No callbacks are invoked by this operation - instead,
467 they will now be invoked when the new stack is closed (either
Nick Coghlan27228272012-05-31 22:17:08 +1000468 explicitly or implicitly at the end of a :keyword:`with` statement).
Nick Coghlan3267a302012-05-21 22:54:43 +1000469
470 For example, a group of files can be opened as an "all or nothing"
471 operation as follows::
472
473 with ExitStack() as stack:
474 files = [stack.enter_context(open(fname)) for fname in filenames]
Barry Warsawd8f870d2013-05-10 11:35:38 -0400475 # Hold onto the close method, but don't call it yet.
476 close_files = stack.pop_all().close
Nick Coghlan3267a302012-05-21 22:54:43 +1000477 # If opening any file fails, all previously opened files will be
478 # closed automatically. If all files are opened successfully,
479 # they will remain open even after the with statement ends.
Barry Warsawd8f870d2013-05-10 11:35:38 -0400480 # close_files() can then be invoked explicitly to close them all.
Nick Coghlan3267a302012-05-21 22:54:43 +1000481
482 .. method:: close()
483
484 Immediately unwinds the callback stack, invoking callbacks in the
485 reverse order of registration. For any context managers and exit
486 callbacks registered, the arguments passed in will indicate that no
487 exception occurred.
488
Ilya Kulakov1aa094f2018-01-25 12:51:18 -0800489.. class:: AsyncExitStack()
490
491 An :ref:`asynchronous context manager <async-context-managers>`, similar
492 to :class:`ExitStack`, that supports combining both synchronous and
493 asynchronous context managers, as well as having coroutines for
494 cleanup logic.
495
496 The :meth:`close` method is not implemented, :meth:`aclose` must be used
497 instead.
498
499 .. method:: enter_async_context(cm)
500
501 Similar to :meth:`enter_context` but expects an asynchronous context
502 manager.
503
504 .. method:: push_async_exit(exit)
505
506 Similar to :meth:`push` but expects either an asynchronous context manager
Nathaniel J. Smitha3c88ef2018-09-18 14:27:59 -0700507 or a coroutine function.
Ilya Kulakov1aa094f2018-01-25 12:51:18 -0800508
Serhiy Storchaka142566c2019-06-05 18:22:31 +0300509 .. method:: push_async_callback(callback, /, *args, **kwds)
Ilya Kulakov1aa094f2018-01-25 12:51:18 -0800510
Nathaniel J. Smitha3c88ef2018-09-18 14:27:59 -0700511 Similar to :meth:`callback` but expects a coroutine function.
Ilya Kulakov1aa094f2018-01-25 12:51:18 -0800512
513 .. method:: aclose()
514
515 Similar to :meth:`close` but properly handles awaitables.
516
517 Continuing the example for :func:`asynccontextmanager`::
518
519 async with AsyncExitStack() as stack:
520 connections = [await stack.enter_async_context(get_connection())
521 for i in range(5)]
522 # All opened connections will automatically be released at the end of
523 # the async with statement, even if attempts to open a connection
524 # later in the list raise an exception.
525
526 .. versionadded:: 3.7
Nick Coghlan3267a302012-05-21 22:54:43 +1000527
528Examples and Recipes
529--------------------
530
531This section describes some examples and recipes for making effective use of
532the tools provided by :mod:`contextlib`.
533
534
Nick Coghlan27228272012-05-31 22:17:08 +1000535Supporting a variable number of context managers
536^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
537
538The primary use case for :class:`ExitStack` is the one given in the class
539documentation: supporting a variable number of context managers and other
540cleanup operations in a single :keyword:`with` statement. The variability
541may come from the number of context managers needed being driven by user
542input (such as opening a user specified collection of files), or from
543some of the context managers being optional::
544
545 with ExitStack() as stack:
546 for resource in resources:
547 stack.enter_context(resource)
Raymond Hettingere8e2df32014-05-25 18:06:04 -0700548 if need_special_resource():
Nick Coghlan27228272012-05-31 22:17:08 +1000549 special = acquire_special_resource()
550 stack.callback(release_special_resource, special)
551 # Perform operations that use the acquired resources
552
553As shown, :class:`ExitStack` also makes it quite easy to use :keyword:`with`
554statements to manage arbitrary resources that don't natively support the
555context management protocol.
556
557
Nick Coghlan27228272012-05-31 22:17:08 +1000558Catching exceptions from ``__enter__`` methods
559^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
560
561It is occasionally desirable to catch exceptions from an ``__enter__``
562method implementation, *without* inadvertently catching exceptions from
563the :keyword:`with` statement body or the context manager's ``__exit__``
564method. By using :class:`ExitStack` the steps in the context management
565protocol can be separated slightly in order to allow this::
566
567 stack = ExitStack()
568 try:
569 x = stack.enter_context(cm)
570 except Exception:
571 # handle __enter__ exception
572 else:
573 with stack:
574 # Handle normal case
575
576Actually needing to do this is likely to indicate that the underlying API
577should be providing a direct resource management interface for use with
578:keyword:`try`/:keyword:`except`/:keyword:`finally` statements, but not
579all APIs are well designed in that regard. When a context manager is the
580only resource management API provided, then :class:`ExitStack` can make it
581easier to handle various situations that can't be handled directly in a
582:keyword:`with` statement.
583
584
Nick Coghlan3267a302012-05-21 22:54:43 +1000585Cleaning up in an ``__enter__`` implementation
586^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
587
588As noted in the documentation of :meth:`ExitStack.push`, this
589method can be useful in cleaning up an already allocated resource if later
590steps in the :meth:`__enter__` implementation fail.
591
592Here's an example of doing this for a context manager that accepts resource
593acquisition and release functions, along with an optional validation function,
594and maps them to the context management protocol::
595
Brett Cannon9e080e02016-04-08 12:15:27 -0700596 from contextlib import contextmanager, AbstractContextManager, ExitStack
Nick Coghlan3267a302012-05-21 22:54:43 +1000597
Brett Cannon9e080e02016-04-08 12:15:27 -0700598 class ResourceManager(AbstractContextManager):
Nick Coghlan3267a302012-05-21 22:54:43 +1000599
600 def __init__(self, acquire_resource, release_resource, check_resource_ok=None):
601 self.acquire_resource = acquire_resource
602 self.release_resource = release_resource
603 if check_resource_ok is None:
604 def check_resource_ok(resource):
605 return True
606 self.check_resource_ok = check_resource_ok
607
608 @contextmanager
609 def _cleanup_on_error(self):
610 with ExitStack() as stack:
611 stack.push(self)
612 yield
613 # The validation check passed and didn't raise an exception
614 # Accordingly, we want to keep the resource, and pass it
615 # back to our caller
616 stack.pop_all()
617
618 def __enter__(self):
619 resource = self.acquire_resource()
620 with self._cleanup_on_error():
621 if not self.check_resource_ok(resource):
622 msg = "Failed validation for {!r}"
623 raise RuntimeError(msg.format(resource))
624 return resource
625
626 def __exit__(self, *exc_details):
627 # We don't need to duplicate any of our resource release logic
628 self.release_resource()
629
630
631Replacing any use of ``try-finally`` and flag variables
632^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
633
634A pattern you will sometimes see is a ``try-finally`` statement with a flag
635variable to indicate whether or not the body of the ``finally`` clause should
636be executed. In its simplest form (that can't already be handled just by
637using an ``except`` clause instead), it looks something like this::
638
639 cleanup_needed = True
640 try:
641 result = perform_operation()
642 if result:
643 cleanup_needed = False
644 finally:
645 if cleanup_needed:
646 cleanup_resources()
647
648As with any ``try`` statement based code, this can cause problems for
649development and review, because the setup code and the cleanup code can end
650up being separated by arbitrarily long sections of code.
651
652:class:`ExitStack` makes it possible to instead register a callback for
653execution at the end of a ``with`` statement, and then later decide to skip
654executing that callback::
655
656 from contextlib import ExitStack
657
658 with ExitStack() as stack:
659 stack.callback(cleanup_resources)
660 result = perform_operation()
661 if result:
662 stack.pop_all()
663
664This allows the intended cleanup up behaviour to be made explicit up front,
665rather than requiring a separate flag variable.
666
667If a particular application uses this pattern a lot, it can be simplified
668even further by means of a small helper class::
669
670 from contextlib import ExitStack
671
672 class Callback(ExitStack):
Serhiy Storchaka2085bd02019-06-01 11:00:15 +0300673 def __init__(self, callback, /, *args, **kwds):
Nick Coghlan3267a302012-05-21 22:54:43 +1000674 super(Callback, self).__init__()
675 self.callback(callback, *args, **kwds)
676
677 def cancel(self):
678 self.pop_all()
679
680 with Callback(cleanup_resources) as cb:
681 result = perform_operation()
682 if result:
683 cb.cancel()
684
685If the resource cleanup isn't already neatly bundled into a standalone
686function, then it is still possible to use the decorator form of
687:meth:`ExitStack.callback` to declare the resource cleanup in
688advance::
689
690 from contextlib import ExitStack
691
692 with ExitStack() as stack:
693 @stack.callback
694 def cleanup_resources():
695 ...
696 result = perform_operation()
697 if result:
698 stack.pop_all()
699
700Due to the way the decorator protocol works, a callback function
701declared this way cannot take any parameters. Instead, any resources to
Martin Panterd21e0b52015-10-10 10:36:22 +0000702be released must be accessed as closure variables.
Nick Coghlan3267a302012-05-21 22:54:43 +1000703
704
705Using a context manager as a function decorator
706^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
707
708:class:`ContextDecorator` makes it possible to use a context manager in
709both an ordinary ``with`` statement and also as a function decorator.
710
711For example, it is sometimes useful to wrap functions or groups of statements
712with a logger that can track the time of entry and time of exit. Rather than
713writing both a function decorator and a context manager for the task,
714inheriting from :class:`ContextDecorator` provides both capabilities in a
715single definition::
716
717 from contextlib import ContextDecorator
718 import logging
719
720 logging.basicConfig(level=logging.INFO)
721
722 class track_entry_and_exit(ContextDecorator):
723 def __init__(self, name):
724 self.name = name
725
726 def __enter__(self):
Vinay Sajipdd917f82016-08-31 08:22:29 +0100727 logging.info('Entering: %s', self.name)
Nick Coghlan3267a302012-05-21 22:54:43 +1000728
729 def __exit__(self, exc_type, exc, exc_tb):
Vinay Sajipdd917f82016-08-31 08:22:29 +0100730 logging.info('Exiting: %s', self.name)
Nick Coghlan3267a302012-05-21 22:54:43 +1000731
732Instances of this class can be used as both a context manager::
733
734 with track_entry_and_exit('widget loader'):
735 print('Some time consuming activity goes here')
736 load_widget()
737
738And also as a function decorator::
739
740 @track_entry_and_exit('widget loader')
741 def activity():
742 print('Some time consuming activity goes here')
743 load_widget()
744
745Note that there is one additional limitation when using context managers
746as function decorators: there's no way to access the return value of
747:meth:`__enter__`. If that value is needed, then it is still necessary to use
748an explicit ``with`` statement.
749
Georg Brandl116aa622007-08-15 14:28:22 +0000750.. seealso::
751
Serhiy Storchakae4ba8722016-03-31 15:30:54 +0300752 :pep:`343` - The "with" statement
Georg Brandl116aa622007-08-15 14:28:22 +0000753 The specification, background, and examples for the Python :keyword:`with`
754 statement.
755
Nick Coghlan0acceb72013-10-20 13:22:21 +1000756.. _single-use-reusable-and-reentrant-cms:
Nick Coghlan8608d262013-10-20 00:30:51 +1000757
Nick Coghlan0acceb72013-10-20 13:22:21 +1000758Single use, reusable and reentrant context managers
759---------------------------------------------------
Nick Coghlan8608d262013-10-20 00:30:51 +1000760
761Most context managers are written in a way that means they can only be
762used effectively in a :keyword:`with` statement once. These single use
763context managers must be created afresh each time they're used -
764attempting to use them a second time will trigger an exception or
765otherwise not work correctly.
766
767This common limitation means that it is generally advisable to create
768context managers directly in the header of the :keyword:`with` statement
769where they are used (as shown in all of the usage examples above).
770
771Files are an example of effectively single use context managers, since
772the first :keyword:`with` statement will close the file, preventing any
773further IO operations using that file object.
774
775Context managers created using :func:`contextmanager` are also single use
776context managers, and will complain about the underlying generator failing
777to yield if an attempt is made to use them a second time::
778
779 >>> from contextlib import contextmanager
780 >>> @contextmanager
781 ... def singleuse():
782 ... print("Before")
783 ... yield
784 ... print("After")
785 ...
786 >>> cm = singleuse()
787 >>> with cm:
788 ... pass
789 ...
790 Before
791 After
792 >>> with cm:
Serhiy Storchakadba90392016-05-10 12:01:23 +0300793 ... pass
Nick Coghlan8608d262013-10-20 00:30:51 +1000794 ...
795 Traceback (most recent call last):
796 ...
797 RuntimeError: generator didn't yield
798
799
800.. _reentrant-cms:
801
802Reentrant context managers
803^^^^^^^^^^^^^^^^^^^^^^^^^^
804
805More sophisticated context managers may be "reentrant". These context
806managers can not only be used in multiple :keyword:`with` statements,
Serhiy Storchaka2b57c432018-12-19 08:09:46 +0200807but may also be used *inside* a :keyword:`!with` statement that is already
Nick Coghlan8608d262013-10-20 00:30:51 +1000808using the same context manager.
809
Nick Coghlan8e113b42013-11-03 17:00:51 +1000810:class:`threading.RLock` is an example of a reentrant context manager, as are
811:func:`suppress` and :func:`redirect_stdout`. Here's a very simple example of
812reentrant use::
Nick Coghlan8608d262013-10-20 00:30:51 +1000813
Nick Coghlan8e113b42013-11-03 17:00:51 +1000814 >>> from contextlib import redirect_stdout
815 >>> from io import StringIO
816 >>> stream = StringIO()
817 >>> write_to_stream = redirect_stdout(stream)
818 >>> with write_to_stream:
819 ... print("This is written to the stream rather than stdout")
820 ... with write_to_stream:
821 ... print("This is also written to the stream")
Nick Coghlan8608d262013-10-20 00:30:51 +1000822 ...
Nick Coghlan8e113b42013-11-03 17:00:51 +1000823 >>> print("This is written directly to stdout")
824 This is written directly to stdout
825 >>> print(stream.getvalue())
826 This is written to the stream rather than stdout
827 This is also written to the stream
828
829Real world examples of reentrancy are more likely to involve multiple
830functions calling each other and hence be far more complicated than this
831example.
832
833Note also that being reentrant is *not* the same thing as being thread safe.
834:func:`redirect_stdout`, for example, is definitely not thread safe, as it
835makes a global modification to the system state by binding :data:`sys.stdout`
836to a different stream.
Nick Coghlan8608d262013-10-20 00:30:51 +1000837
838
839.. _reusable-cms:
840
841Reusable context managers
842^^^^^^^^^^^^^^^^^^^^^^^^^
843
844Distinct from both single use and reentrant context managers are "reusable"
845context managers (or, to be completely explicit, "reusable, but not
846reentrant" context managers, since reentrant context managers are also
847reusable). These context managers support being used multiple times, but
848will fail (or otherwise not work correctly) if the specific context manager
849instance has already been used in a containing with statement.
850
Nick Coghlan8e113b42013-11-03 17:00:51 +1000851:class:`threading.Lock` is an example of a reusable, but not reentrant,
852context manager (for a reentrant lock, it is necessary to use
853:class:`threading.RLock` instead).
Nick Coghlan8608d262013-10-20 00:30:51 +1000854
Nick Coghlan8e113b42013-11-03 17:00:51 +1000855Another example of a reusable, but not reentrant, context manager is
856:class:`ExitStack`, as it invokes *all* currently registered callbacks
857when leaving any with statement, regardless of where those callbacks
858were added::
Nick Coghlan8608d262013-10-20 00:30:51 +1000859
Nick Coghlan8e113b42013-11-03 17:00:51 +1000860 >>> from contextlib import ExitStack
861 >>> stack = ExitStack()
862 >>> with stack:
863 ... stack.callback(print, "Callback: from first context")
864 ... print("Leaving first context")
Nick Coghlan8608d262013-10-20 00:30:51 +1000865 ...
Nick Coghlan8e113b42013-11-03 17:00:51 +1000866 Leaving first context
867 Callback: from first context
868 >>> with stack:
869 ... stack.callback(print, "Callback: from second context")
870 ... print("Leaving second context")
871 ...
872 Leaving second context
873 Callback: from second context
874 >>> with stack:
875 ... stack.callback(print, "Callback: from outer context")
876 ... with stack:
877 ... stack.callback(print, "Callback: from inner context")
878 ... print("Leaving inner context")
879 ... print("Leaving outer context")
880 ...
881 Leaving inner context
882 Callback: from inner context
883 Callback: from outer context
884 Leaving outer context
885
886As the output from the example shows, reusing a single stack object across
887multiple with statements works correctly, but attempting to nest them
888will cause the stack to be cleared at the end of the innermost with
889statement, which is unlikely to be desirable behaviour.
890
891Using separate :class:`ExitStack` instances instead of reusing a single
892instance avoids that problem::
893
894 >>> from contextlib import ExitStack
895 >>> with ExitStack() as outer_stack:
896 ... outer_stack.callback(print, "Callback: from outer context")
897 ... with ExitStack() as inner_stack:
898 ... inner_stack.callback(print, "Callback: from inner context")
899 ... print("Leaving inner context")
900 ... print("Leaving outer context")
901 ...
902 Leaving inner context
903 Callback: from inner context
904 Leaving outer context
905 Callback: from outer context