blob: 7dc5b2989f9c0c02915e32cc6a6cf25efb9c7da8 [file] [log] [blame]
Georg Brandl0eaab972009-06-08 08:00:22 +00001:mod:`contextlib` --- Utilities for :keyword:`with`\ -statement contexts
2========================================================================
Georg Brandl116aa622007-08-15 14:28:22 +00003
4.. module:: contextlib
5 :synopsis: Utilities for with-statement contexts.
6
Raymond Hettinger10480942011-01-10 03:26:08 +00007**Source code:** :source:`Lib/contextlib.py`
Georg Brandl116aa622007-08-15 14:28:22 +00008
Raymond Hettinger4f707fd2011-01-10 19:54:11 +00009--------------
10
Georg Brandl116aa622007-08-15 14:28:22 +000011This module provides utilities for common tasks involving the :keyword:`with`
12statement. For more information see also :ref:`typecontextmanager` and
13:ref:`context-managers`.
14
Georg Brandl116aa622007-08-15 14:28:22 +000015
Nick Coghlan3267a302012-05-21 22:54:43 +100016Utilities
17---------
18
19Functions and classes provided:
Georg Brandl116aa622007-08-15 14:28:22 +000020
Brett Cannon9e080e02016-04-08 12:15:27 -070021.. class:: AbstractContextManager
22
Brett Cannon516f5462016-06-09 15:55:52 -070023 An :term:`abstract base class` for classes that implement
Brett Cannon9e080e02016-04-08 12:15:27 -070024 :meth:`object.__enter__` and :meth:`object.__exit__`. A default
25 implementation for :meth:`object.__enter__` is provided which returns
26 ``self`` while :meth:`object.__exit__` is an abstract method which by default
27 returns ``None``. See also the definition of :ref:`typecontextmanager`.
28
29 .. versionadded:: 3.6
30
31
Jelle Zijlstra176baa32017-12-13 17:19:17 -080032.. class:: AbstractAsyncContextManager
33
34 An :term:`abstract base class` for classes that implement
35 :meth:`object.__aenter__` and :meth:`object.__aexit__`. A default
36 implementation for :meth:`object.__aenter__` is provided which returns
37 ``self`` while :meth:`object.__aexit__` is an abstract method which by default
38 returns ``None``. See also the definition of
39 :ref:`async-context-managers`.
40
41 .. versionadded:: 3.7
42
Brett Cannon9e080e02016-04-08 12:15:27 -070043
Georg Brandl8a1caa22010-07-29 16:01:11 +000044.. decorator:: contextmanager
Georg Brandl116aa622007-08-15 14:28:22 +000045
Christian Heimesd8654cf2007-12-02 15:22:16 +000046 This function is a :term:`decorator` that can be used to define a factory
47 function for :keyword:`with` statement context managers, without needing to
48 create a class or separate :meth:`__enter__` and :meth:`__exit__` methods.
Georg Brandl116aa622007-08-15 14:28:22 +000049
Matthias Bussonnierbde782b2018-07-23 14:10:56 -070050 While many objects natively support use in with statements, sometimes a
51 resource needs to be managed that isn't a context manager in its own right,
52 and doesn't implement a ``close()`` method for use with ``contextlib.closing``
53
54 An abstract example would be the following to ensure correct resource
55 management::
Georg Brandl116aa622007-08-15 14:28:22 +000056
Georg Brandl116aa622007-08-15 14:28:22 +000057 from contextlib import contextmanager
58
59 @contextmanager
Matthias Bussonnierbde782b2018-07-23 14:10:56 -070060 def managed_resource(*args, **kwds):
61 # Code to acquire resource, e.g.:
62 resource = acquire_resource(*args, **kwds)
63 try:
64 yield resource
65 finally:
66 # Code to release resource, e.g.:
67 release_resource(resource)
Georg Brandl116aa622007-08-15 14:28:22 +000068
Matthias Bussonnierbde782b2018-07-23 14:10:56 -070069 >>> with managed_resource(timeout=3600) as resource:
70 ... # Resource is released at the end of this block,
71 ... # even if code in the block raises an exception
Georg Brandl116aa622007-08-15 14:28:22 +000072
Georg Brandl9afde1c2007-11-01 20:32:30 +000073 The function being decorated must return a :term:`generator`-iterator when
74 called. This iterator must yield exactly one value, which will be bound to
75 the targets in the :keyword:`with` statement's :keyword:`as` clause, if any.
Georg Brandl116aa622007-08-15 14:28:22 +000076
77 At the point where the generator yields, the block nested in the :keyword:`with`
78 statement is executed. The generator is then resumed after the block is exited.
79 If an unhandled exception occurs in the block, it is reraised inside the
80 generator at the point where the yield occurred. Thus, you can use a
81 :keyword:`try`...\ :keyword:`except`...\ :keyword:`finally` statement to trap
82 the error (if any), or ensure that some cleanup takes place. If an exception is
83 trapped merely in order to log it or to perform some action (rather than to
84 suppress it entirely), the generator must reraise that exception. Otherwise the
85 generator context manager will indicate to the :keyword:`with` statement that
86 the exception has been handled, and execution will resume with the statement
87 immediately following the :keyword:`with` statement.
88
Nick Coghlan0ded3e32011-05-05 23:49:25 +100089 :func:`contextmanager` uses :class:`ContextDecorator` so the context managers
90 it creates can be used as decorators as well as in :keyword:`with` statements.
91 When used as a decorator, a new generator instance is implicitly created on
92 each function call (this allows the otherwise "one-shot" context managers
93 created by :func:`contextmanager` to meet the requirement that context
94 managers support multiple invocations in order to be used as decorators).
Michael Foordb3a89842010-06-30 12:17:50 +000095
96 .. versionchanged:: 3.2
97 Use of :class:`ContextDecorator`.
Georg Brandl116aa622007-08-15 14:28:22 +000098
Georg Brandl86e78d12010-07-18 13:43:32 +000099
Jelle Zijlstra2e624692017-04-30 18:25:58 -0700100.. decorator:: asynccontextmanager
101
102 Similar to :func:`~contextlib.contextmanager`, but creates an
103 :ref:`asynchronous context manager <async-context-managers>`.
104
105 This function is a :term:`decorator` that can be used to define a factory
106 function for :keyword:`async with` statement asynchronous context managers,
107 without needing to create a class or separate :meth:`__aenter__` and
108 :meth:`__aexit__` methods. It must be applied to an :term:`asynchronous
109 generator` function.
110
111 A simple example::
112
113 from contextlib import asynccontextmanager
114
115 @asynccontextmanager
116 async def get_connection():
117 conn = await acquire_db_connection()
118 try:
Alexander Vasin416cbce2018-08-25 05:38:11 +0300119 yield conn
Jelle Zijlstra2e624692017-04-30 18:25:58 -0700120 finally:
121 await release_db_connection(conn)
122
123 async def get_all_users():
124 async with get_connection() as conn:
125 return conn.query('SELECT ...')
126
127 .. versionadded:: 3.7
128
129
Georg Brandl116aa622007-08-15 14:28:22 +0000130.. function:: closing(thing)
131
132 Return a context manager that closes *thing* upon completion of the block. This
133 is basically equivalent to::
134
135 from contextlib import contextmanager
136
137 @contextmanager
138 def closing(thing):
139 try:
140 yield thing
141 finally:
142 thing.close()
143
144 And lets you write code like this::
145
Georg Brandl116aa622007-08-15 14:28:22 +0000146 from contextlib import closing
Georg Brandl0f7ede42008-06-23 11:23:31 +0000147 from urllib.request import urlopen
Georg Brandl116aa622007-08-15 14:28:22 +0000148
Georg Brandl0f7ede42008-06-23 11:23:31 +0000149 with closing(urlopen('http://www.python.org')) as page:
Georg Brandl116aa622007-08-15 14:28:22 +0000150 for line in page:
Georg Brandl6911e3c2007-09-04 07:15:32 +0000151 print(line)
Georg Brandl116aa622007-08-15 14:28:22 +0000152
153 without needing to explicitly close ``page``. Even if an error occurs,
154 ``page.close()`` will be called when the :keyword:`with` block is exited.
155
Georg Brandla7c17e52013-10-13 22:25:10 +0200156
Jesse-Bakker0784a2e2017-11-23 01:23:28 +0100157.. _simplifying-support-for-single-optional-context-managers:
158
159.. function:: nullcontext(enter_result=None)
160
Daniel Porteousc2875452018-07-09 06:49:29 -0700161 Return a context manager that returns *enter_result* from ``__enter__``, but
Jesse-Bakker0784a2e2017-11-23 01:23:28 +0100162 otherwise does nothing. It is intended to be used as a stand-in for an
163 optional context manager, for example::
164
Daniel Porteousc2875452018-07-09 06:49:29 -0700165 def myfunction(arg, ignore_exceptions=False):
166 if ignore_exceptions:
167 # Use suppress to ignore all exceptions.
168 cm = contextlib.suppress(Exception)
169 else:
170 # Do not ignore any exceptions, cm has no effect.
171 cm = contextlib.nullcontext()
172 with cm:
173 # Do something
174
175 An example using *enter_result*::
176
Jesse-Bakker0784a2e2017-11-23 01:23:28 +0100177 def process_file(file_or_path):
178 if isinstance(file_or_path, str):
179 # If string, open file
180 cm = open(file_or_path)
181 else:
182 # Caller is responsible for closing file
183 cm = nullcontext(file_or_path)
184
185 with cm as file:
186 # Perform processing on the file
187
188 .. versionadded:: 3.7
189
190
Nick Coghlan240f86d2013-10-17 23:40:57 +1000191.. function:: suppress(*exceptions)
Raymond Hettingere318a882013-03-10 22:26:51 -0700192
Nick Coghlan240f86d2013-10-17 23:40:57 +1000193 Return a context manager that suppresses any of the specified exceptions
194 if they occur in the body of a with statement and then resumes execution
195 with the first statement following the end of the with statement.
Raymond Hettingere318a882013-03-10 22:26:51 -0700196
Nick Coghlan240f86d2013-10-17 23:40:57 +1000197 As with any other mechanism that completely suppresses exceptions, this
198 context manager should be used only to cover very specific errors where
199 silently continuing with program execution is known to be the right
200 thing to do.
Nick Coghlanb4534ae2013-10-13 23:23:08 +1000201
Raymond Hettingere318a882013-03-10 22:26:51 -0700202 For example::
203
Nick Coghlan240f86d2013-10-17 23:40:57 +1000204 from contextlib import suppress
Raymond Hettingere318a882013-03-10 22:26:51 -0700205
Nick Coghlan240f86d2013-10-17 23:40:57 +1000206 with suppress(FileNotFoundError):
Raymond Hettingere318a882013-03-10 22:26:51 -0700207 os.remove('somefile.tmp')
208
Nick Coghlan240f86d2013-10-17 23:40:57 +1000209 with suppress(FileNotFoundError):
210 os.remove('someotherfile.tmp')
211
Raymond Hettingere318a882013-03-10 22:26:51 -0700212 This code is equivalent to::
213
214 try:
215 os.remove('somefile.tmp')
Nick Coghlanb4534ae2013-10-13 23:23:08 +1000216 except FileNotFoundError:
Raymond Hettingere318a882013-03-10 22:26:51 -0700217 pass
218
Nick Coghlan240f86d2013-10-17 23:40:57 +1000219 try:
220 os.remove('someotherfile.tmp')
221 except FileNotFoundError:
222 pass
223
Nick Coghlan8608d262013-10-20 00:30:51 +1000224 This context manager is :ref:`reentrant <reentrant-cms>`.
225
Raymond Hettingere318a882013-03-10 22:26:51 -0700226 .. versionadded:: 3.4
Georg Brandl116aa622007-08-15 14:28:22 +0000227
Nick Coghlanb4534ae2013-10-13 23:23:08 +1000228
Raymond Hettinger088cbf22013-10-10 00:46:57 -0700229.. function:: redirect_stdout(new_target)
230
231 Context manager for temporarily redirecting :data:`sys.stdout` to
232 another file or file-like object.
233
234 This tool adds flexibility to existing functions or classes whose output
235 is hardwired to stdout.
236
237 For example, the output of :func:`help` normally is sent to *sys.stdout*.
Serhiy Storchakad65c9492015-11-02 14:10:23 +0200238 You can capture that output in a string by redirecting the output to an
Raymond Hettinger088cbf22013-10-10 00:46:57 -0700239 :class:`io.StringIO` object::
240
241 f = io.StringIO()
242 with redirect_stdout(f):
243 help(pow)
244 s = f.getvalue()
245
246 To send the output of :func:`help` to a file on disk, redirect the output
247 to a regular file::
248
249 with open('help.txt', 'w') as f:
250 with redirect_stdout(f):
251 help(pow)
252
253 To send the output of :func:`help` to *sys.stderr*::
254
255 with redirect_stdout(sys.stderr):
256 help(pow)
257
Nick Coghlanb4534ae2013-10-13 23:23:08 +1000258 Note that the global side effect on :data:`sys.stdout` means that this
259 context manager is not suitable for use in library code and most threaded
260 applications. It also has no effect on the output of subprocesses.
261 However, it is still a useful approach for many utility scripts.
262
Nick Coghlan36d8ef92014-10-12 10:25:00 +1000263 This context manager is :ref:`reentrant <reentrant-cms>`.
Nick Coghlan8608d262013-10-20 00:30:51 +1000264
Raymond Hettinger088cbf22013-10-10 00:46:57 -0700265 .. versionadded:: 3.4
266
Georg Brandla7c17e52013-10-13 22:25:10 +0200267
Berker Peksagbb44fe02014-11-28 23:28:06 +0200268.. function:: redirect_stderr(new_target)
269
270 Similar to :func:`~contextlib.redirect_stdout` but redirecting
271 :data:`sys.stderr` to another file or file-like object.
272
273 This context manager is :ref:`reentrant <reentrant-cms>`.
274
275 .. versionadded:: 3.5
276
277
Michael Foordb3a89842010-06-30 12:17:50 +0000278.. class:: ContextDecorator()
279
280 A base class that enables a context manager to also be used as a decorator.
281
282 Context managers inheriting from ``ContextDecorator`` have to implement
283 ``__enter__`` and ``__exit__`` as normal. ``__exit__`` retains its optional
284 exception handling even when used as a decorator.
285
Georg Brandl86e78d12010-07-18 13:43:32 +0000286 ``ContextDecorator`` is used by :func:`contextmanager`, so you get this
287 functionality automatically.
288
289 Example of ``ContextDecorator``::
Michael Foordb3a89842010-06-30 12:17:50 +0000290
291 from contextlib import ContextDecorator
292
293 class mycontext(ContextDecorator):
Georg Brandl86e78d12010-07-18 13:43:32 +0000294 def __enter__(self):
295 print('Starting')
296 return self
Michael Foordb3a89842010-06-30 12:17:50 +0000297
Georg Brandl86e78d12010-07-18 13:43:32 +0000298 def __exit__(self, *exc):
299 print('Finishing')
300 return False
Michael Foordb3a89842010-06-30 12:17:50 +0000301
302 >>> @mycontext()
303 ... def function():
Georg Brandl86e78d12010-07-18 13:43:32 +0000304 ... print('The bit in the middle')
Michael Foordb3a89842010-06-30 12:17:50 +0000305 ...
306 >>> function()
307 Starting
308 The bit in the middle
309 Finishing
310
311 >>> with mycontext():
Georg Brandl86e78d12010-07-18 13:43:32 +0000312 ... print('The bit in the middle')
Michael Foordb3a89842010-06-30 12:17:50 +0000313 ...
314 Starting
315 The bit in the middle
316 Finishing
317
Georg Brandl86e78d12010-07-18 13:43:32 +0000318 This change is just syntactic sugar for any construct of the following form::
319
320 def f():
321 with cm():
322 # Do stuff
323
324 ``ContextDecorator`` lets you instead write::
325
326 @cm()
327 def f():
328 # Do stuff
329
330 It makes it clear that the ``cm`` applies to the whole function, rather than
331 just a piece of it (and saving an indentation level is nice, too).
332
Michael Foordb3a89842010-06-30 12:17:50 +0000333 Existing context managers that already have a base class can be extended by
334 using ``ContextDecorator`` as a mixin class::
335
336 from contextlib import ContextDecorator
337
338 class mycontext(ContextBaseClass, ContextDecorator):
Georg Brandl86e78d12010-07-18 13:43:32 +0000339 def __enter__(self):
340 return self
Michael Foordb3a89842010-06-30 12:17:50 +0000341
Georg Brandl86e78d12010-07-18 13:43:32 +0000342 def __exit__(self, *exc):
343 return False
Michael Foordb3a89842010-06-30 12:17:50 +0000344
Nick Coghlan0ded3e32011-05-05 23:49:25 +1000345 .. note::
346 As the decorated function must be able to be called multiple times, the
347 underlying context manager must support use in multiple :keyword:`with`
348 statements. If this is not the case, then the original construct with the
349 explicit :keyword:`with` statement inside the function should be used.
350
Michael Foordb3a89842010-06-30 12:17:50 +0000351 .. versionadded:: 3.2
352
353
Nick Coghlan3267a302012-05-21 22:54:43 +1000354.. class:: ExitStack()
355
356 A context manager that is designed to make it easy to programmatically
357 combine other context managers and cleanup functions, especially those
358 that are optional or otherwise driven by input data.
359
360 For example, a set of files may easily be handled in a single with
361 statement as follows::
362
363 with ExitStack() as stack:
364 files = [stack.enter_context(open(fname)) for fname in filenames]
365 # All opened files will automatically be closed at the end of
366 # the with statement, even if attempts to open files later
Andrew Svetlov5b898402012-12-18 21:26:36 +0200367 # in the list raise an exception
Nick Coghlan3267a302012-05-21 22:54:43 +1000368
369 Each instance maintains a stack of registered callbacks that are called in
370 reverse order when the instance is closed (either explicitly or implicitly
Nick Coghlan27228272012-05-31 22:17:08 +1000371 at the end of a :keyword:`with` statement). Note that callbacks are *not*
372 invoked implicitly when the context stack instance is garbage collected.
Nick Coghlan3267a302012-05-21 22:54:43 +1000373
374 This stack model is used so that context managers that acquire their
375 resources in their ``__init__`` method (such as file objects) can be
376 handled correctly.
377
378 Since registered callbacks are invoked in the reverse order of
Nick Coghlan27228272012-05-31 22:17:08 +1000379 registration, this ends up behaving as if multiple nested :keyword:`with`
Nick Coghlan3267a302012-05-21 22:54:43 +1000380 statements had been used with the registered set of callbacks. This even
381 extends to exception handling - if an inner callback suppresses or replaces
382 an exception, then outer callbacks will be passed arguments based on that
383 updated state.
384
385 This is a relatively low level API that takes care of the details of
386 correctly unwinding the stack of exit callbacks. It provides a suitable
387 foundation for higher level context managers that manipulate the exit
388 stack in application specific ways.
389
Nick Coghlana497b442012-05-22 23:02:00 +1000390 .. versionadded:: 3.3
391
Nick Coghlan3267a302012-05-21 22:54:43 +1000392 .. method:: enter_context(cm)
393
394 Enters a new context manager and adds its :meth:`__exit__` method to
395 the callback stack. The return value is the result of the context
396 manager's own :meth:`__enter__` method.
397
398 These context managers may suppress exceptions just as they normally
Nick Coghlan27228272012-05-31 22:17:08 +1000399 would if used directly as part of a :keyword:`with` statement.
Nick Coghlan3267a302012-05-21 22:54:43 +1000400
401 .. method:: push(exit)
402
403 Adds a context manager's :meth:`__exit__` method to the callback stack.
404
405 As ``__enter__`` is *not* invoked, this method can be used to cover
406 part of an :meth:`__enter__` implementation with a context manager's own
407 :meth:`__exit__` method.
408
409 If passed an object that is not a context manager, this method assumes
410 it is a callback with the same signature as a context manager's
411 :meth:`__exit__` method and adds it directly to the callback stack.
412
413 By returning true values, these callbacks can suppress exceptions the
414 same way context manager :meth:`__exit__` methods can.
415
416 The passed in object is returned from the function, allowing this
Nick Coghlan27228272012-05-31 22:17:08 +1000417 method to be used as a function decorator.
Nick Coghlan3267a302012-05-21 22:54:43 +1000418
419 .. method:: callback(callback, *args, **kwds)
420
421 Accepts an arbitrary callback function and arguments and adds it to
422 the callback stack.
423
424 Unlike the other methods, callbacks added this way cannot suppress
425 exceptions (as they are never passed the exception details).
426
427 The passed in callback is returned from the function, allowing this
Nick Coghlan27228272012-05-31 22:17:08 +1000428 method to be used as a function decorator.
Nick Coghlan3267a302012-05-21 22:54:43 +1000429
430 .. method:: pop_all()
431
432 Transfers the callback stack to a fresh :class:`ExitStack` instance
433 and returns it. No callbacks are invoked by this operation - instead,
434 they will now be invoked when the new stack is closed (either
Nick Coghlan27228272012-05-31 22:17:08 +1000435 explicitly or implicitly at the end of a :keyword:`with` statement).
Nick Coghlan3267a302012-05-21 22:54:43 +1000436
437 For example, a group of files can be opened as an "all or nothing"
438 operation as follows::
439
440 with ExitStack() as stack:
441 files = [stack.enter_context(open(fname)) for fname in filenames]
Barry Warsawd8f870d2013-05-10 11:35:38 -0400442 # Hold onto the close method, but don't call it yet.
443 close_files = stack.pop_all().close
Nick Coghlan3267a302012-05-21 22:54:43 +1000444 # If opening any file fails, all previously opened files will be
445 # closed automatically. If all files are opened successfully,
446 # they will remain open even after the with statement ends.
Barry Warsawd8f870d2013-05-10 11:35:38 -0400447 # close_files() can then be invoked explicitly to close them all.
Nick Coghlan3267a302012-05-21 22:54:43 +1000448
449 .. method:: close()
450
451 Immediately unwinds the callback stack, invoking callbacks in the
452 reverse order of registration. For any context managers and exit
453 callbacks registered, the arguments passed in will indicate that no
454 exception occurred.
455
Ilya Kulakov1aa094f2018-01-25 12:51:18 -0800456.. class:: AsyncExitStack()
457
458 An :ref:`asynchronous context manager <async-context-managers>`, similar
459 to :class:`ExitStack`, that supports combining both synchronous and
460 asynchronous context managers, as well as having coroutines for
461 cleanup logic.
462
463 The :meth:`close` method is not implemented, :meth:`aclose` must be used
464 instead.
465
466 .. method:: enter_async_context(cm)
467
468 Similar to :meth:`enter_context` but expects an asynchronous context
469 manager.
470
471 .. method:: push_async_exit(exit)
472
473 Similar to :meth:`push` but expects either an asynchronous context manager
474 or a coroutine.
475
476 .. method:: push_async_callback(callback, *args, **kwds)
477
478 Similar to :meth:`callback` but expects a coroutine.
479
480 .. method:: aclose()
481
482 Similar to :meth:`close` but properly handles awaitables.
483
484 Continuing the example for :func:`asynccontextmanager`::
485
486 async with AsyncExitStack() as stack:
487 connections = [await stack.enter_async_context(get_connection())
488 for i in range(5)]
489 # All opened connections will automatically be released at the end of
490 # the async with statement, even if attempts to open a connection
491 # later in the list raise an exception.
492
493 .. versionadded:: 3.7
Nick Coghlan3267a302012-05-21 22:54:43 +1000494
495Examples and Recipes
496--------------------
497
498This section describes some examples and recipes for making effective use of
499the tools provided by :mod:`contextlib`.
500
501
Nick Coghlan27228272012-05-31 22:17:08 +1000502Supporting a variable number of context managers
503^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
504
505The primary use case for :class:`ExitStack` is the one given in the class
506documentation: supporting a variable number of context managers and other
507cleanup operations in a single :keyword:`with` statement. The variability
508may come from the number of context managers needed being driven by user
509input (such as opening a user specified collection of files), or from
510some of the context managers being optional::
511
512 with ExitStack() as stack:
513 for resource in resources:
514 stack.enter_context(resource)
Raymond Hettingere8e2df32014-05-25 18:06:04 -0700515 if need_special_resource():
Nick Coghlan27228272012-05-31 22:17:08 +1000516 special = acquire_special_resource()
517 stack.callback(release_special_resource, special)
518 # Perform operations that use the acquired resources
519
520As shown, :class:`ExitStack` also makes it quite easy to use :keyword:`with`
521statements to manage arbitrary resources that don't natively support the
522context management protocol.
523
524
Nick Coghlan27228272012-05-31 22:17:08 +1000525Catching exceptions from ``__enter__`` methods
526^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
527
528It is occasionally desirable to catch exceptions from an ``__enter__``
529method implementation, *without* inadvertently catching exceptions from
530the :keyword:`with` statement body or the context manager's ``__exit__``
531method. By using :class:`ExitStack` the steps in the context management
532protocol can be separated slightly in order to allow this::
533
534 stack = ExitStack()
535 try:
536 x = stack.enter_context(cm)
537 except Exception:
538 # handle __enter__ exception
539 else:
540 with stack:
541 # Handle normal case
542
543Actually needing to do this is likely to indicate that the underlying API
544should be providing a direct resource management interface for use with
545:keyword:`try`/:keyword:`except`/:keyword:`finally` statements, but not
546all APIs are well designed in that regard. When a context manager is the
547only resource management API provided, then :class:`ExitStack` can make it
548easier to handle various situations that can't be handled directly in a
549:keyword:`with` statement.
550
551
Nick Coghlan3267a302012-05-21 22:54:43 +1000552Cleaning up in an ``__enter__`` implementation
553^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
554
555As noted in the documentation of :meth:`ExitStack.push`, this
556method can be useful in cleaning up an already allocated resource if later
557steps in the :meth:`__enter__` implementation fail.
558
559Here's an example of doing this for a context manager that accepts resource
560acquisition and release functions, along with an optional validation function,
561and maps them to the context management protocol::
562
Brett Cannon9e080e02016-04-08 12:15:27 -0700563 from contextlib import contextmanager, AbstractContextManager, ExitStack
Nick Coghlan3267a302012-05-21 22:54:43 +1000564
Brett Cannon9e080e02016-04-08 12:15:27 -0700565 class ResourceManager(AbstractContextManager):
Nick Coghlan3267a302012-05-21 22:54:43 +1000566
567 def __init__(self, acquire_resource, release_resource, check_resource_ok=None):
568 self.acquire_resource = acquire_resource
569 self.release_resource = release_resource
570 if check_resource_ok is None:
571 def check_resource_ok(resource):
572 return True
573 self.check_resource_ok = check_resource_ok
574
575 @contextmanager
576 def _cleanup_on_error(self):
577 with ExitStack() as stack:
578 stack.push(self)
579 yield
580 # The validation check passed and didn't raise an exception
581 # Accordingly, we want to keep the resource, and pass it
582 # back to our caller
583 stack.pop_all()
584
585 def __enter__(self):
586 resource = self.acquire_resource()
587 with self._cleanup_on_error():
588 if not self.check_resource_ok(resource):
589 msg = "Failed validation for {!r}"
590 raise RuntimeError(msg.format(resource))
591 return resource
592
593 def __exit__(self, *exc_details):
594 # We don't need to duplicate any of our resource release logic
595 self.release_resource()
596
597
598Replacing any use of ``try-finally`` and flag variables
599^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
600
601A pattern you will sometimes see is a ``try-finally`` statement with a flag
602variable to indicate whether or not the body of the ``finally`` clause should
603be executed. In its simplest form (that can't already be handled just by
604using an ``except`` clause instead), it looks something like this::
605
606 cleanup_needed = True
607 try:
608 result = perform_operation()
609 if result:
610 cleanup_needed = False
611 finally:
612 if cleanup_needed:
613 cleanup_resources()
614
615As with any ``try`` statement based code, this can cause problems for
616development and review, because the setup code and the cleanup code can end
617up being separated by arbitrarily long sections of code.
618
619:class:`ExitStack` makes it possible to instead register a callback for
620execution at the end of a ``with`` statement, and then later decide to skip
621executing that callback::
622
623 from contextlib import ExitStack
624
625 with ExitStack() as stack:
626 stack.callback(cleanup_resources)
627 result = perform_operation()
628 if result:
629 stack.pop_all()
630
631This allows the intended cleanup up behaviour to be made explicit up front,
632rather than requiring a separate flag variable.
633
634If a particular application uses this pattern a lot, it can be simplified
635even further by means of a small helper class::
636
637 from contextlib import ExitStack
638
639 class Callback(ExitStack):
640 def __init__(self, callback, *args, **kwds):
641 super(Callback, self).__init__()
642 self.callback(callback, *args, **kwds)
643
644 def cancel(self):
645 self.pop_all()
646
647 with Callback(cleanup_resources) as cb:
648 result = perform_operation()
649 if result:
650 cb.cancel()
651
652If the resource cleanup isn't already neatly bundled into a standalone
653function, then it is still possible to use the decorator form of
654:meth:`ExitStack.callback` to declare the resource cleanup in
655advance::
656
657 from contextlib import ExitStack
658
659 with ExitStack() as stack:
660 @stack.callback
661 def cleanup_resources():
662 ...
663 result = perform_operation()
664 if result:
665 stack.pop_all()
666
667Due to the way the decorator protocol works, a callback function
668declared this way cannot take any parameters. Instead, any resources to
Martin Panterd21e0b52015-10-10 10:36:22 +0000669be released must be accessed as closure variables.
Nick Coghlan3267a302012-05-21 22:54:43 +1000670
671
672Using a context manager as a function decorator
673^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
674
675:class:`ContextDecorator` makes it possible to use a context manager in
676both an ordinary ``with`` statement and also as a function decorator.
677
678For example, it is sometimes useful to wrap functions or groups of statements
679with a logger that can track the time of entry and time of exit. Rather than
680writing both a function decorator and a context manager for the task,
681inheriting from :class:`ContextDecorator` provides both capabilities in a
682single definition::
683
684 from contextlib import ContextDecorator
685 import logging
686
687 logging.basicConfig(level=logging.INFO)
688
689 class track_entry_and_exit(ContextDecorator):
690 def __init__(self, name):
691 self.name = name
692
693 def __enter__(self):
Vinay Sajipdd917f82016-08-31 08:22:29 +0100694 logging.info('Entering: %s', self.name)
Nick Coghlan3267a302012-05-21 22:54:43 +1000695
696 def __exit__(self, exc_type, exc, exc_tb):
Vinay Sajipdd917f82016-08-31 08:22:29 +0100697 logging.info('Exiting: %s', self.name)
Nick Coghlan3267a302012-05-21 22:54:43 +1000698
699Instances of this class can be used as both a context manager::
700
701 with track_entry_and_exit('widget loader'):
702 print('Some time consuming activity goes here')
703 load_widget()
704
705And also as a function decorator::
706
707 @track_entry_and_exit('widget loader')
708 def activity():
709 print('Some time consuming activity goes here')
710 load_widget()
711
712Note that there is one additional limitation when using context managers
713as function decorators: there's no way to access the return value of
714:meth:`__enter__`. If that value is needed, then it is still necessary to use
715an explicit ``with`` statement.
716
Georg Brandl116aa622007-08-15 14:28:22 +0000717.. seealso::
718
Serhiy Storchakae4ba8722016-03-31 15:30:54 +0300719 :pep:`343` - The "with" statement
Georg Brandl116aa622007-08-15 14:28:22 +0000720 The specification, background, and examples for the Python :keyword:`with`
721 statement.
722
Nick Coghlan0acceb72013-10-20 13:22:21 +1000723.. _single-use-reusable-and-reentrant-cms:
Nick Coghlan8608d262013-10-20 00:30:51 +1000724
Nick Coghlan0acceb72013-10-20 13:22:21 +1000725Single use, reusable and reentrant context managers
726---------------------------------------------------
Nick Coghlan8608d262013-10-20 00:30:51 +1000727
728Most context managers are written in a way that means they can only be
729used effectively in a :keyword:`with` statement once. These single use
730context managers must be created afresh each time they're used -
731attempting to use them a second time will trigger an exception or
732otherwise not work correctly.
733
734This common limitation means that it is generally advisable to create
735context managers directly in the header of the :keyword:`with` statement
736where they are used (as shown in all of the usage examples above).
737
738Files are an example of effectively single use context managers, since
739the first :keyword:`with` statement will close the file, preventing any
740further IO operations using that file object.
741
742Context managers created using :func:`contextmanager` are also single use
743context managers, and will complain about the underlying generator failing
744to yield if an attempt is made to use them a second time::
745
746 >>> from contextlib import contextmanager
747 >>> @contextmanager
748 ... def singleuse():
749 ... print("Before")
750 ... yield
751 ... print("After")
752 ...
753 >>> cm = singleuse()
754 >>> with cm:
755 ... pass
756 ...
757 Before
758 After
759 >>> with cm:
Serhiy Storchakadba90392016-05-10 12:01:23 +0300760 ... pass
Nick Coghlan8608d262013-10-20 00:30:51 +1000761 ...
762 Traceback (most recent call last):
763 ...
764 RuntimeError: generator didn't yield
765
766
767.. _reentrant-cms:
768
769Reentrant context managers
770^^^^^^^^^^^^^^^^^^^^^^^^^^
771
772More sophisticated context managers may be "reentrant". These context
773managers can not only be used in multiple :keyword:`with` statements,
774but may also be used *inside* a :keyword:`with` statement that is already
775using the same context manager.
776
Nick Coghlan8e113b42013-11-03 17:00:51 +1000777:class:`threading.RLock` is an example of a reentrant context manager, as are
778:func:`suppress` and :func:`redirect_stdout`. Here's a very simple example of
779reentrant use::
Nick Coghlan8608d262013-10-20 00:30:51 +1000780
Nick Coghlan8e113b42013-11-03 17:00:51 +1000781 >>> from contextlib import redirect_stdout
782 >>> from io import StringIO
783 >>> stream = StringIO()
784 >>> write_to_stream = redirect_stdout(stream)
785 >>> with write_to_stream:
786 ... print("This is written to the stream rather than stdout")
787 ... with write_to_stream:
788 ... print("This is also written to the stream")
Nick Coghlan8608d262013-10-20 00:30:51 +1000789 ...
Nick Coghlan8e113b42013-11-03 17:00:51 +1000790 >>> print("This is written directly to stdout")
791 This is written directly to stdout
792 >>> print(stream.getvalue())
793 This is written to the stream rather than stdout
794 This is also written to the stream
795
796Real world examples of reentrancy are more likely to involve multiple
797functions calling each other and hence be far more complicated than this
798example.
799
800Note also that being reentrant is *not* the same thing as being thread safe.
801:func:`redirect_stdout`, for example, is definitely not thread safe, as it
802makes a global modification to the system state by binding :data:`sys.stdout`
803to a different stream.
Nick Coghlan8608d262013-10-20 00:30:51 +1000804
805
806.. _reusable-cms:
807
808Reusable context managers
809^^^^^^^^^^^^^^^^^^^^^^^^^
810
811Distinct from both single use and reentrant context managers are "reusable"
812context managers (or, to be completely explicit, "reusable, but not
813reentrant" context managers, since reentrant context managers are also
814reusable). These context managers support being used multiple times, but
815will fail (or otherwise not work correctly) if the specific context manager
816instance has already been used in a containing with statement.
817
Nick Coghlan8e113b42013-11-03 17:00:51 +1000818:class:`threading.Lock` is an example of a reusable, but not reentrant,
819context manager (for a reentrant lock, it is necessary to use
820:class:`threading.RLock` instead).
Nick Coghlan8608d262013-10-20 00:30:51 +1000821
Nick Coghlan8e113b42013-11-03 17:00:51 +1000822Another example of a reusable, but not reentrant, context manager is
823:class:`ExitStack`, as it invokes *all* currently registered callbacks
824when leaving any with statement, regardless of where those callbacks
825were added::
Nick Coghlan8608d262013-10-20 00:30:51 +1000826
Nick Coghlan8e113b42013-11-03 17:00:51 +1000827 >>> from contextlib import ExitStack
828 >>> stack = ExitStack()
829 >>> with stack:
830 ... stack.callback(print, "Callback: from first context")
831 ... print("Leaving first context")
Nick Coghlan8608d262013-10-20 00:30:51 +1000832 ...
Nick Coghlan8e113b42013-11-03 17:00:51 +1000833 Leaving first context
834 Callback: from first context
835 >>> with stack:
836 ... stack.callback(print, "Callback: from second context")
837 ... print("Leaving second context")
838 ...
839 Leaving second context
840 Callback: from second context
841 >>> with stack:
842 ... stack.callback(print, "Callback: from outer context")
843 ... with stack:
844 ... stack.callback(print, "Callback: from inner context")
845 ... print("Leaving inner context")
846 ... print("Leaving outer context")
847 ...
848 Leaving inner context
849 Callback: from inner context
850 Callback: from outer context
851 Leaving outer context
852
853As the output from the example shows, reusing a single stack object across
854multiple with statements works correctly, but attempting to nest them
855will cause the stack to be cleared at the end of the innermost with
856statement, which is unlikely to be desirable behaviour.
857
858Using separate :class:`ExitStack` instances instead of reusing a single
859instance avoids that problem::
860
861 >>> from contextlib import ExitStack
862 >>> with ExitStack() as outer_stack:
863 ... outer_stack.callback(print, "Callback: from outer context")
864 ... with ExitStack() as inner_stack:
865 ... inner_stack.callback(print, "Callback: from inner context")
866 ... print("Leaving inner context")
867 ... print("Leaving outer context")
868 ...
869 Leaving inner context
870 Callback: from inner context
871 Leaving outer context
872 Callback: from outer context