blob: 0b1f4f77dcc8635a20e4e4ce44667e13af3e2f1d [file] [log] [blame]
Georg Brandl0eaab972009-06-08 08:00:22 +00001:mod:`contextlib` --- Utilities for :keyword:`with`\ -statement contexts
2========================================================================
Georg Brandl116aa622007-08-15 14:28:22 +00003
4.. module:: contextlib
5 :synopsis: Utilities for with-statement contexts.
6
Raymond Hettinger10480942011-01-10 03:26:08 +00007**Source code:** :source:`Lib/contextlib.py`
Georg Brandl116aa622007-08-15 14:28:22 +00008
Raymond Hettinger4f707fd2011-01-10 19:54:11 +00009--------------
10
Georg Brandl116aa622007-08-15 14:28:22 +000011This module provides utilities for common tasks involving the :keyword:`with`
12statement. For more information see also :ref:`typecontextmanager` and
13:ref:`context-managers`.
14
Georg Brandl116aa622007-08-15 14:28:22 +000015
Nick Coghlan3267a302012-05-21 22:54:43 +100016Utilities
17---------
18
19Functions and classes provided:
Georg Brandl116aa622007-08-15 14:28:22 +000020
Brett Cannon9e080e02016-04-08 12:15:27 -070021.. class:: AbstractContextManager
22
Brett Cannon516f5462016-06-09 15:55:52 -070023 An :term:`abstract base class` for classes that implement
Brett Cannon9e080e02016-04-08 12:15:27 -070024 :meth:`object.__enter__` and :meth:`object.__exit__`. A default
25 implementation for :meth:`object.__enter__` is provided which returns
26 ``self`` while :meth:`object.__exit__` is an abstract method which by default
27 returns ``None``. See also the definition of :ref:`typecontextmanager`.
28
29 .. versionadded:: 3.6
30
31
Jelle Zijlstra176baa32017-12-13 17:19:17 -080032.. class:: AbstractAsyncContextManager
33
34 An :term:`abstract base class` for classes that implement
35 :meth:`object.__aenter__` and :meth:`object.__aexit__`. A default
36 implementation for :meth:`object.__aenter__` is provided which returns
37 ``self`` while :meth:`object.__aexit__` is an abstract method which by default
38 returns ``None``. See also the definition of
39 :ref:`async-context-managers`.
40
41 .. versionadded:: 3.7
42
Brett Cannon9e080e02016-04-08 12:15:27 -070043
Georg Brandl8a1caa22010-07-29 16:01:11 +000044.. decorator:: contextmanager
Georg Brandl116aa622007-08-15 14:28:22 +000045
Christian Heimesd8654cf2007-12-02 15:22:16 +000046 This function is a :term:`decorator` that can be used to define a factory
47 function for :keyword:`with` statement context managers, without needing to
48 create a class or separate :meth:`__enter__` and :meth:`__exit__` methods.
Georg Brandl116aa622007-08-15 14:28:22 +000049
50 A simple example (this is not recommended as a real way of generating HTML!)::
51
Georg Brandl116aa622007-08-15 14:28:22 +000052 from contextlib import contextmanager
53
54 @contextmanager
55 def tag(name):
Georg Brandl6911e3c2007-09-04 07:15:32 +000056 print("<%s>" % name)
Georg Brandl116aa622007-08-15 14:28:22 +000057 yield
Georg Brandl6911e3c2007-09-04 07:15:32 +000058 print("</%s>" % name)
Georg Brandl116aa622007-08-15 14:28:22 +000059
60 >>> with tag("h1"):
Georg Brandl6911e3c2007-09-04 07:15:32 +000061 ... print("foo")
Georg Brandl116aa622007-08-15 14:28:22 +000062 ...
63 <h1>
64 foo
65 </h1>
66
Georg Brandl9afde1c2007-11-01 20:32:30 +000067 The function being decorated must return a :term:`generator`-iterator when
68 called. This iterator must yield exactly one value, which will be bound to
69 the targets in the :keyword:`with` statement's :keyword:`as` clause, if any.
Georg Brandl116aa622007-08-15 14:28:22 +000070
71 At the point where the generator yields, the block nested in the :keyword:`with`
72 statement is executed. The generator is then resumed after the block is exited.
73 If an unhandled exception occurs in the block, it is reraised inside the
74 generator at the point where the yield occurred. Thus, you can use a
75 :keyword:`try`...\ :keyword:`except`...\ :keyword:`finally` statement to trap
76 the error (if any), or ensure that some cleanup takes place. If an exception is
77 trapped merely in order to log it or to perform some action (rather than to
78 suppress it entirely), the generator must reraise that exception. Otherwise the
79 generator context manager will indicate to the :keyword:`with` statement that
80 the exception has been handled, and execution will resume with the statement
81 immediately following the :keyword:`with` statement.
82
Nick Coghlan0ded3e32011-05-05 23:49:25 +100083 :func:`contextmanager` uses :class:`ContextDecorator` so the context managers
84 it creates can be used as decorators as well as in :keyword:`with` statements.
85 When used as a decorator, a new generator instance is implicitly created on
86 each function call (this allows the otherwise "one-shot" context managers
87 created by :func:`contextmanager` to meet the requirement that context
88 managers support multiple invocations in order to be used as decorators).
Michael Foordb3a89842010-06-30 12:17:50 +000089
90 .. versionchanged:: 3.2
91 Use of :class:`ContextDecorator`.
Georg Brandl116aa622007-08-15 14:28:22 +000092
Georg Brandl86e78d12010-07-18 13:43:32 +000093
Jelle Zijlstra2e624692017-04-30 18:25:58 -070094.. decorator:: asynccontextmanager
95
96 Similar to :func:`~contextlib.contextmanager`, but creates an
97 :ref:`asynchronous context manager <async-context-managers>`.
98
99 This function is a :term:`decorator` that can be used to define a factory
100 function for :keyword:`async with` statement asynchronous context managers,
101 without needing to create a class or separate :meth:`__aenter__` and
102 :meth:`__aexit__` methods. It must be applied to an :term:`asynchronous
103 generator` function.
104
105 A simple example::
106
107 from contextlib import asynccontextmanager
108
109 @asynccontextmanager
110 async def get_connection():
111 conn = await acquire_db_connection()
112 try:
113 yield
114 finally:
115 await release_db_connection(conn)
116
117 async def get_all_users():
118 async with get_connection() as conn:
119 return conn.query('SELECT ...')
120
121 .. versionadded:: 3.7
122
123
Georg Brandl116aa622007-08-15 14:28:22 +0000124.. function:: closing(thing)
125
126 Return a context manager that closes *thing* upon completion of the block. This
127 is basically equivalent to::
128
129 from contextlib import contextmanager
130
131 @contextmanager
132 def closing(thing):
133 try:
134 yield thing
135 finally:
136 thing.close()
137
138 And lets you write code like this::
139
Georg Brandl116aa622007-08-15 14:28:22 +0000140 from contextlib import closing
Georg Brandl0f7ede42008-06-23 11:23:31 +0000141 from urllib.request import urlopen
Georg Brandl116aa622007-08-15 14:28:22 +0000142
Georg Brandl0f7ede42008-06-23 11:23:31 +0000143 with closing(urlopen('http://www.python.org')) as page:
Georg Brandl116aa622007-08-15 14:28:22 +0000144 for line in page:
Georg Brandl6911e3c2007-09-04 07:15:32 +0000145 print(line)
Georg Brandl116aa622007-08-15 14:28:22 +0000146
147 without needing to explicitly close ``page``. Even if an error occurs,
148 ``page.close()`` will be called when the :keyword:`with` block is exited.
149
Georg Brandla7c17e52013-10-13 22:25:10 +0200150
Jesse-Bakker0784a2e2017-11-23 01:23:28 +0100151.. _simplifying-support-for-single-optional-context-managers:
152
153.. function:: nullcontext(enter_result=None)
154
Miss Islington (bot)1b6e2172018-07-09 07:11:42 -0700155 Return a context manager that returns *enter_result* from ``__enter__``, but
Jesse-Bakker0784a2e2017-11-23 01:23:28 +0100156 otherwise does nothing. It is intended to be used as a stand-in for an
157 optional context manager, for example::
158
Miss Islington (bot)1b6e2172018-07-09 07:11:42 -0700159 def myfunction(arg, ignore_exceptions=False):
160 if ignore_exceptions:
161 # Use suppress to ignore all exceptions.
162 cm = contextlib.suppress(Exception)
163 else:
164 # Do not ignore any exceptions, cm has no effect.
165 cm = contextlib.nullcontext()
166 with cm:
167 # Do something
168
169 An example using *enter_result*::
170
Jesse-Bakker0784a2e2017-11-23 01:23:28 +0100171 def process_file(file_or_path):
172 if isinstance(file_or_path, str):
173 # If string, open file
174 cm = open(file_or_path)
175 else:
176 # Caller is responsible for closing file
177 cm = nullcontext(file_or_path)
178
179 with cm as file:
180 # Perform processing on the file
181
182 .. versionadded:: 3.7
183
184
Nick Coghlan240f86d2013-10-17 23:40:57 +1000185.. function:: suppress(*exceptions)
Raymond Hettingere318a882013-03-10 22:26:51 -0700186
Nick Coghlan240f86d2013-10-17 23:40:57 +1000187 Return a context manager that suppresses any of the specified exceptions
188 if they occur in the body of a with statement and then resumes execution
189 with the first statement following the end of the with statement.
Raymond Hettingere318a882013-03-10 22:26:51 -0700190
Nick Coghlan240f86d2013-10-17 23:40:57 +1000191 As with any other mechanism that completely suppresses exceptions, this
192 context manager should be used only to cover very specific errors where
193 silently continuing with program execution is known to be the right
194 thing to do.
Nick Coghlanb4534ae2013-10-13 23:23:08 +1000195
Raymond Hettingere318a882013-03-10 22:26:51 -0700196 For example::
197
Nick Coghlan240f86d2013-10-17 23:40:57 +1000198 from contextlib import suppress
Raymond Hettingere318a882013-03-10 22:26:51 -0700199
Nick Coghlan240f86d2013-10-17 23:40:57 +1000200 with suppress(FileNotFoundError):
Raymond Hettingere318a882013-03-10 22:26:51 -0700201 os.remove('somefile.tmp')
202
Nick Coghlan240f86d2013-10-17 23:40:57 +1000203 with suppress(FileNotFoundError):
204 os.remove('someotherfile.tmp')
205
Raymond Hettingere318a882013-03-10 22:26:51 -0700206 This code is equivalent to::
207
208 try:
209 os.remove('somefile.tmp')
Nick Coghlanb4534ae2013-10-13 23:23:08 +1000210 except FileNotFoundError:
Raymond Hettingere318a882013-03-10 22:26:51 -0700211 pass
212
Nick Coghlan240f86d2013-10-17 23:40:57 +1000213 try:
214 os.remove('someotherfile.tmp')
215 except FileNotFoundError:
216 pass
217
Nick Coghlan8608d262013-10-20 00:30:51 +1000218 This context manager is :ref:`reentrant <reentrant-cms>`.
219
Raymond Hettingere318a882013-03-10 22:26:51 -0700220 .. versionadded:: 3.4
Georg Brandl116aa622007-08-15 14:28:22 +0000221
Nick Coghlanb4534ae2013-10-13 23:23:08 +1000222
Raymond Hettinger088cbf22013-10-10 00:46:57 -0700223.. function:: redirect_stdout(new_target)
224
225 Context manager for temporarily redirecting :data:`sys.stdout` to
226 another file or file-like object.
227
228 This tool adds flexibility to existing functions or classes whose output
229 is hardwired to stdout.
230
231 For example, the output of :func:`help` normally is sent to *sys.stdout*.
Serhiy Storchakad65c9492015-11-02 14:10:23 +0200232 You can capture that output in a string by redirecting the output to an
Raymond Hettinger088cbf22013-10-10 00:46:57 -0700233 :class:`io.StringIO` object::
234
235 f = io.StringIO()
236 with redirect_stdout(f):
237 help(pow)
238 s = f.getvalue()
239
240 To send the output of :func:`help` to a file on disk, redirect the output
241 to a regular file::
242
243 with open('help.txt', 'w') as f:
244 with redirect_stdout(f):
245 help(pow)
246
247 To send the output of :func:`help` to *sys.stderr*::
248
249 with redirect_stdout(sys.stderr):
250 help(pow)
251
Nick Coghlanb4534ae2013-10-13 23:23:08 +1000252 Note that the global side effect on :data:`sys.stdout` means that this
253 context manager is not suitable for use in library code and most threaded
254 applications. It also has no effect on the output of subprocesses.
255 However, it is still a useful approach for many utility scripts.
256
Nick Coghlan36d8ef92014-10-12 10:25:00 +1000257 This context manager is :ref:`reentrant <reentrant-cms>`.
Nick Coghlan8608d262013-10-20 00:30:51 +1000258
Raymond Hettinger088cbf22013-10-10 00:46:57 -0700259 .. versionadded:: 3.4
260
Georg Brandla7c17e52013-10-13 22:25:10 +0200261
Berker Peksagbb44fe02014-11-28 23:28:06 +0200262.. function:: redirect_stderr(new_target)
263
264 Similar to :func:`~contextlib.redirect_stdout` but redirecting
265 :data:`sys.stderr` to another file or file-like object.
266
267 This context manager is :ref:`reentrant <reentrant-cms>`.
268
269 .. versionadded:: 3.5
270
271
Michael Foordb3a89842010-06-30 12:17:50 +0000272.. class:: ContextDecorator()
273
274 A base class that enables a context manager to also be used as a decorator.
275
276 Context managers inheriting from ``ContextDecorator`` have to implement
277 ``__enter__`` and ``__exit__`` as normal. ``__exit__`` retains its optional
278 exception handling even when used as a decorator.
279
Georg Brandl86e78d12010-07-18 13:43:32 +0000280 ``ContextDecorator`` is used by :func:`contextmanager`, so you get this
281 functionality automatically.
282
283 Example of ``ContextDecorator``::
Michael Foordb3a89842010-06-30 12:17:50 +0000284
285 from contextlib import ContextDecorator
286
287 class mycontext(ContextDecorator):
Georg Brandl86e78d12010-07-18 13:43:32 +0000288 def __enter__(self):
289 print('Starting')
290 return self
Michael Foordb3a89842010-06-30 12:17:50 +0000291
Georg Brandl86e78d12010-07-18 13:43:32 +0000292 def __exit__(self, *exc):
293 print('Finishing')
294 return False
Michael Foordb3a89842010-06-30 12:17:50 +0000295
296 >>> @mycontext()
297 ... def function():
Georg Brandl86e78d12010-07-18 13:43:32 +0000298 ... print('The bit in the middle')
Michael Foordb3a89842010-06-30 12:17:50 +0000299 ...
300 >>> function()
301 Starting
302 The bit in the middle
303 Finishing
304
305 >>> with mycontext():
Georg Brandl86e78d12010-07-18 13:43:32 +0000306 ... print('The bit in the middle')
Michael Foordb3a89842010-06-30 12:17:50 +0000307 ...
308 Starting
309 The bit in the middle
310 Finishing
311
Georg Brandl86e78d12010-07-18 13:43:32 +0000312 This change is just syntactic sugar for any construct of the following form::
313
314 def f():
315 with cm():
316 # Do stuff
317
318 ``ContextDecorator`` lets you instead write::
319
320 @cm()
321 def f():
322 # Do stuff
323
324 It makes it clear that the ``cm`` applies to the whole function, rather than
325 just a piece of it (and saving an indentation level is nice, too).
326
Michael Foordb3a89842010-06-30 12:17:50 +0000327 Existing context managers that already have a base class can be extended by
328 using ``ContextDecorator`` as a mixin class::
329
330 from contextlib import ContextDecorator
331
332 class mycontext(ContextBaseClass, ContextDecorator):
Georg Brandl86e78d12010-07-18 13:43:32 +0000333 def __enter__(self):
334 return self
Michael Foordb3a89842010-06-30 12:17:50 +0000335
Georg Brandl86e78d12010-07-18 13:43:32 +0000336 def __exit__(self, *exc):
337 return False
Michael Foordb3a89842010-06-30 12:17:50 +0000338
Nick Coghlan0ded3e32011-05-05 23:49:25 +1000339 .. note::
340 As the decorated function must be able to be called multiple times, the
341 underlying context manager must support use in multiple :keyword:`with`
342 statements. If this is not the case, then the original construct with the
343 explicit :keyword:`with` statement inside the function should be used.
344
Michael Foordb3a89842010-06-30 12:17:50 +0000345 .. versionadded:: 3.2
346
347
Nick Coghlan3267a302012-05-21 22:54:43 +1000348.. class:: ExitStack()
349
350 A context manager that is designed to make it easy to programmatically
351 combine other context managers and cleanup functions, especially those
352 that are optional or otherwise driven by input data.
353
354 For example, a set of files may easily be handled in a single with
355 statement as follows::
356
357 with ExitStack() as stack:
358 files = [stack.enter_context(open(fname)) for fname in filenames]
359 # All opened files will automatically be closed at the end of
360 # the with statement, even if attempts to open files later
Andrew Svetlov5b898402012-12-18 21:26:36 +0200361 # in the list raise an exception
Nick Coghlan3267a302012-05-21 22:54:43 +1000362
363 Each instance maintains a stack of registered callbacks that are called in
364 reverse order when the instance is closed (either explicitly or implicitly
Nick Coghlan27228272012-05-31 22:17:08 +1000365 at the end of a :keyword:`with` statement). Note that callbacks are *not*
366 invoked implicitly when the context stack instance is garbage collected.
Nick Coghlan3267a302012-05-21 22:54:43 +1000367
368 This stack model is used so that context managers that acquire their
369 resources in their ``__init__`` method (such as file objects) can be
370 handled correctly.
371
372 Since registered callbacks are invoked in the reverse order of
Nick Coghlan27228272012-05-31 22:17:08 +1000373 registration, this ends up behaving as if multiple nested :keyword:`with`
Nick Coghlan3267a302012-05-21 22:54:43 +1000374 statements had been used with the registered set of callbacks. This even
375 extends to exception handling - if an inner callback suppresses or replaces
376 an exception, then outer callbacks will be passed arguments based on that
377 updated state.
378
379 This is a relatively low level API that takes care of the details of
380 correctly unwinding the stack of exit callbacks. It provides a suitable
381 foundation for higher level context managers that manipulate the exit
382 stack in application specific ways.
383
Nick Coghlana497b442012-05-22 23:02:00 +1000384 .. versionadded:: 3.3
385
Nick Coghlan3267a302012-05-21 22:54:43 +1000386 .. method:: enter_context(cm)
387
388 Enters a new context manager and adds its :meth:`__exit__` method to
389 the callback stack. The return value is the result of the context
390 manager's own :meth:`__enter__` method.
391
392 These context managers may suppress exceptions just as they normally
Nick Coghlan27228272012-05-31 22:17:08 +1000393 would if used directly as part of a :keyword:`with` statement.
Nick Coghlan3267a302012-05-21 22:54:43 +1000394
395 .. method:: push(exit)
396
397 Adds a context manager's :meth:`__exit__` method to the callback stack.
398
399 As ``__enter__`` is *not* invoked, this method can be used to cover
400 part of an :meth:`__enter__` implementation with a context manager's own
401 :meth:`__exit__` method.
402
403 If passed an object that is not a context manager, this method assumes
404 it is a callback with the same signature as a context manager's
405 :meth:`__exit__` method and adds it directly to the callback stack.
406
407 By returning true values, these callbacks can suppress exceptions the
408 same way context manager :meth:`__exit__` methods can.
409
410 The passed in object is returned from the function, allowing this
Nick Coghlan27228272012-05-31 22:17:08 +1000411 method to be used as a function decorator.
Nick Coghlan3267a302012-05-21 22:54:43 +1000412
413 .. method:: callback(callback, *args, **kwds)
414
415 Accepts an arbitrary callback function and arguments and adds it to
416 the callback stack.
417
418 Unlike the other methods, callbacks added this way cannot suppress
419 exceptions (as they are never passed the exception details).
420
421 The passed in callback is returned from the function, allowing this
Nick Coghlan27228272012-05-31 22:17:08 +1000422 method to be used as a function decorator.
Nick Coghlan3267a302012-05-21 22:54:43 +1000423
424 .. method:: pop_all()
425
426 Transfers the callback stack to a fresh :class:`ExitStack` instance
427 and returns it. No callbacks are invoked by this operation - instead,
428 they will now be invoked when the new stack is closed (either
Nick Coghlan27228272012-05-31 22:17:08 +1000429 explicitly or implicitly at the end of a :keyword:`with` statement).
Nick Coghlan3267a302012-05-21 22:54:43 +1000430
431 For example, a group of files can be opened as an "all or nothing"
432 operation as follows::
433
434 with ExitStack() as stack:
435 files = [stack.enter_context(open(fname)) for fname in filenames]
Barry Warsawd8f870d2013-05-10 11:35:38 -0400436 # Hold onto the close method, but don't call it yet.
437 close_files = stack.pop_all().close
Nick Coghlan3267a302012-05-21 22:54:43 +1000438 # If opening any file fails, all previously opened files will be
439 # closed automatically. If all files are opened successfully,
440 # they will remain open even after the with statement ends.
Barry Warsawd8f870d2013-05-10 11:35:38 -0400441 # close_files() can then be invoked explicitly to close them all.
Nick Coghlan3267a302012-05-21 22:54:43 +1000442
443 .. method:: close()
444
445 Immediately unwinds the callback stack, invoking callbacks in the
446 reverse order of registration. For any context managers and exit
447 callbacks registered, the arguments passed in will indicate that no
448 exception occurred.
449
Ilya Kulakov1aa094f2018-01-25 12:51:18 -0800450.. class:: AsyncExitStack()
451
452 An :ref:`asynchronous context manager <async-context-managers>`, similar
453 to :class:`ExitStack`, that supports combining both synchronous and
454 asynchronous context managers, as well as having coroutines for
455 cleanup logic.
456
457 The :meth:`close` method is not implemented, :meth:`aclose` must be used
458 instead.
459
460 .. method:: enter_async_context(cm)
461
462 Similar to :meth:`enter_context` but expects an asynchronous context
463 manager.
464
465 .. method:: push_async_exit(exit)
466
467 Similar to :meth:`push` but expects either an asynchronous context manager
468 or a coroutine.
469
470 .. method:: push_async_callback(callback, *args, **kwds)
471
472 Similar to :meth:`callback` but expects a coroutine.
473
474 .. method:: aclose()
475
476 Similar to :meth:`close` but properly handles awaitables.
477
478 Continuing the example for :func:`asynccontextmanager`::
479
480 async with AsyncExitStack() as stack:
481 connections = [await stack.enter_async_context(get_connection())
482 for i in range(5)]
483 # All opened connections will automatically be released at the end of
484 # the async with statement, even if attempts to open a connection
485 # later in the list raise an exception.
486
487 .. versionadded:: 3.7
Nick Coghlan3267a302012-05-21 22:54:43 +1000488
489Examples and Recipes
490--------------------
491
492This section describes some examples and recipes for making effective use of
493the tools provided by :mod:`contextlib`.
494
495
Nick Coghlan27228272012-05-31 22:17:08 +1000496Supporting a variable number of context managers
497^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
498
499The primary use case for :class:`ExitStack` is the one given in the class
500documentation: supporting a variable number of context managers and other
501cleanup operations in a single :keyword:`with` statement. The variability
502may come from the number of context managers needed being driven by user
503input (such as opening a user specified collection of files), or from
504some of the context managers being optional::
505
506 with ExitStack() as stack:
507 for resource in resources:
508 stack.enter_context(resource)
Raymond Hettingere8e2df32014-05-25 18:06:04 -0700509 if need_special_resource():
Nick Coghlan27228272012-05-31 22:17:08 +1000510 special = acquire_special_resource()
511 stack.callback(release_special_resource, special)
512 # Perform operations that use the acquired resources
513
514As shown, :class:`ExitStack` also makes it quite easy to use :keyword:`with`
515statements to manage arbitrary resources that don't natively support the
516context management protocol.
517
518
Nick Coghlan27228272012-05-31 22:17:08 +1000519Catching exceptions from ``__enter__`` methods
520^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
521
522It is occasionally desirable to catch exceptions from an ``__enter__``
523method implementation, *without* inadvertently catching exceptions from
524the :keyword:`with` statement body or the context manager's ``__exit__``
525method. By using :class:`ExitStack` the steps in the context management
526protocol can be separated slightly in order to allow this::
527
528 stack = ExitStack()
529 try:
530 x = stack.enter_context(cm)
531 except Exception:
532 # handle __enter__ exception
533 else:
534 with stack:
535 # Handle normal case
536
537Actually needing to do this is likely to indicate that the underlying API
538should be providing a direct resource management interface for use with
539:keyword:`try`/:keyword:`except`/:keyword:`finally` statements, but not
540all APIs are well designed in that regard. When a context manager is the
541only resource management API provided, then :class:`ExitStack` can make it
542easier to handle various situations that can't be handled directly in a
543:keyword:`with` statement.
544
545
Nick Coghlan3267a302012-05-21 22:54:43 +1000546Cleaning up in an ``__enter__`` implementation
547^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
548
549As noted in the documentation of :meth:`ExitStack.push`, this
550method can be useful in cleaning up an already allocated resource if later
551steps in the :meth:`__enter__` implementation fail.
552
553Here's an example of doing this for a context manager that accepts resource
554acquisition and release functions, along with an optional validation function,
555and maps them to the context management protocol::
556
Brett Cannon9e080e02016-04-08 12:15:27 -0700557 from contextlib import contextmanager, AbstractContextManager, ExitStack
Nick Coghlan3267a302012-05-21 22:54:43 +1000558
Brett Cannon9e080e02016-04-08 12:15:27 -0700559 class ResourceManager(AbstractContextManager):
Nick Coghlan3267a302012-05-21 22:54:43 +1000560
561 def __init__(self, acquire_resource, release_resource, check_resource_ok=None):
562 self.acquire_resource = acquire_resource
563 self.release_resource = release_resource
564 if check_resource_ok is None:
565 def check_resource_ok(resource):
566 return True
567 self.check_resource_ok = check_resource_ok
568
569 @contextmanager
570 def _cleanup_on_error(self):
571 with ExitStack() as stack:
572 stack.push(self)
573 yield
574 # The validation check passed and didn't raise an exception
575 # Accordingly, we want to keep the resource, and pass it
576 # back to our caller
577 stack.pop_all()
578
579 def __enter__(self):
580 resource = self.acquire_resource()
581 with self._cleanup_on_error():
582 if not self.check_resource_ok(resource):
583 msg = "Failed validation for {!r}"
584 raise RuntimeError(msg.format(resource))
585 return resource
586
587 def __exit__(self, *exc_details):
588 # We don't need to duplicate any of our resource release logic
589 self.release_resource()
590
591
592Replacing any use of ``try-finally`` and flag variables
593^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
594
595A pattern you will sometimes see is a ``try-finally`` statement with a flag
596variable to indicate whether or not the body of the ``finally`` clause should
597be executed. In its simplest form (that can't already be handled just by
598using an ``except`` clause instead), it looks something like this::
599
600 cleanup_needed = True
601 try:
602 result = perform_operation()
603 if result:
604 cleanup_needed = False
605 finally:
606 if cleanup_needed:
607 cleanup_resources()
608
609As with any ``try`` statement based code, this can cause problems for
610development and review, because the setup code and the cleanup code can end
611up being separated by arbitrarily long sections of code.
612
613:class:`ExitStack` makes it possible to instead register a callback for
614execution at the end of a ``with`` statement, and then later decide to skip
615executing that callback::
616
617 from contextlib import ExitStack
618
619 with ExitStack() as stack:
620 stack.callback(cleanup_resources)
621 result = perform_operation()
622 if result:
623 stack.pop_all()
624
625This allows the intended cleanup up behaviour to be made explicit up front,
626rather than requiring a separate flag variable.
627
628If a particular application uses this pattern a lot, it can be simplified
629even further by means of a small helper class::
630
631 from contextlib import ExitStack
632
633 class Callback(ExitStack):
634 def __init__(self, callback, *args, **kwds):
635 super(Callback, self).__init__()
636 self.callback(callback, *args, **kwds)
637
638 def cancel(self):
639 self.pop_all()
640
641 with Callback(cleanup_resources) as cb:
642 result = perform_operation()
643 if result:
644 cb.cancel()
645
646If the resource cleanup isn't already neatly bundled into a standalone
647function, then it is still possible to use the decorator form of
648:meth:`ExitStack.callback` to declare the resource cleanup in
649advance::
650
651 from contextlib import ExitStack
652
653 with ExitStack() as stack:
654 @stack.callback
655 def cleanup_resources():
656 ...
657 result = perform_operation()
658 if result:
659 stack.pop_all()
660
661Due to the way the decorator protocol works, a callback function
662declared this way cannot take any parameters. Instead, any resources to
Martin Panterd21e0b52015-10-10 10:36:22 +0000663be released must be accessed as closure variables.
Nick Coghlan3267a302012-05-21 22:54:43 +1000664
665
666Using a context manager as a function decorator
667^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
668
669:class:`ContextDecorator` makes it possible to use a context manager in
670both an ordinary ``with`` statement and also as a function decorator.
671
672For example, it is sometimes useful to wrap functions or groups of statements
673with a logger that can track the time of entry and time of exit. Rather than
674writing both a function decorator and a context manager for the task,
675inheriting from :class:`ContextDecorator` provides both capabilities in a
676single definition::
677
678 from contextlib import ContextDecorator
679 import logging
680
681 logging.basicConfig(level=logging.INFO)
682
683 class track_entry_and_exit(ContextDecorator):
684 def __init__(self, name):
685 self.name = name
686
687 def __enter__(self):
Vinay Sajipdd917f82016-08-31 08:22:29 +0100688 logging.info('Entering: %s', self.name)
Nick Coghlan3267a302012-05-21 22:54:43 +1000689
690 def __exit__(self, exc_type, exc, exc_tb):
Vinay Sajipdd917f82016-08-31 08:22:29 +0100691 logging.info('Exiting: %s', self.name)
Nick Coghlan3267a302012-05-21 22:54:43 +1000692
693Instances of this class can be used as both a context manager::
694
695 with track_entry_and_exit('widget loader'):
696 print('Some time consuming activity goes here')
697 load_widget()
698
699And also as a function decorator::
700
701 @track_entry_and_exit('widget loader')
702 def activity():
703 print('Some time consuming activity goes here')
704 load_widget()
705
706Note that there is one additional limitation when using context managers
707as function decorators: there's no way to access the return value of
708:meth:`__enter__`. If that value is needed, then it is still necessary to use
709an explicit ``with`` statement.
710
Georg Brandl116aa622007-08-15 14:28:22 +0000711.. seealso::
712
Serhiy Storchakae4ba8722016-03-31 15:30:54 +0300713 :pep:`343` - The "with" statement
Georg Brandl116aa622007-08-15 14:28:22 +0000714 The specification, background, and examples for the Python :keyword:`with`
715 statement.
716
Nick Coghlan0acceb72013-10-20 13:22:21 +1000717.. _single-use-reusable-and-reentrant-cms:
Nick Coghlan8608d262013-10-20 00:30:51 +1000718
Nick Coghlan0acceb72013-10-20 13:22:21 +1000719Single use, reusable and reentrant context managers
720---------------------------------------------------
Nick Coghlan8608d262013-10-20 00:30:51 +1000721
722Most context managers are written in a way that means they can only be
723used effectively in a :keyword:`with` statement once. These single use
724context managers must be created afresh each time they're used -
725attempting to use them a second time will trigger an exception or
726otherwise not work correctly.
727
728This common limitation means that it is generally advisable to create
729context managers directly in the header of the :keyword:`with` statement
730where they are used (as shown in all of the usage examples above).
731
732Files are an example of effectively single use context managers, since
733the first :keyword:`with` statement will close the file, preventing any
734further IO operations using that file object.
735
736Context managers created using :func:`contextmanager` are also single use
737context managers, and will complain about the underlying generator failing
738to yield if an attempt is made to use them a second time::
739
740 >>> from contextlib import contextmanager
741 >>> @contextmanager
742 ... def singleuse():
743 ... print("Before")
744 ... yield
745 ... print("After")
746 ...
747 >>> cm = singleuse()
748 >>> with cm:
749 ... pass
750 ...
751 Before
752 After
753 >>> with cm:
Serhiy Storchakadba90392016-05-10 12:01:23 +0300754 ... pass
Nick Coghlan8608d262013-10-20 00:30:51 +1000755 ...
756 Traceback (most recent call last):
757 ...
758 RuntimeError: generator didn't yield
759
760
761.. _reentrant-cms:
762
763Reentrant context managers
764^^^^^^^^^^^^^^^^^^^^^^^^^^
765
766More sophisticated context managers may be "reentrant". These context
767managers can not only be used in multiple :keyword:`with` statements,
768but may also be used *inside* a :keyword:`with` statement that is already
769using the same context manager.
770
Nick Coghlan8e113b42013-11-03 17:00:51 +1000771:class:`threading.RLock` is an example of a reentrant context manager, as are
772:func:`suppress` and :func:`redirect_stdout`. Here's a very simple example of
773reentrant use::
Nick Coghlan8608d262013-10-20 00:30:51 +1000774
Nick Coghlan8e113b42013-11-03 17:00:51 +1000775 >>> from contextlib import redirect_stdout
776 >>> from io import StringIO
777 >>> stream = StringIO()
778 >>> write_to_stream = redirect_stdout(stream)
779 >>> with write_to_stream:
780 ... print("This is written to the stream rather than stdout")
781 ... with write_to_stream:
782 ... print("This is also written to the stream")
Nick Coghlan8608d262013-10-20 00:30:51 +1000783 ...
Nick Coghlan8e113b42013-11-03 17:00:51 +1000784 >>> print("This is written directly to stdout")
785 This is written directly to stdout
786 >>> print(stream.getvalue())
787 This is written to the stream rather than stdout
788 This is also written to the stream
789
790Real world examples of reentrancy are more likely to involve multiple
791functions calling each other and hence be far more complicated than this
792example.
793
794Note also that being reentrant is *not* the same thing as being thread safe.
795:func:`redirect_stdout`, for example, is definitely not thread safe, as it
796makes a global modification to the system state by binding :data:`sys.stdout`
797to a different stream.
Nick Coghlan8608d262013-10-20 00:30:51 +1000798
799
800.. _reusable-cms:
801
802Reusable context managers
803^^^^^^^^^^^^^^^^^^^^^^^^^
804
805Distinct from both single use and reentrant context managers are "reusable"
806context managers (or, to be completely explicit, "reusable, but not
807reentrant" context managers, since reentrant context managers are also
808reusable). These context managers support being used multiple times, but
809will fail (or otherwise not work correctly) if the specific context manager
810instance has already been used in a containing with statement.
811
Nick Coghlan8e113b42013-11-03 17:00:51 +1000812:class:`threading.Lock` is an example of a reusable, but not reentrant,
813context manager (for a reentrant lock, it is necessary to use
814:class:`threading.RLock` instead).
Nick Coghlan8608d262013-10-20 00:30:51 +1000815
Nick Coghlan8e113b42013-11-03 17:00:51 +1000816Another example of a reusable, but not reentrant, context manager is
817:class:`ExitStack`, as it invokes *all* currently registered callbacks
818when leaving any with statement, regardless of where those callbacks
819were added::
Nick Coghlan8608d262013-10-20 00:30:51 +1000820
Nick Coghlan8e113b42013-11-03 17:00:51 +1000821 >>> from contextlib import ExitStack
822 >>> stack = ExitStack()
823 >>> with stack:
824 ... stack.callback(print, "Callback: from first context")
825 ... print("Leaving first context")
Nick Coghlan8608d262013-10-20 00:30:51 +1000826 ...
Nick Coghlan8e113b42013-11-03 17:00:51 +1000827 Leaving first context
828 Callback: from first context
829 >>> with stack:
830 ... stack.callback(print, "Callback: from second context")
831 ... print("Leaving second context")
832 ...
833 Leaving second context
834 Callback: from second context
835 >>> with stack:
836 ... stack.callback(print, "Callback: from outer context")
837 ... with stack:
838 ... stack.callback(print, "Callback: from inner context")
839 ... print("Leaving inner context")
840 ... print("Leaving outer context")
841 ...
842 Leaving inner context
843 Callback: from inner context
844 Callback: from outer context
845 Leaving outer context
846
847As the output from the example shows, reusing a single stack object across
848multiple with statements works correctly, but attempting to nest them
849will cause the stack to be cleared at the end of the innermost with
850statement, which is unlikely to be desirable behaviour.
851
852Using separate :class:`ExitStack` instances instead of reusing a single
853instance avoids that problem::
854
855 >>> from contextlib import ExitStack
856 >>> with ExitStack() as outer_stack:
857 ... outer_stack.callback(print, "Callback: from outer context")
858 ... with ExitStack() as inner_stack:
859 ... inner_stack.callback(print, "Callback: from inner context")
860 ... print("Leaving inner context")
861 ... print("Leaving outer context")
862 ...
863 Leaving inner context
864 Callback: from inner context
865 Leaving outer context
866 Callback: from outer context