blob: 8c408923b70a727a014514c6f44e85e0f4d031de [file] [log] [blame]
Georg Brandl6c89a792012-01-25 22:36:25 +01001:mod:`functools` --- Higher-order functions and operations on callable objects
Georg Brandl116aa622007-08-15 14:28:22 +00002==============================================================================
3
4.. module:: functools
Georg Brandl6c89a792012-01-25 22:36:25 +01005 :synopsis: Higher-order functions and operations on callable objects.
Terry Jan Reedyfa089b92016-06-11 15:02:54 -04006
Georg Brandl116aa622007-08-15 14:28:22 +00007.. moduleauthor:: Peter Harris <scav@blueyonder.co.uk>
8.. moduleauthor:: Raymond Hettinger <python@rcn.com>
9.. moduleauthor:: Nick Coghlan <ncoghlan@gmail.com>
Łukasz Langa6f692512013-06-05 12:20:24 +020010.. moduleauthor:: Łukasz Langa <lukasz@langa.pl>
Pablo Galindo99e6c262020-01-23 15:29:52 +000011.. moduleauthor:: Pablo Galindo <pablogsal@gmail.com>
Georg Brandl116aa622007-08-15 14:28:22 +000012.. sectionauthor:: Peter Harris <scav@blueyonder.co.uk>
13
Raymond Hettinger05ce0792011-01-10 21:16:07 +000014**Source code:** :source:`Lib/functools.py`
15
Pablo Galindo99e6c262020-01-23 15:29:52 +000016.. testsetup:: default
17
18 import functools
19 from functools import *
20
Raymond Hettinger05ce0792011-01-10 21:16:07 +000021--------------
Georg Brandl116aa622007-08-15 14:28:22 +000022
Georg Brandl116aa622007-08-15 14:28:22 +000023The :mod:`functools` module is for higher-order functions: functions that act on
24or return other functions. In general, any callable object can be treated as a
25function for the purposes of this module.
26
Thomas Woutersed03b412007-08-28 21:37:11 +000027The :mod:`functools` module defines the following functions:
28
Carl Meyerd658dea2018-08-28 01:11:56 -060029.. decorator:: cached_property(func)
30
31 Transform a method of a class into a property whose value is computed once
32 and then cached as a normal attribute for the life of the instance. Similar
33 to :func:`property`, with the addition of caching. Useful for expensive
34 computed properties of instances that are otherwise effectively immutable.
35
36 Example::
37
38 class DataSet:
39 def __init__(self, sequence_of_numbers):
40 self._data = sequence_of_numbers
41
42 @cached_property
43 def stdev(self):
44 return statistics.stdev(self._data)
45
46 @cached_property
47 def variance(self):
48 return statistics.variance(self._data)
49
50 .. versionadded:: 3.8
51
52 .. note::
53
54 This decorator requires that the ``__dict__`` attribute on each instance
55 be a mutable mapping. This means it will not work with some types, such as
56 metaclasses (since the ``__dict__`` attributes on type instances are
57 read-only proxies for the class namespace), and those that specify
58 ``__slots__`` without including ``__dict__`` as one of the defined slots
59 (as such classes don't provide a ``__dict__`` attribute at all).
60
61
Éric Araujob10089e2010-11-18 14:22:08 +000062.. function:: cmp_to_key(func)
Raymond Hettingerc50846a2010-04-05 18:56:31 +000063
Raymond Hettinger86e9b6b2014-11-09 17:20:56 -080064 Transform an old-style comparison function to a :term:`key function`. Used
65 with tools that accept key functions (such as :func:`sorted`, :func:`min`,
Benjamin Petersoncca65312010-08-09 02:13:10 +000066 :func:`max`, :func:`heapq.nlargest`, :func:`heapq.nsmallest`,
67 :func:`itertools.groupby`). This function is primarily used as a transition
Ezio Melotti9ecb6be2012-01-16 08:28:54 +020068 tool for programs being converted from Python 2 which supported the use of
Benjamin Petersoncca65312010-08-09 02:13:10 +000069 comparison functions.
Raymond Hettingerc50846a2010-04-05 18:56:31 +000070
Georg Brandl6c89a792012-01-25 22:36:25 +010071 A comparison function is any callable that accept two arguments, compares them,
Benjamin Petersoncca65312010-08-09 02:13:10 +000072 and returns a negative number for less-than, zero for equality, or a positive
73 number for greater-than. A key function is a callable that accepts one
Raymond Hettinger86e9b6b2014-11-09 17:20:56 -080074 argument and returns another value to be used as the sort key.
Raymond Hettingerc50846a2010-04-05 18:56:31 +000075
Benjamin Petersoncca65312010-08-09 02:13:10 +000076 Example::
Raymond Hettingerc50846a2010-04-05 18:56:31 +000077
Benjamin Petersoncca65312010-08-09 02:13:10 +000078 sorted(iterable, key=cmp_to_key(locale.strcoll)) # locale-aware sort order
Raymond Hettingerc50846a2010-04-05 18:56:31 +000079
Raymond Hettinger86e9b6b2014-11-09 17:20:56 -080080 For sorting examples and a brief sorting tutorial, see :ref:`sortinghowto`.
81
Raymond Hettingerc50846a2010-04-05 18:56:31 +000082 .. versionadded:: 3.2
83
Georg Brandl67b21b72010-08-17 15:07:14 +000084
Raymond Hettingerb8218682019-05-26 11:27:35 -070085.. decorator:: lru_cache(user_function)
86 lru_cache(maxsize=128, typed=False)
Georg Brandl2e7346a2010-07-31 18:09:23 +000087
88 Decorator to wrap a function with a memoizing callable that saves up to the
89 *maxsize* most recent calls. It can save time when an expensive or I/O bound
90 function is periodically called with the same arguments.
91
Raymond Hettinger7496b412010-11-30 19:15:45 +000092 Since a dictionary is used to cache results, the positional and keyword
93 arguments to the function must be hashable.
Georg Brandl2e7346a2010-07-31 18:09:23 +000094
Raymond Hettinger902bcd92018-09-14 00:53:20 -070095 Distinct argument patterns may be considered to be distinct calls with
96 separate cache entries. For example, `f(a=1, b=2)` and `f(b=2, a=1)`
97 differ in their keyword argument order and may have two separate cache
98 entries.
99
Raymond Hettingerb8218682019-05-26 11:27:35 -0700100 If *user_function* is specified, it must be a callable. This allows the
101 *lru_cache* decorator to be applied directly to a user function, leaving
102 the *maxsize* at its default value of 128::
103
104 @lru_cache
105 def count_vowels(sentence):
106 sentence = sentence.casefold()
107 return sum(sentence.count(vowel) for vowel in 'aeiou')
108
Serhiy Storchakaecf41da2016-10-19 16:29:26 +0300109 If *maxsize* is set to ``None``, the LRU feature is disabled and the cache can
Raymond Hettinger7d74eff2012-06-04 00:32:15 -0700110 grow without bound. The LRU feature performs best when *maxsize* is a
111 power-of-two.
Raymond Hettingerc79fb0e2010-12-01 03:45:41 +0000112
Serhiy Storchaka4adf01c2016-10-19 18:30:05 +0300113 If *typed* is set to true, function arguments of different types will be
Raymond Hettingercd9fdfd2011-10-20 08:57:45 -0700114 cached separately. For example, ``f(3)`` and ``f(3.0)`` will be treated
115 as distinct calls with distinct results.
116
Manjusaka051ff522019-11-12 15:30:18 +0800117 The wrapped function is instrumented with a :func:`cache_parameters`
118 function that returns a new :class:`dict` showing the values for *maxsize*
119 and *typed*. This is for information purposes only. Mutating the values
120 has no effect.
121
Raymond Hettinger7496b412010-11-30 19:15:45 +0000122 To help measure the effectiveness of the cache and tune the *maxsize*
123 parameter, the wrapped function is instrumented with a :func:`cache_info`
124 function that returns a :term:`named tuple` showing *hits*, *misses*,
Raymond Hettingerc79fb0e2010-12-01 03:45:41 +0000125 *maxsize* and *currsize*. In a multi-threaded environment, the hits
126 and misses are approximate.
Nick Coghlan234515a2010-11-30 06:19:46 +0000127
Raymond Hettinger7496b412010-11-30 19:15:45 +0000128 The decorator also provides a :func:`cache_clear` function for clearing or
129 invalidating the cache.
Georg Brandl2e7346a2010-07-31 18:09:23 +0000130
Raymond Hettinger3fccfcb2010-08-17 19:19:29 +0000131 The original underlying function is accessible through the
Raymond Hettinger7496b412010-11-30 19:15:45 +0000132 :attr:`__wrapped__` attribute. This is useful for introspection, for
133 bypassing the cache, or for rewrapping the function with a different cache.
Nick Coghlan98876832010-08-17 06:17:18 +0000134
Raymond Hettingercc038582010-11-30 20:02:57 +0000135 An `LRU (least recently used) cache
Georg Brandl5d941342016-02-26 19:37:12 +0100136 <https://en.wikipedia.org/wiki/Cache_algorithms#Examples>`_ works
Raymond Hettingercd9fdfd2011-10-20 08:57:45 -0700137 best when the most recent calls are the best predictors of upcoming calls (for
138 example, the most popular articles on a news server tend to change each day).
Raymond Hettinger7496b412010-11-30 19:15:45 +0000139 The cache's size limit assures that the cache does not grow without bound on
140 long-running processes such as web servers.
141
Raymond Hettingerf0e0f202018-11-25 16:24:52 -0800142 In general, the LRU cache should only be used when you want to reuse
143 previously computed values. Accordingly, it doesn't make sense to cache
144 functions with side-effects, functions that need to create distinct mutable
145 objects on each call, or impure functions such as time() or random().
146
Raymond Hettingercc038582010-11-30 20:02:57 +0000147 Example of an LRU cache for static web content::
Raymond Hettinger7496b412010-11-30 19:15:45 +0000148
Raymond Hettinger17328e42013-04-06 20:27:33 -0700149 @lru_cache(maxsize=32)
Raymond Hettinger7496b412010-11-30 19:15:45 +0000150 def get_pep(num):
151 'Retrieve text of a Python Enhancement Proposal'
152 resource = 'http://www.python.org/dev/peps/pep-%04d/' % num
153 try:
154 with urllib.request.urlopen(resource) as s:
155 return s.read()
156 except urllib.error.HTTPError:
157 return 'Not Found'
158
159 >>> for n in 8, 290, 308, 320, 8, 218, 320, 279, 289, 320, 9991:
160 ... pep = get_pep(n)
161 ... print(n, len(pep))
162
Raymond Hettinger17328e42013-04-06 20:27:33 -0700163 >>> get_pep.cache_info()
164 CacheInfo(hits=3, misses=8, maxsize=32, currsize=8)
Georg Brandl2e7346a2010-07-31 18:09:23 +0000165
Raymond Hettingerc79fb0e2010-12-01 03:45:41 +0000166 Example of efficiently computing
Georg Brandl5d941342016-02-26 19:37:12 +0100167 `Fibonacci numbers <https://en.wikipedia.org/wiki/Fibonacci_number>`_
Raymond Hettingerc79fb0e2010-12-01 03:45:41 +0000168 using a cache to implement a
Georg Brandl5d941342016-02-26 19:37:12 +0100169 `dynamic programming <https://en.wikipedia.org/wiki/Dynamic_programming>`_
Raymond Hettingerc79fb0e2010-12-01 03:45:41 +0000170 technique::
171
172 @lru_cache(maxsize=None)
173 def fib(n):
174 if n < 2:
175 return n
176 return fib(n-1) + fib(n-2)
177
Raymond Hettinger17328e42013-04-06 20:27:33 -0700178 >>> [fib(n) for n in range(16)]
Raymond Hettingerc79fb0e2010-12-01 03:45:41 +0000179 [0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610]
180
Raymond Hettinger17328e42013-04-06 20:27:33 -0700181 >>> fib.cache_info()
Raymond Hettingerc79fb0e2010-12-01 03:45:41 +0000182 CacheInfo(hits=28, misses=16, maxsize=None, currsize=16)
183
Georg Brandl2e7346a2010-07-31 18:09:23 +0000184 .. versionadded:: 3.2
185
Raymond Hettingercd9fdfd2011-10-20 08:57:45 -0700186 .. versionchanged:: 3.3
187 Added the *typed* option.
188
Raymond Hettingerb8218682019-05-26 11:27:35 -0700189 .. versionchanged:: 3.8
190 Added the *user_function* option.
191
Manjusaka051ff522019-11-12 15:30:18 +0800192 .. versionadded:: 3.9
193 Added the function :func:`cache_parameters`
194
Georg Brandl8a1caa22010-07-29 16:01:11 +0000195.. decorator:: total_ordering
Raymond Hettingerc50846a2010-04-05 18:56:31 +0000196
197 Given a class defining one or more rich comparison ordering methods, this
Benjamin Peterson08bf91c2010-04-11 16:12:57 +0000198 class decorator supplies the rest. This simplifies the effort involved
Raymond Hettingerc50846a2010-04-05 18:56:31 +0000199 in specifying all of the possible rich comparison operations:
200
201 The class must define one of :meth:`__lt__`, :meth:`__le__`,
202 :meth:`__gt__`, or :meth:`__ge__`.
203 In addition, the class should supply an :meth:`__eq__` method.
204
205 For example::
206
207 @total_ordering
208 class Student:
Nick Coghlanf05d9812013-10-02 00:02:03 +1000209 def _is_valid_operand(self, other):
210 return (hasattr(other, "lastname") and
211 hasattr(other, "firstname"))
Raymond Hettingerc50846a2010-04-05 18:56:31 +0000212 def __eq__(self, other):
Nick Coghlanf05d9812013-10-02 00:02:03 +1000213 if not self._is_valid_operand(other):
214 return NotImplemented
Raymond Hettingerc50846a2010-04-05 18:56:31 +0000215 return ((self.lastname.lower(), self.firstname.lower()) ==
216 (other.lastname.lower(), other.firstname.lower()))
217 def __lt__(self, other):
Nick Coghlanf05d9812013-10-02 00:02:03 +1000218 if not self._is_valid_operand(other):
219 return NotImplemented
Raymond Hettingerc50846a2010-04-05 18:56:31 +0000220 return ((self.lastname.lower(), self.firstname.lower()) <
221 (other.lastname.lower(), other.firstname.lower()))
222
Nick Coghlanf05d9812013-10-02 00:02:03 +1000223 .. note::
224
225 While this decorator makes it easy to create well behaved totally
226 ordered types, it *does* come at the cost of slower execution and
227 more complex stack traces for the derived comparison methods. If
228 performance benchmarking indicates this is a bottleneck for a given
229 application, implementing all six rich comparison methods instead is
230 likely to provide an easy speed boost.
231
Raymond Hettingerc50846a2010-04-05 18:56:31 +0000232 .. versionadded:: 3.2
233
Nick Coghlanf05d9812013-10-02 00:02:03 +1000234 .. versionchanged:: 3.4
235 Returning NotImplemented from the underlying comparison function for
236 unrecognised types is now supported.
Georg Brandl67b21b72010-08-17 15:07:14 +0000237
Serhiy Storchaka2085bd02019-06-01 11:00:15 +0300238.. function:: partial(func, /, *args, **keywords)
Georg Brandl116aa622007-08-15 14:28:22 +0000239
Andrei Petre83a07652018-10-22 23:11:20 -0700240 Return a new :ref:`partial object<partial-objects>` which when called
241 will behave like *func* called with the positional arguments *args*
242 and keyword arguments *keywords*. If more arguments are supplied to the
243 call, they are appended to *args*. If additional keyword arguments are
244 supplied, they extend and override *keywords*.
Georg Brandl116aa622007-08-15 14:28:22 +0000245 Roughly equivalent to::
246
Serhiy Storchaka2085bd02019-06-01 11:00:15 +0300247 def partial(func, /, *args, **keywords):
Georg Brandl116aa622007-08-15 14:28:22 +0000248 def newfunc(*fargs, **fkeywords):
Sergey Fedoseevb981fec2018-10-20 02:42:07 +0500249 newkeywords = {**keywords, **fkeywords}
Martin Panter0c0da482016-06-12 01:46:50 +0000250 return func(*args, *fargs, **newkeywords)
Georg Brandl116aa622007-08-15 14:28:22 +0000251 newfunc.func = func
252 newfunc.args = args
253 newfunc.keywords = keywords
254 return newfunc
255
256 The :func:`partial` is used for partial function application which "freezes"
257 some portion of a function's arguments and/or keywords resulting in a new object
258 with a simplified signature. For example, :func:`partial` can be used to create
259 a callable that behaves like the :func:`int` function where the *base* argument
Christian Heimesfe337bf2008-03-23 21:54:12 +0000260 defaults to two:
Georg Brandl116aa622007-08-15 14:28:22 +0000261
Christian Heimesfe337bf2008-03-23 21:54:12 +0000262 >>> from functools import partial
Georg Brandl116aa622007-08-15 14:28:22 +0000263 >>> basetwo = partial(int, base=2)
264 >>> basetwo.__doc__ = 'Convert base 2 string to an int.'
265 >>> basetwo('10010')
266 18
267
268
Serhiy Storchaka70c5f2a2019-06-01 11:38:24 +0300269.. class:: partialmethod(func, /, *args, **keywords)
Nick Coghlanf4cb48a2013-11-03 16:41:46 +1000270
271 Return a new :class:`partialmethod` descriptor which behaves
272 like :class:`partial` except that it is designed to be used as a method
273 definition rather than being directly callable.
274
275 *func* must be a :term:`descriptor` or a callable (objects which are both,
276 like normal functions, are handled as descriptors).
277
278 When *func* is a descriptor (such as a normal Python function,
279 :func:`classmethod`, :func:`staticmethod`, :func:`abstractmethod` or
280 another instance of :class:`partialmethod`), calls to ``__get__`` are
281 delegated to the underlying descriptor, and an appropriate
Andrei Petre83a07652018-10-22 23:11:20 -0700282 :ref:`partial object<partial-objects>` returned as the result.
Nick Coghlanf4cb48a2013-11-03 16:41:46 +1000283
284 When *func* is a non-descriptor callable, an appropriate bound method is
285 created dynamically. This behaves like a normal Python function when
286 used as a method: the *self* argument will be inserted as the first
287 positional argument, even before the *args* and *keywords* supplied to
288 the :class:`partialmethod` constructor.
289
290 Example::
291
Serhiy Storchakae042a452019-06-10 13:35:52 +0300292 >>> class Cell:
Benjamin Peterson3a434032014-03-30 15:07:09 -0400293 ... def __init__(self):
294 ... self._alive = False
Nick Coghlanf4cb48a2013-11-03 16:41:46 +1000295 ... @property
296 ... def alive(self):
297 ... return self._alive
298 ... def set_state(self, state):
299 ... self._alive = bool(state)
Nick Coghlan3daaf5f2013-11-04 23:32:16 +1000300 ... set_alive = partialmethod(set_state, True)
301 ... set_dead = partialmethod(set_state, False)
Nick Coghlanf4cb48a2013-11-03 16:41:46 +1000302 ...
303 >>> c = Cell()
304 >>> c.alive
305 False
306 >>> c.set_alive()
307 >>> c.alive
308 True
309
310 .. versionadded:: 3.4
311
312
Georg Brandl58f9e4f2008-04-19 22:18:33 +0000313.. function:: reduce(function, iterable[, initializer])
Georg Brandl116aa622007-08-15 14:28:22 +0000314
Brendan Jurd9df10022018-10-01 16:52:10 +1000315 Apply *function* of two arguments cumulatively to the items of *iterable*, from
316 left to right, so as to reduce the iterable to a single value. For example,
Georg Brandl116aa622007-08-15 14:28:22 +0000317 ``reduce(lambda x, y: x+y, [1, 2, 3, 4, 5])`` calculates ``((((1+2)+3)+4)+5)``.
318 The left argument, *x*, is the accumulated value and the right argument, *y*, is
Brendan Jurd9df10022018-10-01 16:52:10 +1000319 the update value from the *iterable*. If the optional *initializer* is present,
320 it is placed before the items of the iterable in the calculation, and serves as
321 a default when the iterable is empty. If *initializer* is not given and
322 *iterable* contains only one item, the first item is returned.
Georg Brandl116aa622007-08-15 14:28:22 +0000323
Raymond Hettinger558dcf32014-12-16 18:16:57 -0800324 Roughly equivalent to::
Raymond Hettinger64801682013-10-12 16:04:17 -0700325
326 def reduce(function, iterable, initializer=None):
327 it = iter(iterable)
328 if initializer is None:
329 value = next(it)
330 else:
331 value = initializer
332 for element in it:
333 value = function(value, element)
334 return value
335
Gerrit Hollbd81cbd2018-07-04 23:26:32 +0100336 See :func:`itertools.accumulate` for an iterator that yields all intermediate
337 values.
Georg Brandl116aa622007-08-15 14:28:22 +0000338
Daisuke Miyakawa0e61e672017-10-12 23:39:43 +0900339.. decorator:: singledispatch
Łukasz Langa6f692512013-06-05 12:20:24 +0200340
Daisuke Miyakawa0e61e672017-10-12 23:39:43 +0900341 Transform a function into a :term:`single-dispatch <single
Łukasz Langafdcf2b72013-06-07 22:54:03 +0200342 dispatch>` :term:`generic function`.
Łukasz Langa6f692512013-06-05 12:20:24 +0200343
344 To define a generic function, decorate it with the ``@singledispatch``
345 decorator. Note that the dispatch happens on the type of the first argument,
346 create your function accordingly::
347
348 >>> from functools import singledispatch
349 >>> @singledispatch
350 ... def fun(arg, verbose=False):
351 ... if verbose:
352 ... print("Let me just say,", end=" ")
353 ... print(arg)
354
355 To add overloaded implementations to the function, use the :func:`register`
Łukasz Langae5697532017-12-11 13:56:31 -0800356 attribute of the generic function. It is a decorator. For functions
357 annotated with types, the decorator will infer the type of the first
358 argument automatically::
Łukasz Langa6f692512013-06-05 12:20:24 +0200359
Łukasz Langae5697532017-12-11 13:56:31 -0800360 >>> @fun.register
361 ... def _(arg: int, verbose=False):
Łukasz Langa6f692512013-06-05 12:20:24 +0200362 ... if verbose:
363 ... print("Strength in numbers, eh?", end=" ")
364 ... print(arg)
365 ...
Łukasz Langae5697532017-12-11 13:56:31 -0800366 >>> @fun.register
367 ... def _(arg: list, verbose=False):
Łukasz Langa6f692512013-06-05 12:20:24 +0200368 ... if verbose:
369 ... print("Enumerate this:")
370 ... for i, elem in enumerate(arg):
371 ... print(i, elem)
372
Łukasz Langae5697532017-12-11 13:56:31 -0800373 For code which doesn't use type annotations, the appropriate type
374 argument can be passed explicitly to the decorator itself::
375
376 >>> @fun.register(complex)
377 ... def _(arg, verbose=False):
378 ... if verbose:
379 ... print("Better than complicated.", end=" ")
380 ... print(arg.real, arg.imag)
381 ...
382
383
Łukasz Langa6f692512013-06-05 12:20:24 +0200384 To enable registering lambdas and pre-existing functions, the
385 :func:`register` attribute can be used in a functional form::
386
387 >>> def nothing(arg, verbose=False):
388 ... print("Nothing.")
389 ...
390 >>> fun.register(type(None), nothing)
391
392 The :func:`register` attribute returns the undecorated function which
393 enables decorator stacking, pickling, as well as creating unit tests for
394 each variant independently::
395
396 >>> @fun.register(float)
397 ... @fun.register(Decimal)
398 ... def fun_num(arg, verbose=False):
399 ... if verbose:
400 ... print("Half of your number:", end=" ")
401 ... print(arg / 2)
402 ...
403 >>> fun_num is fun
404 False
405
406 When called, the generic function dispatches on the type of the first
407 argument::
408
409 >>> fun("Hello, world.")
410 Hello, world.
411 >>> fun("test.", verbose=True)
412 Let me just say, test.
413 >>> fun(42, verbose=True)
414 Strength in numbers, eh? 42
415 >>> fun(['spam', 'spam', 'eggs', 'spam'], verbose=True)
416 Enumerate this:
417 0 spam
418 1 spam
419 2 eggs
420 3 spam
421 >>> fun(None)
422 Nothing.
423 >>> fun(1.23)
424 0.615
425
426 Where there is no registered implementation for a specific type, its
427 method resolution order is used to find a more generic implementation.
428 The original function decorated with ``@singledispatch`` is registered
429 for the base ``object`` type, which means it is used if no better
430 implementation is found.
431
Batuhan Taşkaya24555ce2019-11-19 11:16:46 +0300432 If an implementation registered to :term:`abstract base class`, virtual
433 subclasses will be dispatched to that implementation::
434
435 >>> from collections.abc import Mapping
436 >>> @fun.register
437 ... def _(arg: Mapping, verbose=False):
438 ... if verbose:
439 ... print("Keys & Values")
440 ... for key, value in arg.items():
441 ... print(key, "=>", value)
442 ...
443 >>> fun({"a": "b"})
444 a => b
445
Łukasz Langa6f692512013-06-05 12:20:24 +0200446 To check which implementation will the generic function choose for
447 a given type, use the ``dispatch()`` attribute::
448
449 >>> fun.dispatch(float)
450 <function fun_num at 0x1035a2840>
451 >>> fun.dispatch(dict) # note: default implementation
452 <function fun at 0x103fe0000>
453
454 To access all registered implementations, use the read-only ``registry``
455 attribute::
456
457 >>> fun.registry.keys()
458 dict_keys([<class 'NoneType'>, <class 'int'>, <class 'object'>,
459 <class 'decimal.Decimal'>, <class 'list'>,
460 <class 'float'>])
461 >>> fun.registry[float]
462 <function fun_num at 0x1035a2840>
463 >>> fun.registry[object]
464 <function fun at 0x103fe0000>
465
466 .. versionadded:: 3.4
467
Łukasz Langae5697532017-12-11 13:56:31 -0800468 .. versionchanged:: 3.7
469 The :func:`register` attribute supports using type annotations.
470
Łukasz Langa6f692512013-06-05 12:20:24 +0200471
Ethan Smithc6512752018-05-26 16:38:33 -0400472.. class:: singledispatchmethod(func)
473
474 Transform a method into a :term:`single-dispatch <single
475 dispatch>` :term:`generic function`.
476
477 To define a generic method, decorate it with the ``@singledispatchmethod``
478 decorator. Note that the dispatch happens on the type of the first non-self
479 or non-cls argument, create your function accordingly::
480
481 class Negator:
482 @singledispatchmethod
483 def neg(self, arg):
484 raise NotImplementedError("Cannot negate a")
485
486 @neg.register
487 def _(self, arg: int):
488 return -arg
489
490 @neg.register
491 def _(self, arg: bool):
492 return not arg
493
494 ``@singledispatchmethod`` supports nesting with other decorators such as
495 ``@classmethod``. Note that to allow for ``dispatcher.register``,
496 ``singledispatchmethod`` must be the *outer most* decorator. Here is the
497 ``Negator`` class with the ``neg`` methods being class bound::
498
499 class Negator:
500 @singledispatchmethod
501 @classmethod
502 def neg(cls, arg):
503 raise NotImplementedError("Cannot negate a")
504
505 @neg.register
506 @classmethod
507 def _(cls, arg: int):
508 return -arg
509
510 @neg.register
511 @classmethod
512 def _(cls, arg: bool):
513 return not arg
514
515 The same pattern can be used for other similar decorators: ``staticmethod``,
516 ``abstractmethod``, and others.
517
Inada Naokibc284f02019-03-27 18:15:17 +0900518 .. versionadded:: 3.8
519
520
Pablo Galindo99e6c262020-01-23 15:29:52 +0000521.. class:: TopologicalSorter(graph=None)
522
523 Provides functionality to topologically sort a graph of hashable nodes.
524
525 A topological order is a linear ordering of the vertices in a graph such that for
526 every directed edge u -> v from vertex u to vertex v, vertex u comes before vertex
527 v in the ordering. For instance, the vertices of the graph may represent tasks to
528 be performed, and the edges may represent constraints that one task must be
529 performed before another; in this example, a topological ordering is just a valid
530 sequence for the tasks. A complete topological ordering is possible if and only if
531 the graph has no directed cycles, that is, if it is a directed acyclic graph.
532
533 If the optional *graph* argument is provided it must be a dictionary representing
534 a directed acyclic graph where the keys are nodes and the values are iterables of
535 all predecessors of that node in the graph (the nodes that have edges that point
536 to the value in the key). Additional nodes can be added to the graph using the
537 :meth:`~TopologicalSorter.add` method.
538
539 In the general case, the steps required to perform the sorting of a given graph
540 are as follows:
541
542 * Create an instance of the :class:`TopologicalSorter` with an optional initial graph.
543 * Add additional nodes to the graph.
544 * Call :meth:`~TopologicalSorter.prepare` on the graph.
545 * While :meth:`~TopologicalSorter.is_active` is ``True``, iterate over the
546 nodes returned by :meth:`~TopologicalSorter.get_ready` and process them.
547 Call :meth:`~TopologicalSorter.done` on each node as it finishes processing.
548
549 In case just an immediate sorting of the nodes in the graph is required and
550 no parallelism is involved, the convenience method :meth:`TopologicalSorter.static_order`
551 can be used directly. For example, this method can be used to implement a simple
552 version of the C3 linearization algorithm used by Python to calculate the Method
553 Resolution Order (MRO) of a derived class:
554
555 .. doctest::
556
557 >>> class A: pass
558 >>> class B(A): pass
559 >>> class C(A): pass
560 >>> class D(B, C): pass
561
562 >>> D.__mro__
563 (<class 'D'>, <class 'B'>, <class 'C'>, <class 'A'>, <class 'object'>)
564
565 >>> graph = {D: {B, C}, C: {A}, B: {A}, A:{object}}
566 >>> ts = TopologicalSorter(graph)
567 >>> topological_order = tuple(ts.static_order())
568 >>> tuple(reversed(topological_order))
569 (<class 'D'>, <class 'B'>, <class 'C'>, <class 'A'>, <class 'object'>)
570
571 The class is designed to easily support parallel processing of the nodes as they
572 become ready. For instance::
573
574 topological_sorter = TopologicalSorter()
575
576 # Add nodes to 'topological_sorter'...
577
578 topological_sorter.prepare()
579 while topological_sorter.is_active():
580 for node in topological_sorter.get_ready():
581 # Worker threads or processes take nodes to work on off the
582 # 'task_queue' queue.
583 task_queue.put(node)
584
585 # When the work for a node is done, workers put the node in
586 # 'finalized_tasks_queue' so we can get more nodes to work on.
587 # The definition of 'is_active()' guarantees that, at this point, at
588 # least one node has been placed on 'task_queue' that hasn't yet
589 # been passed to 'done()', so this blocking 'get()' must (eventually)
590 # succeed. After calling 'done()', we loop back to call 'get_ready()'
591 # again, so put newly freed nodes on 'task_queue' as soon as
592 # logically possible.
593 node = finalized_tasks_queue.get()
594 topological_sorter.done(node)
595
596 .. method:: add(node, *predecessors)
597
598 Add a new node and its predecessors to the graph. Both the *node* and
599 all elements in *predecessors* must be hashable.
600
601 If called multiple times with the same node argument, the set of dependencies
602 will be the union of all dependencies passed in.
603
604 It is possible to add a node with no dependencies (*predecessors* is not
605 provided) or to provide a dependency twice. If a node that has not been
606 provided before is included among *predecessors* it will be automatically added
607 to the graph with no predecessors of its own.
608
609 Raises :exc:`ValueError` if called after :meth:`~TopologicalSorter.prepare`.
610
611 .. method:: prepare()
612
613 Mark the graph as finished and check for cycles in the graph. If any cycle is
614 detected, :exc:`CycleError` will be raised, but
615 :meth:`~TopologicalSorter.get_ready` can still be used to obtain as many nodes
616 as possible until cycles block more progress. After a call to this function,
617 the graph cannot be modified, and therefore no more nodes can be added using
618 :meth:`~TopologicalSorter.add`.
619
620 .. method:: is_active()
621
622 Returns ``True`` if more progress can be made and ``False`` otherwise. Progress
623 can be made if cycles do not block the resolution and either there are still
624 nodes ready that haven't yet been returned by
625 :meth:`TopologicalSorter.get_ready` or the number of nodes marked
626 :meth:`TopologicalSorter.done` is less than the number that have been returned
627 by :meth:`TopologicalSorter.get_ready`.
628
629 The :meth:`~TopologicalSorter.__bool__` method of this class defers to this
630 function, so instead of::
631
632 if ts.is_active():
633 ...
634
635 if possible to simply do::
636
637 if ts:
638 ...
639
640 Raises :exc:`ValueError` if called without calling :meth:`~TopologicalSorter.prepare`
641 previously.
642
643 .. method:: done(*nodes)
644
645 Marks a set of nodes returned by :meth:`TopologicalSorter.get_ready` as
646 processed, unblocking any successor of each node in *nodes* for being returned
647 in the future by a call to :meth:`TopologicalSorter.get_ready`.
648
649 Raises :exc:`ValueError` if any node in *nodes* has already been marked as
650 processed by a previous call to this method or if a node was not added to the
651 graph by using :meth:`TopologicalSorter.add`, if called without calling
652 :meth:`~TopologicalSorter.prepare` or if node has not yet been returned by
653 :meth:`~TopologicalSorter.get_ready`.
654
655 .. method:: get_ready()
656
657 Returns a ``tuple`` with all the nodes that are ready. Initially it returns all
658 nodes with no predecessors, and once those are marked as processed by calling
659 :meth:`TopologicalSorter.done`, further calls will return all new nodes that
660 have all their predecessors already processed. Once no more progress can be
661 made, empty tuples are returned.
662 made.
663
664 Raises :exc:`ValueError` if called without calling
665 :meth:`~TopologicalSorter.prepare` previously.
666
667 .. method:: static_order()
668
669 Returns an iterable of nodes in a topological order. Using this method
670 does not require to call :meth:`TopologicalSorter.prepare` or
671 :meth:`TopologicalSorter.done`. This method is equivalent to::
672
673 def static_order(self):
674 self.prepare()
675 while self.is_active():
676 node_group = self.get_ready()
677 yield from node_group
678 self.done(*node_group)
679
680 The particular order that is returned may depend on the specific order in
681 which the items were inserted in the graph. For example:
682
683 .. doctest::
684
685 >>> ts = TopologicalSorter()
686 >>> ts.add(3, 2, 1)
687 >>> ts.add(1, 0)
688 >>> print([*ts.static_order()])
689 [2, 0, 1, 3]
690
691 >>> ts2 = TopologicalSorter()
692 >>> ts2.add(1, 0)
693 >>> ts2.add(3, 2, 1)
694 >>> print([*ts2.static_order()])
695 [0, 2, 1, 3]
696
697 This is due to the fact that "0" and "2" are in the same level in the graph (they
698 would have been returned in the same call to :meth:`~TopologicalSorter.get_ready`)
699 and the order between them is determined by the order of insertion.
700
701
702 If any cycle is detected, :exc:`CycleError` will be raised.
703
704 .. versionadded:: 3.9
705
706
Georg Brandl036490d2009-05-17 13:00:36 +0000707.. function:: update_wrapper(wrapper, wrapped, assigned=WRAPPER_ASSIGNMENTS, updated=WRAPPER_UPDATES)
Georg Brandl116aa622007-08-15 14:28:22 +0000708
709 Update a *wrapper* function to look like the *wrapped* function. The optional
710 arguments are tuples to specify which attributes of the original function are
711 assigned directly to the matching attributes on the wrapper function and which
712 attributes of the wrapper function are updated with the corresponding attributes
713 from the original function. The default values for these arguments are the
Berker Peksag472233e2016-04-18 21:20:50 +0300714 module level constants ``WRAPPER_ASSIGNMENTS`` (which assigns to the wrapper
715 function's ``__module__``, ``__name__``, ``__qualname__``, ``__annotations__``
716 and ``__doc__``, the documentation string) and ``WRAPPER_UPDATES`` (which
717 updates the wrapper function's ``__dict__``, i.e. the instance dictionary).
Georg Brandl116aa622007-08-15 14:28:22 +0000718
Nick Coghlan98876832010-08-17 06:17:18 +0000719 To allow access to the original function for introspection and other purposes
720 (e.g. bypassing a caching decorator such as :func:`lru_cache`), this function
Nick Coghlan24c05bc2013-07-15 21:13:08 +1000721 automatically adds a ``__wrapped__`` attribute to the wrapper that refers to
722 the function being wrapped.
Nick Coghlan98876832010-08-17 06:17:18 +0000723
Christian Heimesd8654cf2007-12-02 15:22:16 +0000724 The main intended use for this function is in :term:`decorator` functions which
725 wrap the decorated function and return the wrapper. If the wrapper function is
726 not updated, the metadata of the returned function will reflect the wrapper
Georg Brandl116aa622007-08-15 14:28:22 +0000727 definition rather than the original function definition, which is typically less
728 than helpful.
729
Nick Coghlan98876832010-08-17 06:17:18 +0000730 :func:`update_wrapper` may be used with callables other than functions. Any
731 attributes named in *assigned* or *updated* that are missing from the object
732 being wrapped are ignored (i.e. this function will not attempt to set them
733 on the wrapper function). :exc:`AttributeError` is still raised if the
734 wrapper function itself is missing any attributes named in *updated*.
735
736 .. versionadded:: 3.2
Georg Brandl9e257012010-08-17 14:11:59 +0000737 Automatic addition of the ``__wrapped__`` attribute.
Nick Coghlan98876832010-08-17 06:17:18 +0000738
739 .. versionadded:: 3.2
Georg Brandl9e257012010-08-17 14:11:59 +0000740 Copying of the ``__annotations__`` attribute by default.
Nick Coghlan98876832010-08-17 06:17:18 +0000741
742 .. versionchanged:: 3.2
Georg Brandl9e257012010-08-17 14:11:59 +0000743 Missing attributes no longer trigger an :exc:`AttributeError`.
744
Nick Coghlan24c05bc2013-07-15 21:13:08 +1000745 .. versionchanged:: 3.4
746 The ``__wrapped__`` attribute now always refers to the wrapped
747 function, even if that function defined a ``__wrapped__`` attribute.
748 (see :issue:`17482`)
749
Georg Brandl116aa622007-08-15 14:28:22 +0000750
Georg Brandl8a1caa22010-07-29 16:01:11 +0000751.. decorator:: wraps(wrapped, assigned=WRAPPER_ASSIGNMENTS, updated=WRAPPER_UPDATES)
Georg Brandl116aa622007-08-15 14:28:22 +0000752
Ezio Melotti67f6d5f2014-08-05 08:14:28 +0300753 This is a convenience function for invoking :func:`update_wrapper` as a
754 function decorator when defining a wrapper function. It is equivalent to
755 ``partial(update_wrapper, wrapped=wrapped, assigned=assigned, updated=updated)``.
756 For example::
Georg Brandl116aa622007-08-15 14:28:22 +0000757
Christian Heimesfe337bf2008-03-23 21:54:12 +0000758 >>> from functools import wraps
Georg Brandl116aa622007-08-15 14:28:22 +0000759 >>> def my_decorator(f):
760 ... @wraps(f)
761 ... def wrapper(*args, **kwds):
Georg Brandl6911e3c2007-09-04 07:15:32 +0000762 ... print('Calling decorated function')
Georg Brandl116aa622007-08-15 14:28:22 +0000763 ... return f(*args, **kwds)
764 ... return wrapper
765 ...
766 >>> @my_decorator
767 ... def example():
768 ... """Docstring"""
Georg Brandl6911e3c2007-09-04 07:15:32 +0000769 ... print('Called example function')
Georg Brandl116aa622007-08-15 14:28:22 +0000770 ...
771 >>> example()
772 Calling decorated function
773 Called example function
774 >>> example.__name__
775 'example'
776 >>> example.__doc__
777 'Docstring'
778
779 Without the use of this decorator factory, the name of the example function
780 would have been ``'wrapper'``, and the docstring of the original :func:`example`
781 would have been lost.
782
783
784.. _partial-objects:
785
786:class:`partial` Objects
787------------------------
788
789:class:`partial` objects are callable objects created by :func:`partial`. They
790have three read-only attributes:
791
792
793.. attribute:: partial.func
794
795 A callable object or function. Calls to the :class:`partial` object will be
796 forwarded to :attr:`func` with new arguments and keywords.
797
798
799.. attribute:: partial.args
800
801 The leftmost positional arguments that will be prepended to the positional
802 arguments provided to a :class:`partial` object call.
803
804
805.. attribute:: partial.keywords
806
807 The keyword arguments that will be supplied when the :class:`partial` object is
808 called.
809
810:class:`partial` objects are like :class:`function` objects in that they are
811callable, weak referencable, and can have attributes. There are some important
Martin Panterbae5d812016-06-18 03:57:31 +0000812differences. For instance, the :attr:`~definition.__name__` and :attr:`__doc__` attributes
Georg Brandl116aa622007-08-15 14:28:22 +0000813are not created automatically. Also, :class:`partial` objects defined in
814classes behave like static methods and do not transform into bound methods
815during instance attribute look-up.
Pablo Galindo99e6c262020-01-23 15:29:52 +0000816
817
818Exceptions
819----------
820The :mod:`functools` module defines the following exception classes:
821
822.. exception:: CycleError
823
824 Subclass of :exc:`ValueError` raised by :meth:`TopologicalSorter.prepare` if cycles exist
825 in the working graph. If multiple cycles exist, only one undefined choice among them will
826 be reported and included in the exception.
827
828 The detected cycle can be accessed via the second element in the :attr:`~CycleError.args`
829 attribute of the exception instance and consists in a list of nodes, such that each node is,
830 in the graph, an immediate predecessor of the next node in the list. In the reported list,
831 the first and the last node will be the same, to make it clear that it is cyclic.