blob: 856c1c790ae36164a99949b61957f76489cdc9a7 [file] [log] [blame]
Georg Brandl6c89a792012-01-25 22:36:25 +01001:mod:`functools` --- Higher-order functions and operations on callable objects
Georg Brandl116aa622007-08-15 14:28:22 +00002==============================================================================
3
4.. module:: functools
Georg Brandl6c89a792012-01-25 22:36:25 +01005 :synopsis: Higher-order functions and operations on callable objects.
Terry Jan Reedyfa089b92016-06-11 15:02:54 -04006
Georg Brandl116aa622007-08-15 14:28:22 +00007.. moduleauthor:: Peter Harris <scav@blueyonder.co.uk>
8.. moduleauthor:: Raymond Hettinger <python@rcn.com>
9.. moduleauthor:: Nick Coghlan <ncoghlan@gmail.com>
Łukasz Langa6f692512013-06-05 12:20:24 +020010.. moduleauthor:: Łukasz Langa <lukasz@langa.pl>
Pablo Galindo99e6c262020-01-23 15:29:52 +000011.. moduleauthor:: Pablo Galindo <pablogsal@gmail.com>
Georg Brandl116aa622007-08-15 14:28:22 +000012.. sectionauthor:: Peter Harris <scav@blueyonder.co.uk>
13
Raymond Hettinger05ce0792011-01-10 21:16:07 +000014**Source code:** :source:`Lib/functools.py`
15
Pablo Galindo99e6c262020-01-23 15:29:52 +000016.. testsetup:: default
17
18 import functools
19 from functools import *
20
Raymond Hettinger05ce0792011-01-10 21:16:07 +000021--------------
Georg Brandl116aa622007-08-15 14:28:22 +000022
Georg Brandl116aa622007-08-15 14:28:22 +000023The :mod:`functools` module is for higher-order functions: functions that act on
24or return other functions. In general, any callable object can be treated as a
25function for the purposes of this module.
26
Thomas Woutersed03b412007-08-28 21:37:11 +000027The :mod:`functools` module defines the following functions:
28
Carl Meyerd658dea2018-08-28 01:11:56 -060029.. decorator:: cached_property(func)
30
31 Transform a method of a class into a property whose value is computed once
32 and then cached as a normal attribute for the life of the instance. Similar
33 to :func:`property`, with the addition of caching. Useful for expensive
34 computed properties of instances that are otherwise effectively immutable.
35
36 Example::
37
38 class DataSet:
39 def __init__(self, sequence_of_numbers):
40 self._data = sequence_of_numbers
41
42 @cached_property
43 def stdev(self):
44 return statistics.stdev(self._data)
45
46 @cached_property
47 def variance(self):
48 return statistics.variance(self._data)
49
50 .. versionadded:: 3.8
51
52 .. note::
53
54 This decorator requires that the ``__dict__`` attribute on each instance
55 be a mutable mapping. This means it will not work with some types, such as
56 metaclasses (since the ``__dict__`` attributes on type instances are
57 read-only proxies for the class namespace), and those that specify
58 ``__slots__`` without including ``__dict__`` as one of the defined slots
59 (as such classes don't provide a ``__dict__`` attribute at all).
60
61
Éric Araujob10089e2010-11-18 14:22:08 +000062.. function:: cmp_to_key(func)
Raymond Hettingerc50846a2010-04-05 18:56:31 +000063
Raymond Hettinger86e9b6b2014-11-09 17:20:56 -080064 Transform an old-style comparison function to a :term:`key function`. Used
65 with tools that accept key functions (such as :func:`sorted`, :func:`min`,
Benjamin Petersoncca65312010-08-09 02:13:10 +000066 :func:`max`, :func:`heapq.nlargest`, :func:`heapq.nsmallest`,
67 :func:`itertools.groupby`). This function is primarily used as a transition
Ezio Melotti9ecb6be2012-01-16 08:28:54 +020068 tool for programs being converted from Python 2 which supported the use of
Benjamin Petersoncca65312010-08-09 02:13:10 +000069 comparison functions.
Raymond Hettingerc50846a2010-04-05 18:56:31 +000070
Georg Brandl6c89a792012-01-25 22:36:25 +010071 A comparison function is any callable that accept two arguments, compares them,
Benjamin Petersoncca65312010-08-09 02:13:10 +000072 and returns a negative number for less-than, zero for equality, or a positive
73 number for greater-than. A key function is a callable that accepts one
Raymond Hettinger86e9b6b2014-11-09 17:20:56 -080074 argument and returns another value to be used as the sort key.
Raymond Hettingerc50846a2010-04-05 18:56:31 +000075
Benjamin Petersoncca65312010-08-09 02:13:10 +000076 Example::
Raymond Hettingerc50846a2010-04-05 18:56:31 +000077
Benjamin Petersoncca65312010-08-09 02:13:10 +000078 sorted(iterable, key=cmp_to_key(locale.strcoll)) # locale-aware sort order
Raymond Hettingerc50846a2010-04-05 18:56:31 +000079
Raymond Hettinger86e9b6b2014-11-09 17:20:56 -080080 For sorting examples and a brief sorting tutorial, see :ref:`sortinghowto`.
81
Raymond Hettingerc50846a2010-04-05 18:56:31 +000082 .. versionadded:: 3.2
83
Georg Brandl67b21b72010-08-17 15:07:14 +000084
Raymond Hettingerb8218682019-05-26 11:27:35 -070085.. decorator:: lru_cache(user_function)
86 lru_cache(maxsize=128, typed=False)
Georg Brandl2e7346a2010-07-31 18:09:23 +000087
88 Decorator to wrap a function with a memoizing callable that saves up to the
89 *maxsize* most recent calls. It can save time when an expensive or I/O bound
90 function is periodically called with the same arguments.
91
Raymond Hettinger7496b412010-11-30 19:15:45 +000092 Since a dictionary is used to cache results, the positional and keyword
93 arguments to the function must be hashable.
Georg Brandl2e7346a2010-07-31 18:09:23 +000094
Raymond Hettinger902bcd92018-09-14 00:53:20 -070095 Distinct argument patterns may be considered to be distinct calls with
96 separate cache entries. For example, `f(a=1, b=2)` and `f(b=2, a=1)`
97 differ in their keyword argument order and may have two separate cache
98 entries.
99
Raymond Hettingerb8218682019-05-26 11:27:35 -0700100 If *user_function* is specified, it must be a callable. This allows the
101 *lru_cache* decorator to be applied directly to a user function, leaving
102 the *maxsize* at its default value of 128::
103
104 @lru_cache
105 def count_vowels(sentence):
106 sentence = sentence.casefold()
107 return sum(sentence.count(vowel) for vowel in 'aeiou')
108
Serhiy Storchakaecf41da2016-10-19 16:29:26 +0300109 If *maxsize* is set to ``None``, the LRU feature is disabled and the cache can
Raymond Hettingerad9eaea2020-05-03 16:45:13 -0700110 grow without bound.
Raymond Hettingerc79fb0e2010-12-01 03:45:41 +0000111
Serhiy Storchaka4adf01c2016-10-19 18:30:05 +0300112 If *typed* is set to true, function arguments of different types will be
Raymond Hettingercd9fdfd2011-10-20 08:57:45 -0700113 cached separately. For example, ``f(3)`` and ``f(3.0)`` will be treated
114 as distinct calls with distinct results.
115
Manjusaka051ff522019-11-12 15:30:18 +0800116 The wrapped function is instrumented with a :func:`cache_parameters`
117 function that returns a new :class:`dict` showing the values for *maxsize*
118 and *typed*. This is for information purposes only. Mutating the values
119 has no effect.
120
Raymond Hettinger7496b412010-11-30 19:15:45 +0000121 To help measure the effectiveness of the cache and tune the *maxsize*
122 parameter, the wrapped function is instrumented with a :func:`cache_info`
123 function that returns a :term:`named tuple` showing *hits*, *misses*,
Raymond Hettingerc79fb0e2010-12-01 03:45:41 +0000124 *maxsize* and *currsize*. In a multi-threaded environment, the hits
125 and misses are approximate.
Nick Coghlan234515a2010-11-30 06:19:46 +0000126
Raymond Hettinger7496b412010-11-30 19:15:45 +0000127 The decorator also provides a :func:`cache_clear` function for clearing or
128 invalidating the cache.
Georg Brandl2e7346a2010-07-31 18:09:23 +0000129
Raymond Hettinger3fccfcb2010-08-17 19:19:29 +0000130 The original underlying function is accessible through the
Raymond Hettinger7496b412010-11-30 19:15:45 +0000131 :attr:`__wrapped__` attribute. This is useful for introspection, for
132 bypassing the cache, or for rewrapping the function with a different cache.
Nick Coghlan98876832010-08-17 06:17:18 +0000133
Raymond Hettingercc038582010-11-30 20:02:57 +0000134 An `LRU (least recently used) cache
Georg Brandl5d941342016-02-26 19:37:12 +0100135 <https://en.wikipedia.org/wiki/Cache_algorithms#Examples>`_ works
Raymond Hettingercd9fdfd2011-10-20 08:57:45 -0700136 best when the most recent calls are the best predictors of upcoming calls (for
137 example, the most popular articles on a news server tend to change each day).
Raymond Hettinger7496b412010-11-30 19:15:45 +0000138 The cache's size limit assures that the cache does not grow without bound on
139 long-running processes such as web servers.
140
Raymond Hettingerf0e0f202018-11-25 16:24:52 -0800141 In general, the LRU cache should only be used when you want to reuse
142 previously computed values. Accordingly, it doesn't make sense to cache
143 functions with side-effects, functions that need to create distinct mutable
144 objects on each call, or impure functions such as time() or random().
145
Raymond Hettingercc038582010-11-30 20:02:57 +0000146 Example of an LRU cache for static web content::
Raymond Hettinger7496b412010-11-30 19:15:45 +0000147
Raymond Hettinger17328e42013-04-06 20:27:33 -0700148 @lru_cache(maxsize=32)
Raymond Hettinger7496b412010-11-30 19:15:45 +0000149 def get_pep(num):
150 'Retrieve text of a Python Enhancement Proposal'
151 resource = 'http://www.python.org/dev/peps/pep-%04d/' % num
152 try:
153 with urllib.request.urlopen(resource) as s:
154 return s.read()
155 except urllib.error.HTTPError:
156 return 'Not Found'
157
158 >>> for n in 8, 290, 308, 320, 8, 218, 320, 279, 289, 320, 9991:
159 ... pep = get_pep(n)
160 ... print(n, len(pep))
161
Raymond Hettinger17328e42013-04-06 20:27:33 -0700162 >>> get_pep.cache_info()
163 CacheInfo(hits=3, misses=8, maxsize=32, currsize=8)
Georg Brandl2e7346a2010-07-31 18:09:23 +0000164
Raymond Hettingerc79fb0e2010-12-01 03:45:41 +0000165 Example of efficiently computing
Georg Brandl5d941342016-02-26 19:37:12 +0100166 `Fibonacci numbers <https://en.wikipedia.org/wiki/Fibonacci_number>`_
Raymond Hettingerc79fb0e2010-12-01 03:45:41 +0000167 using a cache to implement a
Georg Brandl5d941342016-02-26 19:37:12 +0100168 `dynamic programming <https://en.wikipedia.org/wiki/Dynamic_programming>`_
Raymond Hettingerc79fb0e2010-12-01 03:45:41 +0000169 technique::
170
171 @lru_cache(maxsize=None)
172 def fib(n):
173 if n < 2:
174 return n
175 return fib(n-1) + fib(n-2)
176
Raymond Hettinger17328e42013-04-06 20:27:33 -0700177 >>> [fib(n) for n in range(16)]
Raymond Hettingerc79fb0e2010-12-01 03:45:41 +0000178 [0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610]
179
Raymond Hettinger17328e42013-04-06 20:27:33 -0700180 >>> fib.cache_info()
Raymond Hettingerc79fb0e2010-12-01 03:45:41 +0000181 CacheInfo(hits=28, misses=16, maxsize=None, currsize=16)
182
Georg Brandl2e7346a2010-07-31 18:09:23 +0000183 .. versionadded:: 3.2
184
Raymond Hettingercd9fdfd2011-10-20 08:57:45 -0700185 .. versionchanged:: 3.3
186 Added the *typed* option.
187
Raymond Hettingerb8218682019-05-26 11:27:35 -0700188 .. versionchanged:: 3.8
189 Added the *user_function* option.
190
Manjusaka051ff522019-11-12 15:30:18 +0800191 .. versionadded:: 3.9
192 Added the function :func:`cache_parameters`
193
Georg Brandl8a1caa22010-07-29 16:01:11 +0000194.. decorator:: total_ordering
Raymond Hettingerc50846a2010-04-05 18:56:31 +0000195
196 Given a class defining one or more rich comparison ordering methods, this
Benjamin Peterson08bf91c2010-04-11 16:12:57 +0000197 class decorator supplies the rest. This simplifies the effort involved
Raymond Hettingerc50846a2010-04-05 18:56:31 +0000198 in specifying all of the possible rich comparison operations:
199
200 The class must define one of :meth:`__lt__`, :meth:`__le__`,
201 :meth:`__gt__`, or :meth:`__ge__`.
202 In addition, the class should supply an :meth:`__eq__` method.
203
204 For example::
205
206 @total_ordering
207 class Student:
Nick Coghlanf05d9812013-10-02 00:02:03 +1000208 def _is_valid_operand(self, other):
209 return (hasattr(other, "lastname") and
210 hasattr(other, "firstname"))
Raymond Hettingerc50846a2010-04-05 18:56:31 +0000211 def __eq__(self, other):
Nick Coghlanf05d9812013-10-02 00:02:03 +1000212 if not self._is_valid_operand(other):
213 return NotImplemented
Raymond Hettingerc50846a2010-04-05 18:56:31 +0000214 return ((self.lastname.lower(), self.firstname.lower()) ==
215 (other.lastname.lower(), other.firstname.lower()))
216 def __lt__(self, other):
Nick Coghlanf05d9812013-10-02 00:02:03 +1000217 if not self._is_valid_operand(other):
218 return NotImplemented
Raymond Hettingerc50846a2010-04-05 18:56:31 +0000219 return ((self.lastname.lower(), self.firstname.lower()) <
220 (other.lastname.lower(), other.firstname.lower()))
221
Nick Coghlanf05d9812013-10-02 00:02:03 +1000222 .. note::
223
224 While this decorator makes it easy to create well behaved totally
225 ordered types, it *does* come at the cost of slower execution and
226 more complex stack traces for the derived comparison methods. If
227 performance benchmarking indicates this is a bottleneck for a given
228 application, implementing all six rich comparison methods instead is
229 likely to provide an easy speed boost.
230
Raymond Hettingerc50846a2010-04-05 18:56:31 +0000231 .. versionadded:: 3.2
232
Nick Coghlanf05d9812013-10-02 00:02:03 +1000233 .. versionchanged:: 3.4
234 Returning NotImplemented from the underlying comparison function for
235 unrecognised types is now supported.
Georg Brandl67b21b72010-08-17 15:07:14 +0000236
Serhiy Storchaka2085bd02019-06-01 11:00:15 +0300237.. function:: partial(func, /, *args, **keywords)
Georg Brandl116aa622007-08-15 14:28:22 +0000238
Andrei Petre83a07652018-10-22 23:11:20 -0700239 Return a new :ref:`partial object<partial-objects>` which when called
240 will behave like *func* called with the positional arguments *args*
241 and keyword arguments *keywords*. If more arguments are supplied to the
242 call, they are appended to *args*. If additional keyword arguments are
243 supplied, they extend and override *keywords*.
Georg Brandl116aa622007-08-15 14:28:22 +0000244 Roughly equivalent to::
245
Serhiy Storchaka2085bd02019-06-01 11:00:15 +0300246 def partial(func, /, *args, **keywords):
Georg Brandl116aa622007-08-15 14:28:22 +0000247 def newfunc(*fargs, **fkeywords):
Sergey Fedoseevb981fec2018-10-20 02:42:07 +0500248 newkeywords = {**keywords, **fkeywords}
Martin Panter0c0da482016-06-12 01:46:50 +0000249 return func(*args, *fargs, **newkeywords)
Georg Brandl116aa622007-08-15 14:28:22 +0000250 newfunc.func = func
251 newfunc.args = args
252 newfunc.keywords = keywords
253 return newfunc
254
255 The :func:`partial` is used for partial function application which "freezes"
256 some portion of a function's arguments and/or keywords resulting in a new object
257 with a simplified signature. For example, :func:`partial` can be used to create
258 a callable that behaves like the :func:`int` function where the *base* argument
Christian Heimesfe337bf2008-03-23 21:54:12 +0000259 defaults to two:
Georg Brandl116aa622007-08-15 14:28:22 +0000260
Christian Heimesfe337bf2008-03-23 21:54:12 +0000261 >>> from functools import partial
Georg Brandl116aa622007-08-15 14:28:22 +0000262 >>> basetwo = partial(int, base=2)
263 >>> basetwo.__doc__ = 'Convert base 2 string to an int.'
264 >>> basetwo('10010')
265 18
266
267
Serhiy Storchaka70c5f2a2019-06-01 11:38:24 +0300268.. class:: partialmethod(func, /, *args, **keywords)
Nick Coghlanf4cb48a2013-11-03 16:41:46 +1000269
270 Return a new :class:`partialmethod` descriptor which behaves
271 like :class:`partial` except that it is designed to be used as a method
272 definition rather than being directly callable.
273
274 *func* must be a :term:`descriptor` or a callable (objects which are both,
275 like normal functions, are handled as descriptors).
276
277 When *func* is a descriptor (such as a normal Python function,
278 :func:`classmethod`, :func:`staticmethod`, :func:`abstractmethod` or
279 another instance of :class:`partialmethod`), calls to ``__get__`` are
280 delegated to the underlying descriptor, and an appropriate
Andrei Petre83a07652018-10-22 23:11:20 -0700281 :ref:`partial object<partial-objects>` returned as the result.
Nick Coghlanf4cb48a2013-11-03 16:41:46 +1000282
283 When *func* is a non-descriptor callable, an appropriate bound method is
284 created dynamically. This behaves like a normal Python function when
285 used as a method: the *self* argument will be inserted as the first
286 positional argument, even before the *args* and *keywords* supplied to
287 the :class:`partialmethod` constructor.
288
289 Example::
290
Serhiy Storchakae042a452019-06-10 13:35:52 +0300291 >>> class Cell:
Benjamin Peterson3a434032014-03-30 15:07:09 -0400292 ... def __init__(self):
293 ... self._alive = False
Nick Coghlanf4cb48a2013-11-03 16:41:46 +1000294 ... @property
295 ... def alive(self):
296 ... return self._alive
297 ... def set_state(self, state):
298 ... self._alive = bool(state)
Nick Coghlan3daaf5f2013-11-04 23:32:16 +1000299 ... set_alive = partialmethod(set_state, True)
300 ... set_dead = partialmethod(set_state, False)
Nick Coghlanf4cb48a2013-11-03 16:41:46 +1000301 ...
302 >>> c = Cell()
303 >>> c.alive
304 False
305 >>> c.set_alive()
306 >>> c.alive
307 True
308
309 .. versionadded:: 3.4
310
311
Georg Brandl58f9e4f2008-04-19 22:18:33 +0000312.. function:: reduce(function, iterable[, initializer])
Georg Brandl116aa622007-08-15 14:28:22 +0000313
Brendan Jurd9df10022018-10-01 16:52:10 +1000314 Apply *function* of two arguments cumulatively to the items of *iterable*, from
315 left to right, so as to reduce the iterable to a single value. For example,
Georg Brandl116aa622007-08-15 14:28:22 +0000316 ``reduce(lambda x, y: x+y, [1, 2, 3, 4, 5])`` calculates ``((((1+2)+3)+4)+5)``.
317 The left argument, *x*, is the accumulated value and the right argument, *y*, is
Brendan Jurd9df10022018-10-01 16:52:10 +1000318 the update value from the *iterable*. If the optional *initializer* is present,
319 it is placed before the items of the iterable in the calculation, and serves as
320 a default when the iterable is empty. If *initializer* is not given and
321 *iterable* contains only one item, the first item is returned.
Georg Brandl116aa622007-08-15 14:28:22 +0000322
Raymond Hettinger558dcf32014-12-16 18:16:57 -0800323 Roughly equivalent to::
Raymond Hettinger64801682013-10-12 16:04:17 -0700324
325 def reduce(function, iterable, initializer=None):
326 it = iter(iterable)
327 if initializer is None:
328 value = next(it)
329 else:
330 value = initializer
331 for element in it:
332 value = function(value, element)
333 return value
334
Gerrit Hollbd81cbd2018-07-04 23:26:32 +0100335 See :func:`itertools.accumulate` for an iterator that yields all intermediate
336 values.
Georg Brandl116aa622007-08-15 14:28:22 +0000337
Daisuke Miyakawa0e61e672017-10-12 23:39:43 +0900338.. decorator:: singledispatch
Łukasz Langa6f692512013-06-05 12:20:24 +0200339
Daisuke Miyakawa0e61e672017-10-12 23:39:43 +0900340 Transform a function into a :term:`single-dispatch <single
Łukasz Langafdcf2b72013-06-07 22:54:03 +0200341 dispatch>` :term:`generic function`.
Łukasz Langa6f692512013-06-05 12:20:24 +0200342
343 To define a generic function, decorate it with the ``@singledispatch``
344 decorator. Note that the dispatch happens on the type of the first argument,
345 create your function accordingly::
346
347 >>> from functools import singledispatch
348 >>> @singledispatch
349 ... def fun(arg, verbose=False):
350 ... if verbose:
351 ... print("Let me just say,", end=" ")
352 ... print(arg)
353
354 To add overloaded implementations to the function, use the :func:`register`
Łukasz Langae5697532017-12-11 13:56:31 -0800355 attribute of the generic function. It is a decorator. For functions
356 annotated with types, the decorator will infer the type of the first
357 argument automatically::
Łukasz Langa6f692512013-06-05 12:20:24 +0200358
Łukasz Langae5697532017-12-11 13:56:31 -0800359 >>> @fun.register
360 ... def _(arg: int, verbose=False):
Łukasz Langa6f692512013-06-05 12:20:24 +0200361 ... if verbose:
362 ... print("Strength in numbers, eh?", end=" ")
363 ... print(arg)
364 ...
Łukasz Langae5697532017-12-11 13:56:31 -0800365 >>> @fun.register
366 ... def _(arg: list, verbose=False):
Łukasz Langa6f692512013-06-05 12:20:24 +0200367 ... if verbose:
368 ... print("Enumerate this:")
369 ... for i, elem in enumerate(arg):
370 ... print(i, elem)
371
Łukasz Langae5697532017-12-11 13:56:31 -0800372 For code which doesn't use type annotations, the appropriate type
373 argument can be passed explicitly to the decorator itself::
374
375 >>> @fun.register(complex)
376 ... def _(arg, verbose=False):
377 ... if verbose:
378 ... print("Better than complicated.", end=" ")
379 ... print(arg.real, arg.imag)
380 ...
381
382
Łukasz Langa6f692512013-06-05 12:20:24 +0200383 To enable registering lambdas and pre-existing functions, the
384 :func:`register` attribute can be used in a functional form::
385
386 >>> def nothing(arg, verbose=False):
387 ... print("Nothing.")
388 ...
389 >>> fun.register(type(None), nothing)
390
391 The :func:`register` attribute returns the undecorated function which
392 enables decorator stacking, pickling, as well as creating unit tests for
393 each variant independently::
394
395 >>> @fun.register(float)
396 ... @fun.register(Decimal)
397 ... def fun_num(arg, verbose=False):
398 ... if verbose:
399 ... print("Half of your number:", end=" ")
400 ... print(arg / 2)
401 ...
402 >>> fun_num is fun
403 False
404
405 When called, the generic function dispatches on the type of the first
406 argument::
407
408 >>> fun("Hello, world.")
409 Hello, world.
410 >>> fun("test.", verbose=True)
411 Let me just say, test.
412 >>> fun(42, verbose=True)
413 Strength in numbers, eh? 42
414 >>> fun(['spam', 'spam', 'eggs', 'spam'], verbose=True)
415 Enumerate this:
416 0 spam
417 1 spam
418 2 eggs
419 3 spam
420 >>> fun(None)
421 Nothing.
422 >>> fun(1.23)
423 0.615
424
425 Where there is no registered implementation for a specific type, its
426 method resolution order is used to find a more generic implementation.
427 The original function decorated with ``@singledispatch`` is registered
428 for the base ``object`` type, which means it is used if no better
429 implementation is found.
430
Batuhan Taşkaya24555ce2019-11-19 11:16:46 +0300431 If an implementation registered to :term:`abstract base class`, virtual
432 subclasses will be dispatched to that implementation::
433
434 >>> from collections.abc import Mapping
435 >>> @fun.register
436 ... def _(arg: Mapping, verbose=False):
437 ... if verbose:
438 ... print("Keys & Values")
439 ... for key, value in arg.items():
440 ... print(key, "=>", value)
441 ...
442 >>> fun({"a": "b"})
443 a => b
444
Łukasz Langa6f692512013-06-05 12:20:24 +0200445 To check which implementation will the generic function choose for
446 a given type, use the ``dispatch()`` attribute::
447
448 >>> fun.dispatch(float)
449 <function fun_num at 0x1035a2840>
450 >>> fun.dispatch(dict) # note: default implementation
451 <function fun at 0x103fe0000>
452
453 To access all registered implementations, use the read-only ``registry``
454 attribute::
455
456 >>> fun.registry.keys()
457 dict_keys([<class 'NoneType'>, <class 'int'>, <class 'object'>,
458 <class 'decimal.Decimal'>, <class 'list'>,
459 <class 'float'>])
460 >>> fun.registry[float]
461 <function fun_num at 0x1035a2840>
462 >>> fun.registry[object]
463 <function fun at 0x103fe0000>
464
465 .. versionadded:: 3.4
466
Łukasz Langae5697532017-12-11 13:56:31 -0800467 .. versionchanged:: 3.7
468 The :func:`register` attribute supports using type annotations.
469
Łukasz Langa6f692512013-06-05 12:20:24 +0200470
Ethan Smithc6512752018-05-26 16:38:33 -0400471.. class:: singledispatchmethod(func)
472
473 Transform a method into a :term:`single-dispatch <single
474 dispatch>` :term:`generic function`.
475
476 To define a generic method, decorate it with the ``@singledispatchmethod``
477 decorator. Note that the dispatch happens on the type of the first non-self
478 or non-cls argument, create your function accordingly::
479
480 class Negator:
481 @singledispatchmethod
482 def neg(self, arg):
483 raise NotImplementedError("Cannot negate a")
484
485 @neg.register
486 def _(self, arg: int):
487 return -arg
488
489 @neg.register
490 def _(self, arg: bool):
491 return not arg
492
493 ``@singledispatchmethod`` supports nesting with other decorators such as
494 ``@classmethod``. Note that to allow for ``dispatcher.register``,
495 ``singledispatchmethod`` must be the *outer most* decorator. Here is the
496 ``Negator`` class with the ``neg`` methods being class bound::
497
498 class Negator:
499 @singledispatchmethod
500 @classmethod
501 def neg(cls, arg):
502 raise NotImplementedError("Cannot negate a")
503
504 @neg.register
505 @classmethod
506 def _(cls, arg: int):
507 return -arg
508
509 @neg.register
510 @classmethod
511 def _(cls, arg: bool):
512 return not arg
513
514 The same pattern can be used for other similar decorators: ``staticmethod``,
515 ``abstractmethod``, and others.
516
Inada Naokibc284f02019-03-27 18:15:17 +0900517 .. versionadded:: 3.8
518
519
Pablo Galindo99e6c262020-01-23 15:29:52 +0000520.. class:: TopologicalSorter(graph=None)
521
522 Provides functionality to topologically sort a graph of hashable nodes.
523
Pablo Galindo65ecc392020-01-23 21:01:50 +0000524 A topological order is a linear ordering of the vertices in a graph such that
525 for every directed edge u -> v from vertex u to vertex v, vertex u comes
526 before vertex v in the ordering. For instance, the vertices of the graph may
527 represent tasks to be performed, and the edges may represent constraints that
528 one task must be performed before another; in this example, a topological
529 ordering is just a valid sequence for the tasks. A complete topological
530 ordering is possible if and only if the graph has no directed cycles, that
531 is, if it is a directed acyclic graph.
Pablo Galindo99e6c262020-01-23 15:29:52 +0000532
Pablo Galindo65ecc392020-01-23 21:01:50 +0000533 If the optional *graph* argument is provided it must be a dictionary
534 representing a directed acyclic graph where the keys are nodes and the values
535 are iterables of all predecessors of that node in the graph (the nodes that
536 have edges that point to the value in the key). Additional nodes can be added
537 to the graph using the :meth:`~TopologicalSorter.add` method.
Pablo Galindo99e6c262020-01-23 15:29:52 +0000538
Pablo Galindo65ecc392020-01-23 21:01:50 +0000539 In the general case, the steps required to perform the sorting of a given
540 graph are as follows:
Pablo Galindo99e6c262020-01-23 15:29:52 +0000541
Pablo Galindo65ecc392020-01-23 21:01:50 +0000542 * Create an instance of the :class:`TopologicalSorter` with an optional
543 initial graph.
Pablo Galindo99e6c262020-01-23 15:29:52 +0000544 * Add additional nodes to the graph.
545 * Call :meth:`~TopologicalSorter.prepare` on the graph.
Pablo Galindo65ecc392020-01-23 21:01:50 +0000546 * While :meth:`~TopologicalSorter.is_active` is ``True``, iterate over
547 the nodes returned by :meth:`~TopologicalSorter.get_ready` and
548 process them. Call :meth:`~TopologicalSorter.done` on each node as it
549 finishes processing.
Pablo Galindo99e6c262020-01-23 15:29:52 +0000550
551 In case just an immediate sorting of the nodes in the graph is required and
Pablo Galindo65ecc392020-01-23 21:01:50 +0000552 no parallelism is involved, the convenience method
553 :meth:`TopologicalSorter.static_order` can be used directly:
Pablo Galindo99e6c262020-01-23 15:29:52 +0000554
555 .. doctest::
556
Pablo Galindo65ecc392020-01-23 21:01:50 +0000557 >>> graph = {"D": {"B", "C"}, "C": {"A"}, "B": {"A"}}
Pablo Galindo99e6c262020-01-23 15:29:52 +0000558 >>> ts = TopologicalSorter(graph)
Pablo Galindo65ecc392020-01-23 21:01:50 +0000559 >>> tuple(ts.static_order())
560 ('A', 'C', 'B', 'D')
Pablo Galindo99e6c262020-01-23 15:29:52 +0000561
Pablo Galindo65ecc392020-01-23 21:01:50 +0000562 The class is designed to easily support parallel processing of the nodes as
563 they become ready. For instance::
Pablo Galindo99e6c262020-01-23 15:29:52 +0000564
565 topological_sorter = TopologicalSorter()
566
567 # Add nodes to 'topological_sorter'...
568
569 topological_sorter.prepare()
570 while topological_sorter.is_active():
571 for node in topological_sorter.get_ready():
572 # Worker threads or processes take nodes to work on off the
573 # 'task_queue' queue.
574 task_queue.put(node)
575
576 # When the work for a node is done, workers put the node in
577 # 'finalized_tasks_queue' so we can get more nodes to work on.
578 # The definition of 'is_active()' guarantees that, at this point, at
579 # least one node has been placed on 'task_queue' that hasn't yet
580 # been passed to 'done()', so this blocking 'get()' must (eventually)
581 # succeed. After calling 'done()', we loop back to call 'get_ready()'
582 # again, so put newly freed nodes on 'task_queue' as soon as
583 # logically possible.
584 node = finalized_tasks_queue.get()
585 topological_sorter.done(node)
586
587 .. method:: add(node, *predecessors)
588
Pablo Galindo65ecc392020-01-23 21:01:50 +0000589 Add a new node and its predecessors to the graph. Both the *node* and all
590 elements in *predecessors* must be hashable.
Pablo Galindo99e6c262020-01-23 15:29:52 +0000591
Pablo Galindo65ecc392020-01-23 21:01:50 +0000592 If called multiple times with the same node argument, the set of
593 dependencies will be the union of all dependencies passed in.
Pablo Galindo99e6c262020-01-23 15:29:52 +0000594
595 It is possible to add a node with no dependencies (*predecessors* is not
596 provided) or to provide a dependency twice. If a node that has not been
Pablo Galindo65ecc392020-01-23 21:01:50 +0000597 provided before is included among *predecessors* it will be automatically
598 added to the graph with no predecessors of its own.
Pablo Galindo99e6c262020-01-23 15:29:52 +0000599
600 Raises :exc:`ValueError` if called after :meth:`~TopologicalSorter.prepare`.
601
602 .. method:: prepare()
603
Pablo Galindo65ecc392020-01-23 21:01:50 +0000604 Mark the graph as finished and check for cycles in the graph. If any cycle
605 is detected, :exc:`CycleError` will be raised, but
606 :meth:`~TopologicalSorter.get_ready` can still be used to obtain as many
607 nodes as possible until cycles block more progress. After a call to this
608 function, the graph cannot be modified, and therefore no more nodes can be
609 added using :meth:`~TopologicalSorter.add`.
Pablo Galindo99e6c262020-01-23 15:29:52 +0000610
611 .. method:: is_active()
612
Pablo Galindo65ecc392020-01-23 21:01:50 +0000613 Returns ``True`` if more progress can be made and ``False`` otherwise.
614 Progress can be made if cycles do not block the resolution and either
615 there are still nodes ready that haven't yet been returned by
Pablo Galindo99e6c262020-01-23 15:29:52 +0000616 :meth:`TopologicalSorter.get_ready` or the number of nodes marked
Pablo Galindo65ecc392020-01-23 21:01:50 +0000617 :meth:`TopologicalSorter.done` is less than the number that have been
618 returned by :meth:`TopologicalSorter.get_ready`.
Pablo Galindo99e6c262020-01-23 15:29:52 +0000619
Pablo Galindo65ecc392020-01-23 21:01:50 +0000620 The :meth:`~TopologicalSorter.__bool__` method of this class defers to
621 this function, so instead of::
Pablo Galindo99e6c262020-01-23 15:29:52 +0000622
623 if ts.is_active():
624 ...
625
626 if possible to simply do::
627
628 if ts:
629 ...
630
Pablo Galindo65ecc392020-01-23 21:01:50 +0000631 Raises :exc:`ValueError` if called without calling
632 :meth:`~TopologicalSorter.prepare` previously.
Pablo Galindo99e6c262020-01-23 15:29:52 +0000633
634 .. method:: done(*nodes)
635
636 Marks a set of nodes returned by :meth:`TopologicalSorter.get_ready` as
Pablo Galindo65ecc392020-01-23 21:01:50 +0000637 processed, unblocking any successor of each node in *nodes* for being
638 returned in the future by a call to :meth:`TopologicalSorter.get_ready`.
Pablo Galindo99e6c262020-01-23 15:29:52 +0000639
640 Raises :exc:`ValueError` if any node in *nodes* has already been marked as
Pablo Galindo65ecc392020-01-23 21:01:50 +0000641 processed by a previous call to this method or if a node was not added to
642 the graph by using :meth:`TopologicalSorter.add`, if called without
643 calling :meth:`~TopologicalSorter.prepare` or if node has not yet been
644 returned by :meth:`~TopologicalSorter.get_ready`.
Pablo Galindo99e6c262020-01-23 15:29:52 +0000645
646 .. method:: get_ready()
647
Pablo Galindo65ecc392020-01-23 21:01:50 +0000648 Returns a ``tuple`` with all the nodes that are ready. Initially it
649 returns all nodes with no predecessors, and once those are marked as
650 processed by calling :meth:`TopologicalSorter.done`, further calls will
651 return all new nodes that have all their predecessors already processed.
652 Once no more progress can be made, empty tuples are returned.
Pablo Galindo99e6c262020-01-23 15:29:52 +0000653
654 Raises :exc:`ValueError` if called without calling
655 :meth:`~TopologicalSorter.prepare` previously.
656
657 .. method:: static_order()
658
659 Returns an iterable of nodes in a topological order. Using this method
660 does not require to call :meth:`TopologicalSorter.prepare` or
661 :meth:`TopologicalSorter.done`. This method is equivalent to::
662
663 def static_order(self):
664 self.prepare()
665 while self.is_active():
666 node_group = self.get_ready()
667 yield from node_group
668 self.done(*node_group)
669
670 The particular order that is returned may depend on the specific order in
671 which the items were inserted in the graph. For example:
672
673 .. doctest::
674
675 >>> ts = TopologicalSorter()
676 >>> ts.add(3, 2, 1)
677 >>> ts.add(1, 0)
678 >>> print([*ts.static_order()])
679 [2, 0, 1, 3]
680
681 >>> ts2 = TopologicalSorter()
682 >>> ts2.add(1, 0)
683 >>> ts2.add(3, 2, 1)
684 >>> print([*ts2.static_order()])
685 [0, 2, 1, 3]
686
Pablo Galindo65ecc392020-01-23 21:01:50 +0000687 This is due to the fact that "0" and "2" are in the same level in the
688 graph (they would have been returned in the same call to
689 :meth:`~TopologicalSorter.get_ready`) and the order between them is
690 determined by the order of insertion.
Pablo Galindo99e6c262020-01-23 15:29:52 +0000691
692
693 If any cycle is detected, :exc:`CycleError` will be raised.
694
695 .. versionadded:: 3.9
696
697
Georg Brandl036490d2009-05-17 13:00:36 +0000698.. function:: update_wrapper(wrapper, wrapped, assigned=WRAPPER_ASSIGNMENTS, updated=WRAPPER_UPDATES)
Georg Brandl116aa622007-08-15 14:28:22 +0000699
700 Update a *wrapper* function to look like the *wrapped* function. The optional
701 arguments are tuples to specify which attributes of the original function are
702 assigned directly to the matching attributes on the wrapper function and which
703 attributes of the wrapper function are updated with the corresponding attributes
704 from the original function. The default values for these arguments are the
Berker Peksag472233e2016-04-18 21:20:50 +0300705 module level constants ``WRAPPER_ASSIGNMENTS`` (which assigns to the wrapper
706 function's ``__module__``, ``__name__``, ``__qualname__``, ``__annotations__``
707 and ``__doc__``, the documentation string) and ``WRAPPER_UPDATES`` (which
708 updates the wrapper function's ``__dict__``, i.e. the instance dictionary).
Georg Brandl116aa622007-08-15 14:28:22 +0000709
Nick Coghlan98876832010-08-17 06:17:18 +0000710 To allow access to the original function for introspection and other purposes
711 (e.g. bypassing a caching decorator such as :func:`lru_cache`), this function
Nick Coghlan24c05bc2013-07-15 21:13:08 +1000712 automatically adds a ``__wrapped__`` attribute to the wrapper that refers to
713 the function being wrapped.
Nick Coghlan98876832010-08-17 06:17:18 +0000714
Christian Heimesd8654cf2007-12-02 15:22:16 +0000715 The main intended use for this function is in :term:`decorator` functions which
716 wrap the decorated function and return the wrapper. If the wrapper function is
717 not updated, the metadata of the returned function will reflect the wrapper
Georg Brandl116aa622007-08-15 14:28:22 +0000718 definition rather than the original function definition, which is typically less
719 than helpful.
720
Nick Coghlan98876832010-08-17 06:17:18 +0000721 :func:`update_wrapper` may be used with callables other than functions. Any
722 attributes named in *assigned* or *updated* that are missing from the object
723 being wrapped are ignored (i.e. this function will not attempt to set them
724 on the wrapper function). :exc:`AttributeError` is still raised if the
725 wrapper function itself is missing any attributes named in *updated*.
726
727 .. versionadded:: 3.2
Georg Brandl9e257012010-08-17 14:11:59 +0000728 Automatic addition of the ``__wrapped__`` attribute.
Nick Coghlan98876832010-08-17 06:17:18 +0000729
730 .. versionadded:: 3.2
Georg Brandl9e257012010-08-17 14:11:59 +0000731 Copying of the ``__annotations__`` attribute by default.
Nick Coghlan98876832010-08-17 06:17:18 +0000732
733 .. versionchanged:: 3.2
Georg Brandl9e257012010-08-17 14:11:59 +0000734 Missing attributes no longer trigger an :exc:`AttributeError`.
735
Nick Coghlan24c05bc2013-07-15 21:13:08 +1000736 .. versionchanged:: 3.4
737 The ``__wrapped__`` attribute now always refers to the wrapped
738 function, even if that function defined a ``__wrapped__`` attribute.
739 (see :issue:`17482`)
740
Georg Brandl116aa622007-08-15 14:28:22 +0000741
Georg Brandl8a1caa22010-07-29 16:01:11 +0000742.. decorator:: wraps(wrapped, assigned=WRAPPER_ASSIGNMENTS, updated=WRAPPER_UPDATES)
Georg Brandl116aa622007-08-15 14:28:22 +0000743
Ezio Melotti67f6d5f2014-08-05 08:14:28 +0300744 This is a convenience function for invoking :func:`update_wrapper` as a
745 function decorator when defining a wrapper function. It is equivalent to
746 ``partial(update_wrapper, wrapped=wrapped, assigned=assigned, updated=updated)``.
747 For example::
Georg Brandl116aa622007-08-15 14:28:22 +0000748
Christian Heimesfe337bf2008-03-23 21:54:12 +0000749 >>> from functools import wraps
Georg Brandl116aa622007-08-15 14:28:22 +0000750 >>> def my_decorator(f):
751 ... @wraps(f)
752 ... def wrapper(*args, **kwds):
Georg Brandl6911e3c2007-09-04 07:15:32 +0000753 ... print('Calling decorated function')
Georg Brandl116aa622007-08-15 14:28:22 +0000754 ... return f(*args, **kwds)
755 ... return wrapper
756 ...
757 >>> @my_decorator
758 ... def example():
759 ... """Docstring"""
Georg Brandl6911e3c2007-09-04 07:15:32 +0000760 ... print('Called example function')
Georg Brandl116aa622007-08-15 14:28:22 +0000761 ...
762 >>> example()
763 Calling decorated function
764 Called example function
765 >>> example.__name__
766 'example'
767 >>> example.__doc__
768 'Docstring'
769
770 Without the use of this decorator factory, the name of the example function
771 would have been ``'wrapper'``, and the docstring of the original :func:`example`
772 would have been lost.
773
774
775.. _partial-objects:
776
777:class:`partial` Objects
778------------------------
779
780:class:`partial` objects are callable objects created by :func:`partial`. They
781have three read-only attributes:
782
783
784.. attribute:: partial.func
785
786 A callable object or function. Calls to the :class:`partial` object will be
787 forwarded to :attr:`func` with new arguments and keywords.
788
789
790.. attribute:: partial.args
791
792 The leftmost positional arguments that will be prepended to the positional
793 arguments provided to a :class:`partial` object call.
794
795
796.. attribute:: partial.keywords
797
798 The keyword arguments that will be supplied when the :class:`partial` object is
799 called.
800
801:class:`partial` objects are like :class:`function` objects in that they are
802callable, weak referencable, and can have attributes. There are some important
Martin Panterbae5d812016-06-18 03:57:31 +0000803differences. For instance, the :attr:`~definition.__name__` and :attr:`__doc__` attributes
Georg Brandl116aa622007-08-15 14:28:22 +0000804are not created automatically. Also, :class:`partial` objects defined in
805classes behave like static methods and do not transform into bound methods
806during instance attribute look-up.
Pablo Galindo99e6c262020-01-23 15:29:52 +0000807
808
809Exceptions
810----------
811The :mod:`functools` module defines the following exception classes:
812
813.. exception:: CycleError
814
815 Subclass of :exc:`ValueError` raised by :meth:`TopologicalSorter.prepare` if cycles exist
816 in the working graph. If multiple cycles exist, only one undefined choice among them will
817 be reported and included in the exception.
818
819 The detected cycle can be accessed via the second element in the :attr:`~CycleError.args`
820 attribute of the exception instance and consists in a list of nodes, such that each node is,
821 in the graph, an immediate predecessor of the next node in the list. In the reported list,
822 the first and the last node will be the same, to make it clear that it is cyclic.