blob: e708a0d99cd004095d6a3841dfa3a96cbdeafe9b [file] [log] [blame]
Georg Brandl6c89a792012-01-25 22:36:25 +01001:mod:`functools` --- Higher-order functions and operations on callable objects
Georg Brandl116aa622007-08-15 14:28:22 +00002==============================================================================
3
4.. module:: functools
Georg Brandl6c89a792012-01-25 22:36:25 +01005 :synopsis: Higher-order functions and operations on callable objects.
Terry Jan Reedyfa089b92016-06-11 15:02:54 -04006
Georg Brandl116aa622007-08-15 14:28:22 +00007.. moduleauthor:: Peter Harris <scav@blueyonder.co.uk>
8.. moduleauthor:: Raymond Hettinger <python@rcn.com>
9.. moduleauthor:: Nick Coghlan <ncoghlan@gmail.com>
Łukasz Langa6f692512013-06-05 12:20:24 +020010.. moduleauthor:: Łukasz Langa <lukasz@langa.pl>
Pablo Galindo99e6c262020-01-23 15:29:52 +000011.. moduleauthor:: Pablo Galindo <pablogsal@gmail.com>
Georg Brandl116aa622007-08-15 14:28:22 +000012.. sectionauthor:: Peter Harris <scav@blueyonder.co.uk>
13
Raymond Hettinger05ce0792011-01-10 21:16:07 +000014**Source code:** :source:`Lib/functools.py`
15
Pablo Galindo99e6c262020-01-23 15:29:52 +000016.. testsetup:: default
17
18 import functools
19 from functools import *
20
Raymond Hettinger05ce0792011-01-10 21:16:07 +000021--------------
Georg Brandl116aa622007-08-15 14:28:22 +000022
Georg Brandl116aa622007-08-15 14:28:22 +000023The :mod:`functools` module is for higher-order functions: functions that act on
24or return other functions. In general, any callable object can be treated as a
25function for the purposes of this module.
26
Thomas Woutersed03b412007-08-28 21:37:11 +000027The :mod:`functools` module defines the following functions:
28
Carl Meyerd658dea2018-08-28 01:11:56 -060029.. decorator:: cached_property(func)
30
31 Transform a method of a class into a property whose value is computed once
32 and then cached as a normal attribute for the life of the instance. Similar
33 to :func:`property`, with the addition of caching. Useful for expensive
34 computed properties of instances that are otherwise effectively immutable.
35
36 Example::
37
38 class DataSet:
39 def __init__(self, sequence_of_numbers):
40 self._data = sequence_of_numbers
41
42 @cached_property
43 def stdev(self):
44 return statistics.stdev(self._data)
45
46 @cached_property
47 def variance(self):
48 return statistics.variance(self._data)
49
50 .. versionadded:: 3.8
51
52 .. note::
53
54 This decorator requires that the ``__dict__`` attribute on each instance
55 be a mutable mapping. This means it will not work with some types, such as
56 metaclasses (since the ``__dict__`` attributes on type instances are
57 read-only proxies for the class namespace), and those that specify
58 ``__slots__`` without including ``__dict__`` as one of the defined slots
59 (as such classes don't provide a ``__dict__`` attribute at all).
60
61
Éric Araujob10089e2010-11-18 14:22:08 +000062.. function:: cmp_to_key(func)
Raymond Hettingerc50846a2010-04-05 18:56:31 +000063
Raymond Hettinger86e9b6b2014-11-09 17:20:56 -080064 Transform an old-style comparison function to a :term:`key function`. Used
65 with tools that accept key functions (such as :func:`sorted`, :func:`min`,
Benjamin Petersoncca65312010-08-09 02:13:10 +000066 :func:`max`, :func:`heapq.nlargest`, :func:`heapq.nsmallest`,
67 :func:`itertools.groupby`). This function is primarily used as a transition
Ezio Melotti9ecb6be2012-01-16 08:28:54 +020068 tool for programs being converted from Python 2 which supported the use of
Benjamin Petersoncca65312010-08-09 02:13:10 +000069 comparison functions.
Raymond Hettingerc50846a2010-04-05 18:56:31 +000070
Georg Brandl6c89a792012-01-25 22:36:25 +010071 A comparison function is any callable that accept two arguments, compares them,
Benjamin Petersoncca65312010-08-09 02:13:10 +000072 and returns a negative number for less-than, zero for equality, or a positive
73 number for greater-than. A key function is a callable that accepts one
Raymond Hettinger86e9b6b2014-11-09 17:20:56 -080074 argument and returns another value to be used as the sort key.
Raymond Hettingerc50846a2010-04-05 18:56:31 +000075
Benjamin Petersoncca65312010-08-09 02:13:10 +000076 Example::
Raymond Hettingerc50846a2010-04-05 18:56:31 +000077
Benjamin Petersoncca65312010-08-09 02:13:10 +000078 sorted(iterable, key=cmp_to_key(locale.strcoll)) # locale-aware sort order
Raymond Hettingerc50846a2010-04-05 18:56:31 +000079
Raymond Hettinger86e9b6b2014-11-09 17:20:56 -080080 For sorting examples and a brief sorting tutorial, see :ref:`sortinghowto`.
81
Raymond Hettingerc50846a2010-04-05 18:56:31 +000082 .. versionadded:: 3.2
83
Georg Brandl67b21b72010-08-17 15:07:14 +000084
Raymond Hettingerb8218682019-05-26 11:27:35 -070085.. decorator:: lru_cache(user_function)
86 lru_cache(maxsize=128, typed=False)
Georg Brandl2e7346a2010-07-31 18:09:23 +000087
88 Decorator to wrap a function with a memoizing callable that saves up to the
89 *maxsize* most recent calls. It can save time when an expensive or I/O bound
90 function is periodically called with the same arguments.
91
Raymond Hettinger7496b412010-11-30 19:15:45 +000092 Since a dictionary is used to cache results, the positional and keyword
93 arguments to the function must be hashable.
Georg Brandl2e7346a2010-07-31 18:09:23 +000094
Raymond Hettinger902bcd92018-09-14 00:53:20 -070095 Distinct argument patterns may be considered to be distinct calls with
96 separate cache entries. For example, `f(a=1, b=2)` and `f(b=2, a=1)`
97 differ in their keyword argument order and may have two separate cache
98 entries.
99
Raymond Hettingerb8218682019-05-26 11:27:35 -0700100 If *user_function* is specified, it must be a callable. This allows the
101 *lru_cache* decorator to be applied directly to a user function, leaving
102 the *maxsize* at its default value of 128::
103
104 @lru_cache
105 def count_vowels(sentence):
106 sentence = sentence.casefold()
107 return sum(sentence.count(vowel) for vowel in 'aeiou')
108
Serhiy Storchakaecf41da2016-10-19 16:29:26 +0300109 If *maxsize* is set to ``None``, the LRU feature is disabled and the cache can
Raymond Hettinger7d74eff2012-06-04 00:32:15 -0700110 grow without bound. The LRU feature performs best when *maxsize* is a
111 power-of-two.
Raymond Hettingerc79fb0e2010-12-01 03:45:41 +0000112
Serhiy Storchaka4adf01c2016-10-19 18:30:05 +0300113 If *typed* is set to true, function arguments of different types will be
Raymond Hettingercd9fdfd2011-10-20 08:57:45 -0700114 cached separately. For example, ``f(3)`` and ``f(3.0)`` will be treated
115 as distinct calls with distinct results.
116
Manjusaka051ff522019-11-12 15:30:18 +0800117 The wrapped function is instrumented with a :func:`cache_parameters`
118 function that returns a new :class:`dict` showing the values for *maxsize*
119 and *typed*. This is for information purposes only. Mutating the values
120 has no effect.
121
Raymond Hettinger7496b412010-11-30 19:15:45 +0000122 To help measure the effectiveness of the cache and tune the *maxsize*
123 parameter, the wrapped function is instrumented with a :func:`cache_info`
124 function that returns a :term:`named tuple` showing *hits*, *misses*,
Raymond Hettingerc79fb0e2010-12-01 03:45:41 +0000125 *maxsize* and *currsize*. In a multi-threaded environment, the hits
126 and misses are approximate.
Nick Coghlan234515a2010-11-30 06:19:46 +0000127
Raymond Hettinger7496b412010-11-30 19:15:45 +0000128 The decorator also provides a :func:`cache_clear` function for clearing or
129 invalidating the cache.
Georg Brandl2e7346a2010-07-31 18:09:23 +0000130
Raymond Hettinger3fccfcb2010-08-17 19:19:29 +0000131 The original underlying function is accessible through the
Raymond Hettinger7496b412010-11-30 19:15:45 +0000132 :attr:`__wrapped__` attribute. This is useful for introspection, for
133 bypassing the cache, or for rewrapping the function with a different cache.
Nick Coghlan98876832010-08-17 06:17:18 +0000134
Raymond Hettingercc038582010-11-30 20:02:57 +0000135 An `LRU (least recently used) cache
Georg Brandl5d941342016-02-26 19:37:12 +0100136 <https://en.wikipedia.org/wiki/Cache_algorithms#Examples>`_ works
Raymond Hettingercd9fdfd2011-10-20 08:57:45 -0700137 best when the most recent calls are the best predictors of upcoming calls (for
138 example, the most popular articles on a news server tend to change each day).
Raymond Hettinger7496b412010-11-30 19:15:45 +0000139 The cache's size limit assures that the cache does not grow without bound on
140 long-running processes such as web servers.
141
Raymond Hettingerf0e0f202018-11-25 16:24:52 -0800142 In general, the LRU cache should only be used when you want to reuse
143 previously computed values. Accordingly, it doesn't make sense to cache
144 functions with side-effects, functions that need to create distinct mutable
145 objects on each call, or impure functions such as time() or random().
146
Raymond Hettingercc038582010-11-30 20:02:57 +0000147 Example of an LRU cache for static web content::
Raymond Hettinger7496b412010-11-30 19:15:45 +0000148
Raymond Hettinger17328e42013-04-06 20:27:33 -0700149 @lru_cache(maxsize=32)
Raymond Hettinger7496b412010-11-30 19:15:45 +0000150 def get_pep(num):
151 'Retrieve text of a Python Enhancement Proposal'
152 resource = 'http://www.python.org/dev/peps/pep-%04d/' % num
153 try:
154 with urllib.request.urlopen(resource) as s:
155 return s.read()
156 except urllib.error.HTTPError:
157 return 'Not Found'
158
159 >>> for n in 8, 290, 308, 320, 8, 218, 320, 279, 289, 320, 9991:
160 ... pep = get_pep(n)
161 ... print(n, len(pep))
162
Raymond Hettinger17328e42013-04-06 20:27:33 -0700163 >>> get_pep.cache_info()
164 CacheInfo(hits=3, misses=8, maxsize=32, currsize=8)
Georg Brandl2e7346a2010-07-31 18:09:23 +0000165
Raymond Hettingerc79fb0e2010-12-01 03:45:41 +0000166 Example of efficiently computing
Georg Brandl5d941342016-02-26 19:37:12 +0100167 `Fibonacci numbers <https://en.wikipedia.org/wiki/Fibonacci_number>`_
Raymond Hettingerc79fb0e2010-12-01 03:45:41 +0000168 using a cache to implement a
Georg Brandl5d941342016-02-26 19:37:12 +0100169 `dynamic programming <https://en.wikipedia.org/wiki/Dynamic_programming>`_
Raymond Hettingerc79fb0e2010-12-01 03:45:41 +0000170 technique::
171
172 @lru_cache(maxsize=None)
173 def fib(n):
174 if n < 2:
175 return n
176 return fib(n-1) + fib(n-2)
177
Raymond Hettinger17328e42013-04-06 20:27:33 -0700178 >>> [fib(n) for n in range(16)]
Raymond Hettingerc79fb0e2010-12-01 03:45:41 +0000179 [0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610]
180
Raymond Hettinger17328e42013-04-06 20:27:33 -0700181 >>> fib.cache_info()
Raymond Hettingerc79fb0e2010-12-01 03:45:41 +0000182 CacheInfo(hits=28, misses=16, maxsize=None, currsize=16)
183
Georg Brandl2e7346a2010-07-31 18:09:23 +0000184 .. versionadded:: 3.2
185
Raymond Hettingercd9fdfd2011-10-20 08:57:45 -0700186 .. versionchanged:: 3.3
187 Added the *typed* option.
188
Raymond Hettingerb8218682019-05-26 11:27:35 -0700189 .. versionchanged:: 3.8
190 Added the *user_function* option.
191
Manjusaka051ff522019-11-12 15:30:18 +0800192 .. versionadded:: 3.9
193 Added the function :func:`cache_parameters`
194
Georg Brandl8a1caa22010-07-29 16:01:11 +0000195.. decorator:: total_ordering
Raymond Hettingerc50846a2010-04-05 18:56:31 +0000196
197 Given a class defining one or more rich comparison ordering methods, this
Benjamin Peterson08bf91c2010-04-11 16:12:57 +0000198 class decorator supplies the rest. This simplifies the effort involved
Raymond Hettingerc50846a2010-04-05 18:56:31 +0000199 in specifying all of the possible rich comparison operations:
200
201 The class must define one of :meth:`__lt__`, :meth:`__le__`,
202 :meth:`__gt__`, or :meth:`__ge__`.
203 In addition, the class should supply an :meth:`__eq__` method.
204
205 For example::
206
207 @total_ordering
208 class Student:
Nick Coghlanf05d9812013-10-02 00:02:03 +1000209 def _is_valid_operand(self, other):
210 return (hasattr(other, "lastname") and
211 hasattr(other, "firstname"))
Raymond Hettingerc50846a2010-04-05 18:56:31 +0000212 def __eq__(self, other):
Nick Coghlanf05d9812013-10-02 00:02:03 +1000213 if not self._is_valid_operand(other):
214 return NotImplemented
Raymond Hettingerc50846a2010-04-05 18:56:31 +0000215 return ((self.lastname.lower(), self.firstname.lower()) ==
216 (other.lastname.lower(), other.firstname.lower()))
217 def __lt__(self, other):
Nick Coghlanf05d9812013-10-02 00:02:03 +1000218 if not self._is_valid_operand(other):
219 return NotImplemented
Raymond Hettingerc50846a2010-04-05 18:56:31 +0000220 return ((self.lastname.lower(), self.firstname.lower()) <
221 (other.lastname.lower(), other.firstname.lower()))
222
Nick Coghlanf05d9812013-10-02 00:02:03 +1000223 .. note::
224
225 While this decorator makes it easy to create well behaved totally
226 ordered types, it *does* come at the cost of slower execution and
227 more complex stack traces for the derived comparison methods. If
228 performance benchmarking indicates this is a bottleneck for a given
229 application, implementing all six rich comparison methods instead is
230 likely to provide an easy speed boost.
231
Raymond Hettingerc50846a2010-04-05 18:56:31 +0000232 .. versionadded:: 3.2
233
Nick Coghlanf05d9812013-10-02 00:02:03 +1000234 .. versionchanged:: 3.4
235 Returning NotImplemented from the underlying comparison function for
236 unrecognised types is now supported.
Georg Brandl67b21b72010-08-17 15:07:14 +0000237
Serhiy Storchaka2085bd02019-06-01 11:00:15 +0300238.. function:: partial(func, /, *args, **keywords)
Georg Brandl116aa622007-08-15 14:28:22 +0000239
Andrei Petre83a07652018-10-22 23:11:20 -0700240 Return a new :ref:`partial object<partial-objects>` which when called
241 will behave like *func* called with the positional arguments *args*
242 and keyword arguments *keywords*. If more arguments are supplied to the
243 call, they are appended to *args*. If additional keyword arguments are
244 supplied, they extend and override *keywords*.
Georg Brandl116aa622007-08-15 14:28:22 +0000245 Roughly equivalent to::
246
Serhiy Storchaka2085bd02019-06-01 11:00:15 +0300247 def partial(func, /, *args, **keywords):
Georg Brandl116aa622007-08-15 14:28:22 +0000248 def newfunc(*fargs, **fkeywords):
Sergey Fedoseevb981fec2018-10-20 02:42:07 +0500249 newkeywords = {**keywords, **fkeywords}
Martin Panter0c0da482016-06-12 01:46:50 +0000250 return func(*args, *fargs, **newkeywords)
Georg Brandl116aa622007-08-15 14:28:22 +0000251 newfunc.func = func
252 newfunc.args = args
253 newfunc.keywords = keywords
254 return newfunc
255
256 The :func:`partial` is used for partial function application which "freezes"
257 some portion of a function's arguments and/or keywords resulting in a new object
258 with a simplified signature. For example, :func:`partial` can be used to create
259 a callable that behaves like the :func:`int` function where the *base* argument
Christian Heimesfe337bf2008-03-23 21:54:12 +0000260 defaults to two:
Georg Brandl116aa622007-08-15 14:28:22 +0000261
Christian Heimesfe337bf2008-03-23 21:54:12 +0000262 >>> from functools import partial
Georg Brandl116aa622007-08-15 14:28:22 +0000263 >>> basetwo = partial(int, base=2)
264 >>> basetwo.__doc__ = 'Convert base 2 string to an int.'
265 >>> basetwo('10010')
266 18
267
268
Serhiy Storchaka70c5f2a2019-06-01 11:38:24 +0300269.. class:: partialmethod(func, /, *args, **keywords)
Nick Coghlanf4cb48a2013-11-03 16:41:46 +1000270
271 Return a new :class:`partialmethod` descriptor which behaves
272 like :class:`partial` except that it is designed to be used as a method
273 definition rather than being directly callable.
274
275 *func* must be a :term:`descriptor` or a callable (objects which are both,
276 like normal functions, are handled as descriptors).
277
278 When *func* is a descriptor (such as a normal Python function,
279 :func:`classmethod`, :func:`staticmethod`, :func:`abstractmethod` or
280 another instance of :class:`partialmethod`), calls to ``__get__`` are
281 delegated to the underlying descriptor, and an appropriate
Andrei Petre83a07652018-10-22 23:11:20 -0700282 :ref:`partial object<partial-objects>` returned as the result.
Nick Coghlanf4cb48a2013-11-03 16:41:46 +1000283
284 When *func* is a non-descriptor callable, an appropriate bound method is
285 created dynamically. This behaves like a normal Python function when
286 used as a method: the *self* argument will be inserted as the first
287 positional argument, even before the *args* and *keywords* supplied to
288 the :class:`partialmethod` constructor.
289
290 Example::
291
Serhiy Storchakae042a452019-06-10 13:35:52 +0300292 >>> class Cell:
Benjamin Peterson3a434032014-03-30 15:07:09 -0400293 ... def __init__(self):
294 ... self._alive = False
Nick Coghlanf4cb48a2013-11-03 16:41:46 +1000295 ... @property
296 ... def alive(self):
297 ... return self._alive
298 ... def set_state(self, state):
299 ... self._alive = bool(state)
Nick Coghlan3daaf5f2013-11-04 23:32:16 +1000300 ... set_alive = partialmethod(set_state, True)
301 ... set_dead = partialmethod(set_state, False)
Nick Coghlanf4cb48a2013-11-03 16:41:46 +1000302 ...
303 >>> c = Cell()
304 >>> c.alive
305 False
306 >>> c.set_alive()
307 >>> c.alive
308 True
309
310 .. versionadded:: 3.4
311
312
Georg Brandl58f9e4f2008-04-19 22:18:33 +0000313.. function:: reduce(function, iterable[, initializer])
Georg Brandl116aa622007-08-15 14:28:22 +0000314
Brendan Jurd9df10022018-10-01 16:52:10 +1000315 Apply *function* of two arguments cumulatively to the items of *iterable*, from
316 left to right, so as to reduce the iterable to a single value. For example,
Georg Brandl116aa622007-08-15 14:28:22 +0000317 ``reduce(lambda x, y: x+y, [1, 2, 3, 4, 5])`` calculates ``((((1+2)+3)+4)+5)``.
318 The left argument, *x*, is the accumulated value and the right argument, *y*, is
Brendan Jurd9df10022018-10-01 16:52:10 +1000319 the update value from the *iterable*. If the optional *initializer* is present,
320 it is placed before the items of the iterable in the calculation, and serves as
321 a default when the iterable is empty. If *initializer* is not given and
322 *iterable* contains only one item, the first item is returned.
Georg Brandl116aa622007-08-15 14:28:22 +0000323
Raymond Hettinger558dcf32014-12-16 18:16:57 -0800324 Roughly equivalent to::
Raymond Hettinger64801682013-10-12 16:04:17 -0700325
326 def reduce(function, iterable, initializer=None):
327 it = iter(iterable)
328 if initializer is None:
329 value = next(it)
330 else:
331 value = initializer
332 for element in it:
333 value = function(value, element)
334 return value
335
Gerrit Hollbd81cbd2018-07-04 23:26:32 +0100336 See :func:`itertools.accumulate` for an iterator that yields all intermediate
337 values.
Georg Brandl116aa622007-08-15 14:28:22 +0000338
Daisuke Miyakawa0e61e672017-10-12 23:39:43 +0900339.. decorator:: singledispatch
Łukasz Langa6f692512013-06-05 12:20:24 +0200340
Daisuke Miyakawa0e61e672017-10-12 23:39:43 +0900341 Transform a function into a :term:`single-dispatch <single
Łukasz Langafdcf2b72013-06-07 22:54:03 +0200342 dispatch>` :term:`generic function`.
Łukasz Langa6f692512013-06-05 12:20:24 +0200343
344 To define a generic function, decorate it with the ``@singledispatch``
345 decorator. Note that the dispatch happens on the type of the first argument,
346 create your function accordingly::
347
348 >>> from functools import singledispatch
349 >>> @singledispatch
350 ... def fun(arg, verbose=False):
351 ... if verbose:
352 ... print("Let me just say,", end=" ")
353 ... print(arg)
354
355 To add overloaded implementations to the function, use the :func:`register`
Łukasz Langae5697532017-12-11 13:56:31 -0800356 attribute of the generic function. It is a decorator. For functions
357 annotated with types, the decorator will infer the type of the first
358 argument automatically::
Łukasz Langa6f692512013-06-05 12:20:24 +0200359
Łukasz Langae5697532017-12-11 13:56:31 -0800360 >>> @fun.register
361 ... def _(arg: int, verbose=False):
Łukasz Langa6f692512013-06-05 12:20:24 +0200362 ... if verbose:
363 ... print("Strength in numbers, eh?", end=" ")
364 ... print(arg)
365 ...
Łukasz Langae5697532017-12-11 13:56:31 -0800366 >>> @fun.register
367 ... def _(arg: list, verbose=False):
Łukasz Langa6f692512013-06-05 12:20:24 +0200368 ... if verbose:
369 ... print("Enumerate this:")
370 ... for i, elem in enumerate(arg):
371 ... print(i, elem)
372
Łukasz Langae5697532017-12-11 13:56:31 -0800373 For code which doesn't use type annotations, the appropriate type
374 argument can be passed explicitly to the decorator itself::
375
376 >>> @fun.register(complex)
377 ... def _(arg, verbose=False):
378 ... if verbose:
379 ... print("Better than complicated.", end=" ")
380 ... print(arg.real, arg.imag)
381 ...
382
383
Łukasz Langa6f692512013-06-05 12:20:24 +0200384 To enable registering lambdas and pre-existing functions, the
385 :func:`register` attribute can be used in a functional form::
386
387 >>> def nothing(arg, verbose=False):
388 ... print("Nothing.")
389 ...
390 >>> fun.register(type(None), nothing)
391
392 The :func:`register` attribute returns the undecorated function which
393 enables decorator stacking, pickling, as well as creating unit tests for
394 each variant independently::
395
396 >>> @fun.register(float)
397 ... @fun.register(Decimal)
398 ... def fun_num(arg, verbose=False):
399 ... if verbose:
400 ... print("Half of your number:", end=" ")
401 ... print(arg / 2)
402 ...
403 >>> fun_num is fun
404 False
405
406 When called, the generic function dispatches on the type of the first
407 argument::
408
409 >>> fun("Hello, world.")
410 Hello, world.
411 >>> fun("test.", verbose=True)
412 Let me just say, test.
413 >>> fun(42, verbose=True)
414 Strength in numbers, eh? 42
415 >>> fun(['spam', 'spam', 'eggs', 'spam'], verbose=True)
416 Enumerate this:
417 0 spam
418 1 spam
419 2 eggs
420 3 spam
421 >>> fun(None)
422 Nothing.
423 >>> fun(1.23)
424 0.615
425
426 Where there is no registered implementation for a specific type, its
427 method resolution order is used to find a more generic implementation.
428 The original function decorated with ``@singledispatch`` is registered
429 for the base ``object`` type, which means it is used if no better
430 implementation is found.
431
Batuhan Taşkaya24555ce2019-11-19 11:16:46 +0300432 If an implementation registered to :term:`abstract base class`, virtual
433 subclasses will be dispatched to that implementation::
434
435 >>> from collections.abc import Mapping
436 >>> @fun.register
437 ... def _(arg: Mapping, verbose=False):
438 ... if verbose:
439 ... print("Keys & Values")
440 ... for key, value in arg.items():
441 ... print(key, "=>", value)
442 ...
443 >>> fun({"a": "b"})
444 a => b
445
Łukasz Langa6f692512013-06-05 12:20:24 +0200446 To check which implementation will the generic function choose for
447 a given type, use the ``dispatch()`` attribute::
448
449 >>> fun.dispatch(float)
450 <function fun_num at 0x1035a2840>
451 >>> fun.dispatch(dict) # note: default implementation
452 <function fun at 0x103fe0000>
453
454 To access all registered implementations, use the read-only ``registry``
455 attribute::
456
457 >>> fun.registry.keys()
458 dict_keys([<class 'NoneType'>, <class 'int'>, <class 'object'>,
459 <class 'decimal.Decimal'>, <class 'list'>,
460 <class 'float'>])
461 >>> fun.registry[float]
462 <function fun_num at 0x1035a2840>
463 >>> fun.registry[object]
464 <function fun at 0x103fe0000>
465
466 .. versionadded:: 3.4
467
Łukasz Langae5697532017-12-11 13:56:31 -0800468 .. versionchanged:: 3.7
469 The :func:`register` attribute supports using type annotations.
470
Łukasz Langa6f692512013-06-05 12:20:24 +0200471
Ethan Smithc6512752018-05-26 16:38:33 -0400472.. class:: singledispatchmethod(func)
473
474 Transform a method into a :term:`single-dispatch <single
475 dispatch>` :term:`generic function`.
476
477 To define a generic method, decorate it with the ``@singledispatchmethod``
478 decorator. Note that the dispatch happens on the type of the first non-self
479 or non-cls argument, create your function accordingly::
480
481 class Negator:
482 @singledispatchmethod
483 def neg(self, arg):
484 raise NotImplementedError("Cannot negate a")
485
486 @neg.register
487 def _(self, arg: int):
488 return -arg
489
490 @neg.register
491 def _(self, arg: bool):
492 return not arg
493
494 ``@singledispatchmethod`` supports nesting with other decorators such as
495 ``@classmethod``. Note that to allow for ``dispatcher.register``,
496 ``singledispatchmethod`` must be the *outer most* decorator. Here is the
497 ``Negator`` class with the ``neg`` methods being class bound::
498
499 class Negator:
500 @singledispatchmethod
501 @classmethod
502 def neg(cls, arg):
503 raise NotImplementedError("Cannot negate a")
504
505 @neg.register
506 @classmethod
507 def _(cls, arg: int):
508 return -arg
509
510 @neg.register
511 @classmethod
512 def _(cls, arg: bool):
513 return not arg
514
515 The same pattern can be used for other similar decorators: ``staticmethod``,
516 ``abstractmethod``, and others.
517
Inada Naokibc284f02019-03-27 18:15:17 +0900518 .. versionadded:: 3.8
519
520
Pablo Galindo99e6c262020-01-23 15:29:52 +0000521.. class:: TopologicalSorter(graph=None)
522
523 Provides functionality to topologically sort a graph of hashable nodes.
524
Pablo Galindo65ecc392020-01-23 21:01:50 +0000525 A topological order is a linear ordering of the vertices in a graph such that
526 for every directed edge u -> v from vertex u to vertex v, vertex u comes
527 before vertex v in the ordering. For instance, the vertices of the graph may
528 represent tasks to be performed, and the edges may represent constraints that
529 one task must be performed before another; in this example, a topological
530 ordering is just a valid sequence for the tasks. A complete topological
531 ordering is possible if and only if the graph has no directed cycles, that
532 is, if it is a directed acyclic graph.
Pablo Galindo99e6c262020-01-23 15:29:52 +0000533
Pablo Galindo65ecc392020-01-23 21:01:50 +0000534 If the optional *graph* argument is provided it must be a dictionary
535 representing a directed acyclic graph where the keys are nodes and the values
536 are iterables of all predecessors of that node in the graph (the nodes that
537 have edges that point to the value in the key). Additional nodes can be added
538 to the graph using the :meth:`~TopologicalSorter.add` method.
Pablo Galindo99e6c262020-01-23 15:29:52 +0000539
Pablo Galindo65ecc392020-01-23 21:01:50 +0000540 In the general case, the steps required to perform the sorting of a given
541 graph are as follows:
Pablo Galindo99e6c262020-01-23 15:29:52 +0000542
Pablo Galindo65ecc392020-01-23 21:01:50 +0000543 * Create an instance of the :class:`TopologicalSorter` with an optional
544 initial graph.
Pablo Galindo99e6c262020-01-23 15:29:52 +0000545 * Add additional nodes to the graph.
546 * Call :meth:`~TopologicalSorter.prepare` on the graph.
Pablo Galindo65ecc392020-01-23 21:01:50 +0000547 * While :meth:`~TopologicalSorter.is_active` is ``True``, iterate over
548 the nodes returned by :meth:`~TopologicalSorter.get_ready` and
549 process them. Call :meth:`~TopologicalSorter.done` on each node as it
550 finishes processing.
Pablo Galindo99e6c262020-01-23 15:29:52 +0000551
552 In case just an immediate sorting of the nodes in the graph is required and
Pablo Galindo65ecc392020-01-23 21:01:50 +0000553 no parallelism is involved, the convenience method
554 :meth:`TopologicalSorter.static_order` can be used directly:
Pablo Galindo99e6c262020-01-23 15:29:52 +0000555
556 .. doctest::
557
Pablo Galindo65ecc392020-01-23 21:01:50 +0000558 >>> graph = {"D": {"B", "C"}, "C": {"A"}, "B": {"A"}}
Pablo Galindo99e6c262020-01-23 15:29:52 +0000559 >>> ts = TopologicalSorter(graph)
Pablo Galindo65ecc392020-01-23 21:01:50 +0000560 >>> tuple(ts.static_order())
561 ('A', 'C', 'B', 'D')
Pablo Galindo99e6c262020-01-23 15:29:52 +0000562
Pablo Galindo65ecc392020-01-23 21:01:50 +0000563 The class is designed to easily support parallel processing of the nodes as
564 they become ready. For instance::
Pablo Galindo99e6c262020-01-23 15:29:52 +0000565
566 topological_sorter = TopologicalSorter()
567
568 # Add nodes to 'topological_sorter'...
569
570 topological_sorter.prepare()
571 while topological_sorter.is_active():
572 for node in topological_sorter.get_ready():
573 # Worker threads or processes take nodes to work on off the
574 # 'task_queue' queue.
575 task_queue.put(node)
576
577 # When the work for a node is done, workers put the node in
578 # 'finalized_tasks_queue' so we can get more nodes to work on.
579 # The definition of 'is_active()' guarantees that, at this point, at
580 # least one node has been placed on 'task_queue' that hasn't yet
581 # been passed to 'done()', so this blocking 'get()' must (eventually)
582 # succeed. After calling 'done()', we loop back to call 'get_ready()'
583 # again, so put newly freed nodes on 'task_queue' as soon as
584 # logically possible.
585 node = finalized_tasks_queue.get()
586 topological_sorter.done(node)
587
588 .. method:: add(node, *predecessors)
589
Pablo Galindo65ecc392020-01-23 21:01:50 +0000590 Add a new node and its predecessors to the graph. Both the *node* and all
591 elements in *predecessors* must be hashable.
Pablo Galindo99e6c262020-01-23 15:29:52 +0000592
Pablo Galindo65ecc392020-01-23 21:01:50 +0000593 If called multiple times with the same node argument, the set of
594 dependencies will be the union of all dependencies passed in.
Pablo Galindo99e6c262020-01-23 15:29:52 +0000595
596 It is possible to add a node with no dependencies (*predecessors* is not
597 provided) or to provide a dependency twice. If a node that has not been
Pablo Galindo65ecc392020-01-23 21:01:50 +0000598 provided before is included among *predecessors* it will be automatically
599 added to the graph with no predecessors of its own.
Pablo Galindo99e6c262020-01-23 15:29:52 +0000600
601 Raises :exc:`ValueError` if called after :meth:`~TopologicalSorter.prepare`.
602
603 .. method:: prepare()
604
Pablo Galindo65ecc392020-01-23 21:01:50 +0000605 Mark the graph as finished and check for cycles in the graph. If any cycle
606 is detected, :exc:`CycleError` will be raised, but
607 :meth:`~TopologicalSorter.get_ready` can still be used to obtain as many
608 nodes as possible until cycles block more progress. After a call to this
609 function, the graph cannot be modified, and therefore no more nodes can be
610 added using :meth:`~TopologicalSorter.add`.
Pablo Galindo99e6c262020-01-23 15:29:52 +0000611
612 .. method:: is_active()
613
Pablo Galindo65ecc392020-01-23 21:01:50 +0000614 Returns ``True`` if more progress can be made and ``False`` otherwise.
615 Progress can be made if cycles do not block the resolution and either
616 there are still nodes ready that haven't yet been returned by
Pablo Galindo99e6c262020-01-23 15:29:52 +0000617 :meth:`TopologicalSorter.get_ready` or the number of nodes marked
Pablo Galindo65ecc392020-01-23 21:01:50 +0000618 :meth:`TopologicalSorter.done` is less than the number that have been
619 returned by :meth:`TopologicalSorter.get_ready`.
Pablo Galindo99e6c262020-01-23 15:29:52 +0000620
Pablo Galindo65ecc392020-01-23 21:01:50 +0000621 The :meth:`~TopologicalSorter.__bool__` method of this class defers to
622 this function, so instead of::
Pablo Galindo99e6c262020-01-23 15:29:52 +0000623
624 if ts.is_active():
625 ...
626
627 if possible to simply do::
628
629 if ts:
630 ...
631
Pablo Galindo65ecc392020-01-23 21:01:50 +0000632 Raises :exc:`ValueError` if called without calling
633 :meth:`~TopologicalSorter.prepare` previously.
Pablo Galindo99e6c262020-01-23 15:29:52 +0000634
635 .. method:: done(*nodes)
636
637 Marks a set of nodes returned by :meth:`TopologicalSorter.get_ready` as
Pablo Galindo65ecc392020-01-23 21:01:50 +0000638 processed, unblocking any successor of each node in *nodes* for being
639 returned in the future by a call to :meth:`TopologicalSorter.get_ready`.
Pablo Galindo99e6c262020-01-23 15:29:52 +0000640
641 Raises :exc:`ValueError` if any node in *nodes* has already been marked as
Pablo Galindo65ecc392020-01-23 21:01:50 +0000642 processed by a previous call to this method or if a node was not added to
643 the graph by using :meth:`TopologicalSorter.add`, if called without
644 calling :meth:`~TopologicalSorter.prepare` or if node has not yet been
645 returned by :meth:`~TopologicalSorter.get_ready`.
Pablo Galindo99e6c262020-01-23 15:29:52 +0000646
647 .. method:: get_ready()
648
Pablo Galindo65ecc392020-01-23 21:01:50 +0000649 Returns a ``tuple`` with all the nodes that are ready. Initially it
650 returns all nodes with no predecessors, and once those are marked as
651 processed by calling :meth:`TopologicalSorter.done`, further calls will
652 return all new nodes that have all their predecessors already processed.
653 Once no more progress can be made, empty tuples are returned.
Pablo Galindo99e6c262020-01-23 15:29:52 +0000654
655 Raises :exc:`ValueError` if called without calling
656 :meth:`~TopologicalSorter.prepare` previously.
657
658 .. method:: static_order()
659
660 Returns an iterable of nodes in a topological order. Using this method
661 does not require to call :meth:`TopologicalSorter.prepare` or
662 :meth:`TopologicalSorter.done`. This method is equivalent to::
663
664 def static_order(self):
665 self.prepare()
666 while self.is_active():
667 node_group = self.get_ready()
668 yield from node_group
669 self.done(*node_group)
670
671 The particular order that is returned may depend on the specific order in
672 which the items were inserted in the graph. For example:
673
674 .. doctest::
675
676 >>> ts = TopologicalSorter()
677 >>> ts.add(3, 2, 1)
678 >>> ts.add(1, 0)
679 >>> print([*ts.static_order()])
680 [2, 0, 1, 3]
681
682 >>> ts2 = TopologicalSorter()
683 >>> ts2.add(1, 0)
684 >>> ts2.add(3, 2, 1)
685 >>> print([*ts2.static_order()])
686 [0, 2, 1, 3]
687
Pablo Galindo65ecc392020-01-23 21:01:50 +0000688 This is due to the fact that "0" and "2" are in the same level in the
689 graph (they would have been returned in the same call to
690 :meth:`~TopologicalSorter.get_ready`) and the order between them is
691 determined by the order of insertion.
Pablo Galindo99e6c262020-01-23 15:29:52 +0000692
693
694 If any cycle is detected, :exc:`CycleError` will be raised.
695
696 .. versionadded:: 3.9
697
698
Georg Brandl036490d2009-05-17 13:00:36 +0000699.. function:: update_wrapper(wrapper, wrapped, assigned=WRAPPER_ASSIGNMENTS, updated=WRAPPER_UPDATES)
Georg Brandl116aa622007-08-15 14:28:22 +0000700
701 Update a *wrapper* function to look like the *wrapped* function. The optional
702 arguments are tuples to specify which attributes of the original function are
703 assigned directly to the matching attributes on the wrapper function and which
704 attributes of the wrapper function are updated with the corresponding attributes
705 from the original function. The default values for these arguments are the
Berker Peksag472233e2016-04-18 21:20:50 +0300706 module level constants ``WRAPPER_ASSIGNMENTS`` (which assigns to the wrapper
707 function's ``__module__``, ``__name__``, ``__qualname__``, ``__annotations__``
708 and ``__doc__``, the documentation string) and ``WRAPPER_UPDATES`` (which
709 updates the wrapper function's ``__dict__``, i.e. the instance dictionary).
Georg Brandl116aa622007-08-15 14:28:22 +0000710
Nick Coghlan98876832010-08-17 06:17:18 +0000711 To allow access to the original function for introspection and other purposes
712 (e.g. bypassing a caching decorator such as :func:`lru_cache`), this function
Nick Coghlan24c05bc2013-07-15 21:13:08 +1000713 automatically adds a ``__wrapped__`` attribute to the wrapper that refers to
714 the function being wrapped.
Nick Coghlan98876832010-08-17 06:17:18 +0000715
Christian Heimesd8654cf2007-12-02 15:22:16 +0000716 The main intended use for this function is in :term:`decorator` functions which
717 wrap the decorated function and return the wrapper. If the wrapper function is
718 not updated, the metadata of the returned function will reflect the wrapper
Georg Brandl116aa622007-08-15 14:28:22 +0000719 definition rather than the original function definition, which is typically less
720 than helpful.
721
Nick Coghlan98876832010-08-17 06:17:18 +0000722 :func:`update_wrapper` may be used with callables other than functions. Any
723 attributes named in *assigned* or *updated* that are missing from the object
724 being wrapped are ignored (i.e. this function will not attempt to set them
725 on the wrapper function). :exc:`AttributeError` is still raised if the
726 wrapper function itself is missing any attributes named in *updated*.
727
728 .. versionadded:: 3.2
Georg Brandl9e257012010-08-17 14:11:59 +0000729 Automatic addition of the ``__wrapped__`` attribute.
Nick Coghlan98876832010-08-17 06:17:18 +0000730
731 .. versionadded:: 3.2
Georg Brandl9e257012010-08-17 14:11:59 +0000732 Copying of the ``__annotations__`` attribute by default.
Nick Coghlan98876832010-08-17 06:17:18 +0000733
734 .. versionchanged:: 3.2
Georg Brandl9e257012010-08-17 14:11:59 +0000735 Missing attributes no longer trigger an :exc:`AttributeError`.
736
Nick Coghlan24c05bc2013-07-15 21:13:08 +1000737 .. versionchanged:: 3.4
738 The ``__wrapped__`` attribute now always refers to the wrapped
739 function, even if that function defined a ``__wrapped__`` attribute.
740 (see :issue:`17482`)
741
Georg Brandl116aa622007-08-15 14:28:22 +0000742
Georg Brandl8a1caa22010-07-29 16:01:11 +0000743.. decorator:: wraps(wrapped, assigned=WRAPPER_ASSIGNMENTS, updated=WRAPPER_UPDATES)
Georg Brandl116aa622007-08-15 14:28:22 +0000744
Ezio Melotti67f6d5f2014-08-05 08:14:28 +0300745 This is a convenience function for invoking :func:`update_wrapper` as a
746 function decorator when defining a wrapper function. It is equivalent to
747 ``partial(update_wrapper, wrapped=wrapped, assigned=assigned, updated=updated)``.
748 For example::
Georg Brandl116aa622007-08-15 14:28:22 +0000749
Christian Heimesfe337bf2008-03-23 21:54:12 +0000750 >>> from functools import wraps
Georg Brandl116aa622007-08-15 14:28:22 +0000751 >>> def my_decorator(f):
752 ... @wraps(f)
753 ... def wrapper(*args, **kwds):
Georg Brandl6911e3c2007-09-04 07:15:32 +0000754 ... print('Calling decorated function')
Georg Brandl116aa622007-08-15 14:28:22 +0000755 ... return f(*args, **kwds)
756 ... return wrapper
757 ...
758 >>> @my_decorator
759 ... def example():
760 ... """Docstring"""
Georg Brandl6911e3c2007-09-04 07:15:32 +0000761 ... print('Called example function')
Georg Brandl116aa622007-08-15 14:28:22 +0000762 ...
763 >>> example()
764 Calling decorated function
765 Called example function
766 >>> example.__name__
767 'example'
768 >>> example.__doc__
769 'Docstring'
770
771 Without the use of this decorator factory, the name of the example function
772 would have been ``'wrapper'``, and the docstring of the original :func:`example`
773 would have been lost.
774
775
776.. _partial-objects:
777
778:class:`partial` Objects
779------------------------
780
781:class:`partial` objects are callable objects created by :func:`partial`. They
782have three read-only attributes:
783
784
785.. attribute:: partial.func
786
787 A callable object or function. Calls to the :class:`partial` object will be
788 forwarded to :attr:`func` with new arguments and keywords.
789
790
791.. attribute:: partial.args
792
793 The leftmost positional arguments that will be prepended to the positional
794 arguments provided to a :class:`partial` object call.
795
796
797.. attribute:: partial.keywords
798
799 The keyword arguments that will be supplied when the :class:`partial` object is
800 called.
801
802:class:`partial` objects are like :class:`function` objects in that they are
803callable, weak referencable, and can have attributes. There are some important
Martin Panterbae5d812016-06-18 03:57:31 +0000804differences. For instance, the :attr:`~definition.__name__` and :attr:`__doc__` attributes
Georg Brandl116aa622007-08-15 14:28:22 +0000805are not created automatically. Also, :class:`partial` objects defined in
806classes behave like static methods and do not transform into bound methods
807during instance attribute look-up.
Pablo Galindo99e6c262020-01-23 15:29:52 +0000808
809
810Exceptions
811----------
812The :mod:`functools` module defines the following exception classes:
813
814.. exception:: CycleError
815
816 Subclass of :exc:`ValueError` raised by :meth:`TopologicalSorter.prepare` if cycles exist
817 in the working graph. If multiple cycles exist, only one undefined choice among them will
818 be reported and included in the exception.
819
820 The detected cycle can be accessed via the second element in the :attr:`~CycleError.args`
821 attribute of the exception instance and consists in a list of nodes, such that each node is,
822 in the graph, an immediate predecessor of the next node in the list. In the reported list,
823 the first and the last node will be the same, to make it clear that it is cyclic.