Georg Brandl | 6c89a79 | 2012-01-25 22:36:25 +0100 | [diff] [blame] | 1 | :mod:`functools` --- Higher-order functions and operations on callable objects |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 2 | ============================================================================== |
| 3 | |
| 4 | .. module:: functools |
Georg Brandl | 6c89a79 | 2012-01-25 22:36:25 +0100 | [diff] [blame] | 5 | :synopsis: Higher-order functions and operations on callable objects. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 6 | .. moduleauthor:: Peter Harris <scav@blueyonder.co.uk> |
| 7 | .. moduleauthor:: Raymond Hettinger <python@rcn.com> |
| 8 | .. moduleauthor:: Nick Coghlan <ncoghlan@gmail.com> |
| 9 | .. sectionauthor:: Peter Harris <scav@blueyonder.co.uk> |
| 10 | |
Raymond Hettinger | 05ce079 | 2011-01-10 21:16:07 +0000 | [diff] [blame] | 11 | **Source code:** :source:`Lib/functools.py` |
| 12 | |
| 13 | -------------- |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 14 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 15 | The :mod:`functools` module is for higher-order functions: functions that act on |
| 16 | or return other functions. In general, any callable object can be treated as a |
| 17 | function for the purposes of this module. |
| 18 | |
Thomas Wouters | ed03b41 | 2007-08-28 21:37:11 +0000 | [diff] [blame] | 19 | The :mod:`functools` module defines the following functions: |
| 20 | |
Éric Araujo | b10089e | 2010-11-18 14:22:08 +0000 | [diff] [blame] | 21 | .. function:: cmp_to_key(func) |
Raymond Hettinger | c50846a | 2010-04-05 18:56:31 +0000 | [diff] [blame] | 22 | |
Georg Brandl | 3b65fd7 | 2012-01-23 20:19:33 +0100 | [diff] [blame] | 23 | Transform an old-style comparison function to a key function. Used with |
Benjamin Peterson | cca6531 | 2010-08-09 02:13:10 +0000 | [diff] [blame] | 24 | tools that accept key functions (such as :func:`sorted`, :func:`min`, |
| 25 | :func:`max`, :func:`heapq.nlargest`, :func:`heapq.nsmallest`, |
| 26 | :func:`itertools.groupby`). This function is primarily used as a transition |
Ezio Melotti | 9ecb6be | 2012-01-16 08:28:54 +0200 | [diff] [blame] | 27 | tool for programs being converted from Python 2 which supported the use of |
Benjamin Peterson | cca6531 | 2010-08-09 02:13:10 +0000 | [diff] [blame] | 28 | comparison functions. |
Raymond Hettinger | c50846a | 2010-04-05 18:56:31 +0000 | [diff] [blame] | 29 | |
Georg Brandl | 6c89a79 | 2012-01-25 22:36:25 +0100 | [diff] [blame] | 30 | A comparison function is any callable that accept two arguments, compares them, |
Benjamin Peterson | cca6531 | 2010-08-09 02:13:10 +0000 | [diff] [blame] | 31 | and returns a negative number for less-than, zero for equality, or a positive |
| 32 | number for greater-than. A key function is a callable that accepts one |
Raymond Hettinger | c79fb0e | 2010-12-01 03:45:41 +0000 | [diff] [blame] | 33 | argument and returns another value indicating the position in the desired |
Benjamin Peterson | cca6531 | 2010-08-09 02:13:10 +0000 | [diff] [blame] | 34 | collation sequence. |
Raymond Hettinger | c50846a | 2010-04-05 18:56:31 +0000 | [diff] [blame] | 35 | |
Benjamin Peterson | cca6531 | 2010-08-09 02:13:10 +0000 | [diff] [blame] | 36 | Example:: |
Raymond Hettinger | c50846a | 2010-04-05 18:56:31 +0000 | [diff] [blame] | 37 | |
Benjamin Peterson | cca6531 | 2010-08-09 02:13:10 +0000 | [diff] [blame] | 38 | sorted(iterable, key=cmp_to_key(locale.strcoll)) # locale-aware sort order |
Raymond Hettinger | c50846a | 2010-04-05 18:56:31 +0000 | [diff] [blame] | 39 | |
| 40 | .. versionadded:: 3.2 |
| 41 | |
Georg Brandl | 67b21b7 | 2010-08-17 15:07:14 +0000 | [diff] [blame] | 42 | |
Raymond Hettinger | cd9fdfd | 2011-10-20 08:57:45 -0700 | [diff] [blame] | 43 | .. decorator:: lru_cache(maxsize=100, typed=False) |
Georg Brandl | 2e7346a | 2010-07-31 18:09:23 +0000 | [diff] [blame] | 44 | |
| 45 | Decorator to wrap a function with a memoizing callable that saves up to the |
| 46 | *maxsize* most recent calls. It can save time when an expensive or I/O bound |
| 47 | function is periodically called with the same arguments. |
| 48 | |
Raymond Hettinger | 7496b41 | 2010-11-30 19:15:45 +0000 | [diff] [blame] | 49 | Since a dictionary is used to cache results, the positional and keyword |
| 50 | arguments to the function must be hashable. |
Georg Brandl | 2e7346a | 2010-07-31 18:09:23 +0000 | [diff] [blame] | 51 | |
Raymond Hettinger | c79fb0e | 2010-12-01 03:45:41 +0000 | [diff] [blame] | 52 | If *maxsize* is set to None, the LRU feature is disabled and the cache |
| 53 | can grow without bound. |
| 54 | |
Raymond Hettinger | cd9fdfd | 2011-10-20 08:57:45 -0700 | [diff] [blame] | 55 | If *typed* is set to True, function arguments of different types will be |
| 56 | cached separately. For example, ``f(3)`` and ``f(3.0)`` will be treated |
| 57 | as distinct calls with distinct results. |
| 58 | |
Raymond Hettinger | 7496b41 | 2010-11-30 19:15:45 +0000 | [diff] [blame] | 59 | To help measure the effectiveness of the cache and tune the *maxsize* |
| 60 | parameter, the wrapped function is instrumented with a :func:`cache_info` |
| 61 | function that returns a :term:`named tuple` showing *hits*, *misses*, |
Raymond Hettinger | c79fb0e | 2010-12-01 03:45:41 +0000 | [diff] [blame] | 62 | *maxsize* and *currsize*. In a multi-threaded environment, the hits |
| 63 | and misses are approximate. |
Nick Coghlan | 234515a | 2010-11-30 06:19:46 +0000 | [diff] [blame] | 64 | |
Raymond Hettinger | 7496b41 | 2010-11-30 19:15:45 +0000 | [diff] [blame] | 65 | The decorator also provides a :func:`cache_clear` function for clearing or |
| 66 | invalidating the cache. |
Georg Brandl | 2e7346a | 2010-07-31 18:09:23 +0000 | [diff] [blame] | 67 | |
Raymond Hettinger | 3fccfcb | 2010-08-17 19:19:29 +0000 | [diff] [blame] | 68 | The original underlying function is accessible through the |
Raymond Hettinger | 7496b41 | 2010-11-30 19:15:45 +0000 | [diff] [blame] | 69 | :attr:`__wrapped__` attribute. This is useful for introspection, for |
| 70 | bypassing the cache, or for rewrapping the function with a different cache. |
Nick Coghlan | 9887683 | 2010-08-17 06:17:18 +0000 | [diff] [blame] | 71 | |
Raymond Hettinger | cc03858 | 2010-11-30 20:02:57 +0000 | [diff] [blame] | 72 | An `LRU (least recently used) cache |
Raymond Hettinger | 7496b41 | 2010-11-30 19:15:45 +0000 | [diff] [blame] | 73 | <http://en.wikipedia.org/wiki/Cache_algorithms#Least_Recently_Used>`_ works |
Raymond Hettinger | cd9fdfd | 2011-10-20 08:57:45 -0700 | [diff] [blame] | 74 | best when the most recent calls are the best predictors of upcoming calls (for |
| 75 | example, the most popular articles on a news server tend to change each day). |
Raymond Hettinger | 7496b41 | 2010-11-30 19:15:45 +0000 | [diff] [blame] | 76 | The cache's size limit assures that the cache does not grow without bound on |
| 77 | long-running processes such as web servers. |
| 78 | |
Raymond Hettinger | cc03858 | 2010-11-30 20:02:57 +0000 | [diff] [blame] | 79 | Example of an LRU cache for static web content:: |
Raymond Hettinger | 7496b41 | 2010-11-30 19:15:45 +0000 | [diff] [blame] | 80 | |
Raymond Hettinger | cc03858 | 2010-11-30 20:02:57 +0000 | [diff] [blame] | 81 | @lru_cache(maxsize=20) |
Raymond Hettinger | 7496b41 | 2010-11-30 19:15:45 +0000 | [diff] [blame] | 82 | def get_pep(num): |
| 83 | 'Retrieve text of a Python Enhancement Proposal' |
| 84 | resource = 'http://www.python.org/dev/peps/pep-%04d/' % num |
| 85 | try: |
| 86 | with urllib.request.urlopen(resource) as s: |
| 87 | return s.read() |
| 88 | except urllib.error.HTTPError: |
| 89 | return 'Not Found' |
| 90 | |
| 91 | >>> for n in 8, 290, 308, 320, 8, 218, 320, 279, 289, 320, 9991: |
| 92 | ... pep = get_pep(n) |
| 93 | ... print(n, len(pep)) |
| 94 | |
| 95 | >>> print(get_pep.cache_info()) |
| 96 | CacheInfo(hits=3, misses=8, maxsize=20, currsize=8) |
Georg Brandl | 2e7346a | 2010-07-31 18:09:23 +0000 | [diff] [blame] | 97 | |
Raymond Hettinger | c79fb0e | 2010-12-01 03:45:41 +0000 | [diff] [blame] | 98 | Example of efficiently computing |
| 99 | `Fibonacci numbers <http://en.wikipedia.org/wiki/Fibonacci_number>`_ |
| 100 | using a cache to implement a |
| 101 | `dynamic programming <http://en.wikipedia.org/wiki/Dynamic_programming>`_ |
| 102 | technique:: |
| 103 | |
| 104 | @lru_cache(maxsize=None) |
| 105 | def fib(n): |
| 106 | if n < 2: |
| 107 | return n |
| 108 | return fib(n-1) + fib(n-2) |
| 109 | |
| 110 | >>> print([fib(n) for n in range(16)]) |
| 111 | [0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610] |
| 112 | |
| 113 | >>> print(fib.cache_info()) |
| 114 | CacheInfo(hits=28, misses=16, maxsize=None, currsize=16) |
| 115 | |
Georg Brandl | 2e7346a | 2010-07-31 18:09:23 +0000 | [diff] [blame] | 116 | .. versionadded:: 3.2 |
| 117 | |
Raymond Hettinger | cd9fdfd | 2011-10-20 08:57:45 -0700 | [diff] [blame] | 118 | .. versionchanged:: 3.3 |
| 119 | Added the *typed* option. |
| 120 | |
Georg Brandl | 8a1caa2 | 2010-07-29 16:01:11 +0000 | [diff] [blame] | 121 | .. decorator:: total_ordering |
Raymond Hettinger | c50846a | 2010-04-05 18:56:31 +0000 | [diff] [blame] | 122 | |
| 123 | Given a class defining one or more rich comparison ordering methods, this |
Benjamin Peterson | 08bf91c | 2010-04-11 16:12:57 +0000 | [diff] [blame] | 124 | class decorator supplies the rest. This simplifies the effort involved |
Raymond Hettinger | c50846a | 2010-04-05 18:56:31 +0000 | [diff] [blame] | 125 | in specifying all of the possible rich comparison operations: |
| 126 | |
| 127 | The class must define one of :meth:`__lt__`, :meth:`__le__`, |
| 128 | :meth:`__gt__`, or :meth:`__ge__`. |
| 129 | In addition, the class should supply an :meth:`__eq__` method. |
| 130 | |
| 131 | For example:: |
| 132 | |
| 133 | @total_ordering |
| 134 | class Student: |
| 135 | def __eq__(self, other): |
| 136 | return ((self.lastname.lower(), self.firstname.lower()) == |
| 137 | (other.lastname.lower(), other.firstname.lower())) |
| 138 | def __lt__(self, other): |
| 139 | return ((self.lastname.lower(), self.firstname.lower()) < |
| 140 | (other.lastname.lower(), other.firstname.lower())) |
| 141 | |
| 142 | .. versionadded:: 3.2 |
| 143 | |
Georg Brandl | 67b21b7 | 2010-08-17 15:07:14 +0000 | [diff] [blame] | 144 | |
Georg Brandl | 036490d | 2009-05-17 13:00:36 +0000 | [diff] [blame] | 145 | .. function:: partial(func, *args, **keywords) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 146 | |
| 147 | Return a new :class:`partial` object which when called will behave like *func* |
| 148 | called with the positional arguments *args* and keyword arguments *keywords*. If |
| 149 | more arguments are supplied to the call, they are appended to *args*. If |
| 150 | additional keyword arguments are supplied, they extend and override *keywords*. |
| 151 | Roughly equivalent to:: |
| 152 | |
| 153 | def partial(func, *args, **keywords): |
| 154 | def newfunc(*fargs, **fkeywords): |
| 155 | newkeywords = keywords.copy() |
| 156 | newkeywords.update(fkeywords) |
| 157 | return func(*(args + fargs), **newkeywords) |
| 158 | newfunc.func = func |
| 159 | newfunc.args = args |
| 160 | newfunc.keywords = keywords |
| 161 | return newfunc |
| 162 | |
| 163 | The :func:`partial` is used for partial function application which "freezes" |
| 164 | some portion of a function's arguments and/or keywords resulting in a new object |
| 165 | with a simplified signature. For example, :func:`partial` can be used to create |
| 166 | a callable that behaves like the :func:`int` function where the *base* argument |
Christian Heimes | fe337bf | 2008-03-23 21:54:12 +0000 | [diff] [blame] | 167 | defaults to two: |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 168 | |
Christian Heimes | fe337bf | 2008-03-23 21:54:12 +0000 | [diff] [blame] | 169 | >>> from functools import partial |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 170 | >>> basetwo = partial(int, base=2) |
| 171 | >>> basetwo.__doc__ = 'Convert base 2 string to an int.' |
| 172 | >>> basetwo('10010') |
| 173 | 18 |
| 174 | |
| 175 | |
Georg Brandl | 58f9e4f | 2008-04-19 22:18:33 +0000 | [diff] [blame] | 176 | .. function:: reduce(function, iterable[, initializer]) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 177 | |
| 178 | Apply *function* of two arguments cumulatively to the items of *sequence*, from |
| 179 | left to right, so as to reduce the sequence to a single value. For example, |
| 180 | ``reduce(lambda x, y: x+y, [1, 2, 3, 4, 5])`` calculates ``((((1+2)+3)+4)+5)``. |
| 181 | The left argument, *x*, is the accumulated value and the right argument, *y*, is |
| 182 | the update value from the *sequence*. If the optional *initializer* is present, |
| 183 | it is placed before the items of the sequence in the calculation, and serves as |
| 184 | a default when the sequence is empty. If *initializer* is not given and |
| 185 | *sequence* contains only one item, the first item is returned. |
| 186 | |
| 187 | |
Georg Brandl | 036490d | 2009-05-17 13:00:36 +0000 | [diff] [blame] | 188 | .. function:: update_wrapper(wrapper, wrapped, assigned=WRAPPER_ASSIGNMENTS, updated=WRAPPER_UPDATES) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 189 | |
| 190 | Update a *wrapper* function to look like the *wrapped* function. The optional |
| 191 | arguments are tuples to specify which attributes of the original function are |
| 192 | assigned directly to the matching attributes on the wrapper function and which |
| 193 | attributes of the wrapper function are updated with the corresponding attributes |
| 194 | from the original function. The default values for these arguments are the |
| 195 | module level constants *WRAPPER_ASSIGNMENTS* (which assigns to the wrapper |
Antoine Pitrou | 560f764 | 2010-08-04 18:28:02 +0000 | [diff] [blame] | 196 | function's *__name__*, *__module__*, *__annotations__* and *__doc__*, the |
| 197 | documentation string) and *WRAPPER_UPDATES* (which updates the wrapper |
| 198 | function's *__dict__*, i.e. the instance dictionary). |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 199 | |
Nick Coghlan | 9887683 | 2010-08-17 06:17:18 +0000 | [diff] [blame] | 200 | To allow access to the original function for introspection and other purposes |
| 201 | (e.g. bypassing a caching decorator such as :func:`lru_cache`), this function |
Éric Araujo | c6ecb01 | 2010-11-06 06:33:03 +0000 | [diff] [blame] | 202 | automatically adds a __wrapped__ attribute to the wrapper that refers to |
Nick Coghlan | 9887683 | 2010-08-17 06:17:18 +0000 | [diff] [blame] | 203 | the original function. |
| 204 | |
Christian Heimes | d8654cf | 2007-12-02 15:22:16 +0000 | [diff] [blame] | 205 | The main intended use for this function is in :term:`decorator` functions which |
| 206 | wrap the decorated function and return the wrapper. If the wrapper function is |
| 207 | not updated, the metadata of the returned function will reflect the wrapper |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 208 | definition rather than the original function definition, which is typically less |
| 209 | than helpful. |
| 210 | |
Nick Coghlan | 9887683 | 2010-08-17 06:17:18 +0000 | [diff] [blame] | 211 | :func:`update_wrapper` may be used with callables other than functions. Any |
| 212 | attributes named in *assigned* or *updated* that are missing from the object |
| 213 | being wrapped are ignored (i.e. this function will not attempt to set them |
| 214 | on the wrapper function). :exc:`AttributeError` is still raised if the |
| 215 | wrapper function itself is missing any attributes named in *updated*. |
| 216 | |
| 217 | .. versionadded:: 3.2 |
Georg Brandl | 9e25701 | 2010-08-17 14:11:59 +0000 | [diff] [blame] | 218 | Automatic addition of the ``__wrapped__`` attribute. |
Nick Coghlan | 9887683 | 2010-08-17 06:17:18 +0000 | [diff] [blame] | 219 | |
| 220 | .. versionadded:: 3.2 |
Georg Brandl | 9e25701 | 2010-08-17 14:11:59 +0000 | [diff] [blame] | 221 | Copying of the ``__annotations__`` attribute by default. |
Nick Coghlan | 9887683 | 2010-08-17 06:17:18 +0000 | [diff] [blame] | 222 | |
| 223 | .. versionchanged:: 3.2 |
Georg Brandl | 9e25701 | 2010-08-17 14:11:59 +0000 | [diff] [blame] | 224 | Missing attributes no longer trigger an :exc:`AttributeError`. |
| 225 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 226 | |
Georg Brandl | 8a1caa2 | 2010-07-29 16:01:11 +0000 | [diff] [blame] | 227 | .. decorator:: wraps(wrapped, assigned=WRAPPER_ASSIGNMENTS, updated=WRAPPER_UPDATES) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 228 | |
| 229 | This is a convenience function for invoking ``partial(update_wrapper, |
| 230 | wrapped=wrapped, assigned=assigned, updated=updated)`` as a function decorator |
Christian Heimes | fe337bf | 2008-03-23 21:54:12 +0000 | [diff] [blame] | 231 | when defining a wrapper function. For example: |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 232 | |
Christian Heimes | fe337bf | 2008-03-23 21:54:12 +0000 | [diff] [blame] | 233 | >>> from functools import wraps |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 234 | >>> def my_decorator(f): |
| 235 | ... @wraps(f) |
| 236 | ... def wrapper(*args, **kwds): |
Georg Brandl | 6911e3c | 2007-09-04 07:15:32 +0000 | [diff] [blame] | 237 | ... print('Calling decorated function') |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 238 | ... return f(*args, **kwds) |
| 239 | ... return wrapper |
| 240 | ... |
| 241 | >>> @my_decorator |
| 242 | ... def example(): |
| 243 | ... """Docstring""" |
Georg Brandl | 6911e3c | 2007-09-04 07:15:32 +0000 | [diff] [blame] | 244 | ... print('Called example function') |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 245 | ... |
| 246 | >>> example() |
| 247 | Calling decorated function |
| 248 | Called example function |
| 249 | >>> example.__name__ |
| 250 | 'example' |
| 251 | >>> example.__doc__ |
| 252 | 'Docstring' |
| 253 | |
| 254 | Without the use of this decorator factory, the name of the example function |
| 255 | would have been ``'wrapper'``, and the docstring of the original :func:`example` |
| 256 | would have been lost. |
| 257 | |
| 258 | |
| 259 | .. _partial-objects: |
| 260 | |
| 261 | :class:`partial` Objects |
| 262 | ------------------------ |
| 263 | |
| 264 | :class:`partial` objects are callable objects created by :func:`partial`. They |
| 265 | have three read-only attributes: |
| 266 | |
| 267 | |
| 268 | .. attribute:: partial.func |
| 269 | |
| 270 | A callable object or function. Calls to the :class:`partial` object will be |
| 271 | forwarded to :attr:`func` with new arguments and keywords. |
| 272 | |
| 273 | |
| 274 | .. attribute:: partial.args |
| 275 | |
| 276 | The leftmost positional arguments that will be prepended to the positional |
| 277 | arguments provided to a :class:`partial` object call. |
| 278 | |
| 279 | |
| 280 | .. attribute:: partial.keywords |
| 281 | |
| 282 | The keyword arguments that will be supplied when the :class:`partial` object is |
| 283 | called. |
| 284 | |
| 285 | :class:`partial` objects are like :class:`function` objects in that they are |
| 286 | callable, weak referencable, and can have attributes. There are some important |
| 287 | differences. For instance, the :attr:`__name__` and :attr:`__doc__` attributes |
| 288 | are not created automatically. Also, :class:`partial` objects defined in |
| 289 | classes behave like static methods and do not transform into bound methods |
| 290 | during instance attribute look-up. |
| 291 | |