blob: 8bcc2ac1be76f1c96e75e476225d03e27209c36d [file] [log] [blame]
Georg Brandl116aa622007-08-15 14:28:22 +00001
Raymond Hettinger53dbe392008-02-12 20:03:09 +00002:mod:`collections` --- Container datatypes
3==========================================
Georg Brandl116aa622007-08-15 14:28:22 +00004
5.. module:: collections
Raymond Hettinger53dbe392008-02-12 20:03:09 +00006 :synopsis: Container datatypes
Georg Brandl116aa622007-08-15 14:28:22 +00007.. moduleauthor:: Raymond Hettinger <python@rcn.com>
8.. sectionauthor:: Raymond Hettinger <python@rcn.com>
9
Christian Heimesfe337bf2008-03-23 21:54:12 +000010.. testsetup:: *
11
12 from collections import *
13 import itertools
14 __name__ = '<doctest>'
Georg Brandl116aa622007-08-15 14:28:22 +000015
Georg Brandl116aa622007-08-15 14:28:22 +000016This module implements high-performance container datatypes. Currently,
17there are two datatypes, :class:`deque` and :class:`defaultdict`, and
Mark Summerfield71316b02008-02-14 16:28:00 +000018one datatype factory function, :func:`namedtuple`. This module also
19provides the :class:`UserDict` and :class:`UserList` classes which may
20be useful when inheriting directly from :class:`dict` or
21:class:`list` isn't convenient.
Christian Heimes0bd4e112008-02-12 22:59:25 +000022
Raymond Hettingerebcee3f2008-02-06 19:54:00 +000023The specialized containers provided in this module provide alternatives
Christian Heimesfe337bf2008-03-23 21:54:12 +000024to Python's general purpose built-in containers, :class:`dict`,
Raymond Hettingerebcee3f2008-02-06 19:54:00 +000025:class:`list`, :class:`set`, and :class:`tuple`.
Georg Brandl116aa622007-08-15 14:28:22 +000026
Mark Summerfield08898b42007-09-05 08:43:04 +000027In addition to containers, the collections module provides some ABCs
Christian Heimesfe337bf2008-03-23 21:54:12 +000028(abstract base classes) that can be used to test whether a class
Raymond Hettingerebcee3f2008-02-06 19:54:00 +000029provides a particular interface, for example, is it hashable or
Mark Summerfield71316b02008-02-14 16:28:00 +000030a mapping, and some of them can also be used as mixin classes.
Raymond Hettingerebcee3f2008-02-06 19:54:00 +000031
32ABCs - abstract base classes
33----------------------------
34
35The collections module offers the following ABCs:
Mark Summerfield08898b42007-09-05 08:43:04 +000036
Georg Brandl86b2fb92008-07-16 03:43:04 +000037========================= ===================== ====================== ====================================================
38ABC Inherits Abstract Methods Mixin Methods
39========================= ===================== ====================== ====================================================
40:class:`Container` ``__contains__``
41:class:`Hashable` ``__hash__``
42:class:`Iterable` ``__iter__``
43:class:`Iterator` :class:`Iterable` ``__next__`` ``__iter__``
44:class:`Sized` ``__len__``
45:class:`Callable` ``__call__``
46
47:class:`Sequence` :class:`Sized`, ``__getitem__`` ``__contains__``. ``__iter__``, ``__reversed__``.
48 :class:`Iterable`, and ``__len__`` ``index``, and ``count``
49 :class:`Container`
50
51:class:`MutableSequnce` :class:`Sequence` ``__getitem__`` Inherited Sequence methods and
52 ``__delitem__``, ``append``, ``reverse``, ``extend``, ``pop``,
53 ``insert``, ``remove``, and ``__iadd__``
54 and ``__len__``
55
56:class:`Set` :class:`Sized`, ``__len__``, ``__le__``, ``__lt__``, ``__eq__``, ``__ne__``,
57 :class:`Iterable`, ``__iter__``, and ``__gt__``, ``__ge__``, ``__and__``, ``__or__``
58 :class:`Container` ``__contains__`` ``__sub__``, ``__xor__``, and ``isdisjoint``
59
60:class:`MutableSet` :class:`Set` ``add`` and Inherited Set methods and
61 ``discard`` ``clear``, ``pop``, ``remove``, ``__ior__``,
62 ``__iand__``, ``__ixor__``, and ``__isub__``
63
64:class:`Mapping` :class:`Sized`, ``__getitem__``, ``__contains__``, ``keys``, ``items``, ``values``,
65 :class:`Iterable`, ``__len__``. and ``get``, ``__eq__``, and ``__ne__``
66 :class:`Container` ``__iter__``
67
68:class:`MutableMapping` :class:`Mapping` ``__getitem__`` Inherited Mapping methods and
69 ``__setitem__``, ``pop``, ``popitem``, ``clear``, ``update``,
70 ``__delitem__``, and ``setdefault``
71 ``__iter__``, and
72 ``__len__``
73
74:class:`MappingView` :class:`Sized` ``__len__``
75:class:`KeysView` :class:`MappingView`, ``__contains__``,
76 :class:`Set` ``__iter__``
77:class:`ItemsView` :class:`MappingView`, ``__contains__``,
78 :class:`Set` ``__iter__``
79:class:`ValuesView` :class:`MappingView` ``__contains__``, ``__iter__``
80========================= ===================== ====================== ====================================================
Mark Summerfield08898b42007-09-05 08:43:04 +000081
Mark Summerfield08898b42007-09-05 08:43:04 +000082These ABCs allow us to ask classes or instances if they provide
83particular functionality, for example::
84
Mark Summerfield08898b42007-09-05 08:43:04 +000085 size = None
Raymond Hettingerebcee3f2008-02-06 19:54:00 +000086 if isinstance(myvar, collections.Sized):
Mark Summerfield08898b42007-09-05 08:43:04 +000087 size = len(myvar)
88
Raymond Hettingerebcee3f2008-02-06 19:54:00 +000089Several of the ABCs are also useful as mixins that make it easier to develop
90classes supporting container APIs. For example, to write a class supporting
91the full :class:`Set` API, it only necessary to supply the three underlying
92abstract methods: :meth:`__contains__`, :meth:`__iter__`, and :meth:`__len__`.
93The ABC supplies the remaining methods such as :meth:`__and__` and
94:meth:`isdisjoint` ::
95
96 class ListBasedSet(collections.Set):
Raymond Hettingerc1b6a4a2008-02-08 23:46:23 +000097 ''' Alternate set implementation favoring space over speed
98 and not requiring the set elements to be hashable. '''
Raymond Hettingerebcee3f2008-02-06 19:54:00 +000099 def __init__(self, iterable):
Raymond Hettingerc1b6a4a2008-02-08 23:46:23 +0000100 self.elements = lst = []
101 for value in iterable:
102 if value not in lst:
103 lst.append(value)
Raymond Hettingerebcee3f2008-02-06 19:54:00 +0000104 def __iter__(self):
105 return iter(self.elements)
106 def __contains__(self, value):
107 return value in self.elements
108 def __len__(self):
109 return len(self.elements)
110
111 s1 = ListBasedSet('abcdef')
112 s2 = ListBasedSet('defghi')
113 overlap = s1 & s2 # The __and__() method is supported automatically
114
Raymond Hettinger7aebb642008-02-09 03:25:08 +0000115Notes on using :class:`Set` and :class:`MutableSet` as a mixin:
116
Christian Heimesfe337bf2008-03-23 21:54:12 +0000117(1)
Raymond Hettinger7aebb642008-02-09 03:25:08 +0000118 Since some set operations create new sets, the default mixin methods need
Christian Heimesfe337bf2008-03-23 21:54:12 +0000119 a way to create new instances from an iterable. The class constructor is
120 assumed to have a signature in the form ``ClassName(iterable)``.
Benjamin Peterson2b7411d2008-05-26 17:36:47 +0000121 That assumption is factored-out to an internal classmethod called
Raymond Hettinger7aebb642008-02-09 03:25:08 +0000122 :meth:`_from_iterable` which calls ``cls(iterable)`` to produce a new set.
123 If the :class:`Set` mixin is being used in a class with a different
Christian Heimesfe337bf2008-03-23 21:54:12 +0000124 constructor signature, you will need to override :meth:`from_iterable`
125 with a classmethod that can construct new instances from
Raymond Hettinger7aebb642008-02-09 03:25:08 +0000126 an iterable argument.
127
128(2)
129 To override the comparisons (presumably for speed, as the
130 semantics are fixed), redefine :meth:`__le__` and
131 then the other operations will automatically follow suit.
Raymond Hettingerebcee3f2008-02-06 19:54:00 +0000132
Raymond Hettinger0dbdab22008-02-09 03:48:16 +0000133(3)
134 The :class:`Set` mixin provides a :meth:`_hash` method to compute a hash value
135 for the set; however, :meth:`__hash__` is not defined because not all sets
136 are hashable or immutable. To add set hashabilty using mixins,
137 inherit from both :meth:`Set` and :meth:`Hashable`, then define
138 ``__hash__ = Set._hash``.
139
Mark Summerfield08898b42007-09-05 08:43:04 +0000140(For more about ABCs, see the :mod:`abc` module and :pep:`3119`.)
141
142
Georg Brandl116aa622007-08-15 14:28:22 +0000143.. _deque-objects:
144
145:class:`deque` objects
146----------------------
147
148
Georg Brandl9afde1c2007-11-01 20:32:30 +0000149.. class:: deque([iterable[, maxlen]])
Georg Brandl116aa622007-08-15 14:28:22 +0000150
151 Returns a new deque object initialized left-to-right (using :meth:`append`) with
152 data from *iterable*. If *iterable* is not specified, the new deque is empty.
153
154 Deques are a generalization of stacks and queues (the name is pronounced "deck"
155 and is short for "double-ended queue"). Deques support thread-safe, memory
156 efficient appends and pops from either side of the deque with approximately the
157 same O(1) performance in either direction.
158
159 Though :class:`list` objects support similar operations, they are optimized for
160 fast fixed-length operations and incur O(n) memory movement costs for
161 ``pop(0)`` and ``insert(0, v)`` operations which change both the size and
162 position of the underlying data representation.
163
Georg Brandl116aa622007-08-15 14:28:22 +0000164
Georg Brandl9afde1c2007-11-01 20:32:30 +0000165 If *maxlen* is not specified or is *None*, deques may grow to an
166 arbitrary length. Otherwise, the deque is bounded to the specified maximum
167 length. Once a bounded length deque is full, when new items are added, a
168 corresponding number of items are discarded from the opposite end. Bounded
169 length deques provide functionality similar to the ``tail`` filter in
170 Unix. They are also useful for tracking transactions and other pools of data
171 where only the most recent activity is of interest.
172
Georg Brandl9afde1c2007-11-01 20:32:30 +0000173
Benjamin Petersone41251e2008-04-25 01:59:09 +0000174 Deque objects support the following methods:
Georg Brandl116aa622007-08-15 14:28:22 +0000175
Benjamin Petersone41251e2008-04-25 01:59:09 +0000176 .. method:: append(x)
Georg Brandl116aa622007-08-15 14:28:22 +0000177
Benjamin Petersone41251e2008-04-25 01:59:09 +0000178 Add *x* to the right side of the deque.
Georg Brandl116aa622007-08-15 14:28:22 +0000179
180
Benjamin Petersone41251e2008-04-25 01:59:09 +0000181 .. method:: appendleft(x)
Georg Brandl116aa622007-08-15 14:28:22 +0000182
Benjamin Petersone41251e2008-04-25 01:59:09 +0000183 Add *x* to the left side of the deque.
Georg Brandl116aa622007-08-15 14:28:22 +0000184
185
Benjamin Petersone41251e2008-04-25 01:59:09 +0000186 .. method:: clear()
Georg Brandl116aa622007-08-15 14:28:22 +0000187
Benjamin Petersone41251e2008-04-25 01:59:09 +0000188 Remove all elements from the deque leaving it with length 0.
Georg Brandl116aa622007-08-15 14:28:22 +0000189
190
Benjamin Petersone41251e2008-04-25 01:59:09 +0000191 .. method:: extend(iterable)
Georg Brandl116aa622007-08-15 14:28:22 +0000192
Benjamin Petersone41251e2008-04-25 01:59:09 +0000193 Extend the right side of the deque by appending elements from the iterable
194 argument.
Georg Brandl116aa622007-08-15 14:28:22 +0000195
196
Benjamin Petersone41251e2008-04-25 01:59:09 +0000197 .. method:: extendleft(iterable)
Georg Brandl116aa622007-08-15 14:28:22 +0000198
Benjamin Petersone41251e2008-04-25 01:59:09 +0000199 Extend the left side of the deque by appending elements from *iterable*.
200 Note, the series of left appends results in reversing the order of
201 elements in the iterable argument.
Georg Brandl116aa622007-08-15 14:28:22 +0000202
203
Benjamin Petersone41251e2008-04-25 01:59:09 +0000204 .. method:: pop()
Georg Brandl116aa622007-08-15 14:28:22 +0000205
Benjamin Petersone41251e2008-04-25 01:59:09 +0000206 Remove and return an element from the right side of the deque. If no
207 elements are present, raises an :exc:`IndexError`.
Georg Brandl116aa622007-08-15 14:28:22 +0000208
209
Benjamin Petersone41251e2008-04-25 01:59:09 +0000210 .. method:: popleft()
Georg Brandl116aa622007-08-15 14:28:22 +0000211
Benjamin Petersone41251e2008-04-25 01:59:09 +0000212 Remove and return an element from the left side of the deque. If no
213 elements are present, raises an :exc:`IndexError`.
Georg Brandl116aa622007-08-15 14:28:22 +0000214
215
Benjamin Petersone41251e2008-04-25 01:59:09 +0000216 .. method:: remove(value)
Georg Brandl116aa622007-08-15 14:28:22 +0000217
Benjamin Petersone41251e2008-04-25 01:59:09 +0000218 Removed the first occurrence of *value*. If not found, raises a
219 :exc:`ValueError`.
Georg Brandl116aa622007-08-15 14:28:22 +0000220
Georg Brandl116aa622007-08-15 14:28:22 +0000221
Benjamin Petersone41251e2008-04-25 01:59:09 +0000222 .. method:: rotate(n)
Georg Brandl116aa622007-08-15 14:28:22 +0000223
Benjamin Petersone41251e2008-04-25 01:59:09 +0000224 Rotate the deque *n* steps to the right. If *n* is negative, rotate to
225 the left. Rotating one step to the right is equivalent to:
226 ``d.appendleft(d.pop())``.
227
Georg Brandl116aa622007-08-15 14:28:22 +0000228
229In addition to the above, deques support iteration, pickling, ``len(d)``,
230``reversed(d)``, ``copy.copy(d)``, ``copy.deepcopy(d)``, membership testing with
231the :keyword:`in` operator, and subscript references such as ``d[-1]``.
232
Christian Heimesfe337bf2008-03-23 21:54:12 +0000233Example:
234
235.. doctest::
Georg Brandl116aa622007-08-15 14:28:22 +0000236
237 >>> from collections import deque
238 >>> d = deque('ghi') # make a new deque with three items
239 >>> for elem in d: # iterate over the deque's elements
Neal Norwitz752abd02008-05-13 04:55:24 +0000240 ... print(elem.upper())
Georg Brandl116aa622007-08-15 14:28:22 +0000241 G
242 H
243 I
244
245 >>> d.append('j') # add a new entry to the right side
246 >>> d.appendleft('f') # add a new entry to the left side
247 >>> d # show the representation of the deque
248 deque(['f', 'g', 'h', 'i', 'j'])
249
250 >>> d.pop() # return and remove the rightmost item
251 'j'
252 >>> d.popleft() # return and remove the leftmost item
253 'f'
254 >>> list(d) # list the contents of the deque
255 ['g', 'h', 'i']
256 >>> d[0] # peek at leftmost item
257 'g'
258 >>> d[-1] # peek at rightmost item
259 'i'
260
261 >>> list(reversed(d)) # list the contents of a deque in reverse
262 ['i', 'h', 'g']
263 >>> 'h' in d # search the deque
264 True
265 >>> d.extend('jkl') # add multiple elements at once
266 >>> d
267 deque(['g', 'h', 'i', 'j', 'k', 'l'])
268 >>> d.rotate(1) # right rotation
269 >>> d
270 deque(['l', 'g', 'h', 'i', 'j', 'k'])
271 >>> d.rotate(-1) # left rotation
272 >>> d
273 deque(['g', 'h', 'i', 'j', 'k', 'l'])
274
275 >>> deque(reversed(d)) # make a new deque in reverse order
276 deque(['l', 'k', 'j', 'i', 'h', 'g'])
277 >>> d.clear() # empty the deque
278 >>> d.pop() # cannot pop from an empty deque
279 Traceback (most recent call last):
280 File "<pyshell#6>", line 1, in -toplevel-
281 d.pop()
282 IndexError: pop from an empty deque
283
284 >>> d.extendleft('abc') # extendleft() reverses the input order
285 >>> d
286 deque(['c', 'b', 'a'])
287
288
289.. _deque-recipes:
290
Georg Brandl9afde1c2007-11-01 20:32:30 +0000291:class:`deque` Recipes
292^^^^^^^^^^^^^^^^^^^^^^
Georg Brandl116aa622007-08-15 14:28:22 +0000293
294This section shows various approaches to working with deques.
295
296The :meth:`rotate` method provides a way to implement :class:`deque` slicing and
297deletion. For example, a pure python implementation of ``del d[n]`` relies on
298the :meth:`rotate` method to position elements to be popped::
299
300 def delete_nth(d, n):
301 d.rotate(-n)
302 d.popleft()
303 d.rotate(n)
304
305To implement :class:`deque` slicing, use a similar approach applying
306:meth:`rotate` to bring a target element to the left side of the deque. Remove
307old entries with :meth:`popleft`, add new entries with :meth:`extend`, and then
308reverse the rotation.
Georg Brandl116aa622007-08-15 14:28:22 +0000309With minor variations on that approach, it is easy to implement Forth style
310stack manipulations such as ``dup``, ``drop``, ``swap``, ``over``, ``pick``,
311``rot``, and ``roll``.
312
Georg Brandl116aa622007-08-15 14:28:22 +0000313Multi-pass data reduction algorithms can be succinctly expressed and efficiently
314coded by extracting elements with multiple calls to :meth:`popleft`, applying
Georg Brandl9afde1c2007-11-01 20:32:30 +0000315a reduction function, and calling :meth:`append` to add the result back to the
316deque.
Georg Brandl116aa622007-08-15 14:28:22 +0000317
318For example, building a balanced binary tree of nested lists entails reducing
Christian Heimesfe337bf2008-03-23 21:54:12 +0000319two adjacent nodes into one by grouping them in a list:
Georg Brandl116aa622007-08-15 14:28:22 +0000320
321 >>> def maketree(iterable):
322 ... d = deque(iterable)
323 ... while len(d) > 1:
324 ... pair = [d.popleft(), d.popleft()]
325 ... d.append(pair)
326 ... return list(d)
327 ...
Georg Brandl6911e3c2007-09-04 07:15:32 +0000328 >>> print(maketree('abcdefgh'))
Georg Brandl116aa622007-08-15 14:28:22 +0000329 [[[['a', 'b'], ['c', 'd']], [['e', 'f'], ['g', 'h']]]]
330
Georg Brandl9afde1c2007-11-01 20:32:30 +0000331Bounded length deques provide functionality similar to the ``tail`` filter
332in Unix::
Georg Brandl116aa622007-08-15 14:28:22 +0000333
Georg Brandl9afde1c2007-11-01 20:32:30 +0000334 def tail(filename, n=10):
335 'Return the last n lines of a file'
336 return deque(open(filename), n)
Georg Brandl116aa622007-08-15 14:28:22 +0000337
338.. _defaultdict-objects:
339
340:class:`defaultdict` objects
341----------------------------
342
343
344.. class:: defaultdict([default_factory[, ...]])
345
346 Returns a new dictionary-like object. :class:`defaultdict` is a subclass of the
347 builtin :class:`dict` class. It overrides one method and adds one writable
348 instance variable. The remaining functionality is the same as for the
349 :class:`dict` class and is not documented here.
350
351 The first argument provides the initial value for the :attr:`default_factory`
352 attribute; it defaults to ``None``. All remaining arguments are treated the same
353 as if they were passed to the :class:`dict` constructor, including keyword
354 arguments.
355
Georg Brandl116aa622007-08-15 14:28:22 +0000356
Benjamin Petersone41251e2008-04-25 01:59:09 +0000357 :class:`defaultdict` objects support the following method in addition to the
358 standard :class:`dict` operations:
Georg Brandl116aa622007-08-15 14:28:22 +0000359
Benjamin Petersone41251e2008-04-25 01:59:09 +0000360 .. method:: defaultdict.__missing__(key)
Georg Brandl116aa622007-08-15 14:28:22 +0000361
Benjamin Peterson5478b472008-09-17 22:25:09 +0000362 If the :attr:`default_factory` attribute is ``None``, this raises a
Benjamin Petersone41251e2008-04-25 01:59:09 +0000363 :exc:`KeyError` exception with the *key* as argument.
Georg Brandl116aa622007-08-15 14:28:22 +0000364
Benjamin Petersone41251e2008-04-25 01:59:09 +0000365 If :attr:`default_factory` is not ``None``, it is called without arguments
366 to provide a default value for the given *key*, this value is inserted in
367 the dictionary for the *key*, and returned.
Georg Brandl116aa622007-08-15 14:28:22 +0000368
Benjamin Petersone41251e2008-04-25 01:59:09 +0000369 If calling :attr:`default_factory` raises an exception this exception is
370 propagated unchanged.
Georg Brandl116aa622007-08-15 14:28:22 +0000371
Benjamin Petersone41251e2008-04-25 01:59:09 +0000372 This method is called by the :meth:`__getitem__` method of the
373 :class:`dict` class when the requested key is not found; whatever it
374 returns or raises is then returned or raised by :meth:`__getitem__`.
Georg Brandl116aa622007-08-15 14:28:22 +0000375
376
Benjamin Petersone41251e2008-04-25 01:59:09 +0000377 :class:`defaultdict` objects support the following instance variable:
Georg Brandl116aa622007-08-15 14:28:22 +0000378
Benjamin Petersone41251e2008-04-25 01:59:09 +0000379
380 .. attribute:: defaultdict.default_factory
381
382 This attribute is used by the :meth:`__missing__` method; it is
383 initialized from the first argument to the constructor, if present, or to
384 ``None``, if absent.
Georg Brandl116aa622007-08-15 14:28:22 +0000385
386
387.. _defaultdict-examples:
388
389:class:`defaultdict` Examples
390^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
391
392Using :class:`list` as the :attr:`default_factory`, it is easy to group a
Christian Heimesfe337bf2008-03-23 21:54:12 +0000393sequence of key-value pairs into a dictionary of lists:
Georg Brandl116aa622007-08-15 14:28:22 +0000394
395 >>> s = [('yellow', 1), ('blue', 2), ('yellow', 3), ('blue', 4), ('red', 1)]
396 >>> d = defaultdict(list)
397 >>> for k, v in s:
398 ... d[k].append(v)
399 ...
400 >>> d.items()
401 [('blue', [2, 4]), ('red', [1]), ('yellow', [1, 3])]
402
403When each key is encountered for the first time, it is not already in the
404mapping; so an entry is automatically created using the :attr:`default_factory`
405function which returns an empty :class:`list`. The :meth:`list.append`
406operation then attaches the value to the new list. When keys are encountered
407again, the look-up proceeds normally (returning the list for that key) and the
408:meth:`list.append` operation adds another value to the list. This technique is
Christian Heimesfe337bf2008-03-23 21:54:12 +0000409simpler and faster than an equivalent technique using :meth:`dict.setdefault`:
Georg Brandl116aa622007-08-15 14:28:22 +0000410
411 >>> d = {}
412 >>> for k, v in s:
413 ... d.setdefault(k, []).append(v)
414 ...
415 >>> d.items()
416 [('blue', [2, 4]), ('red', [1]), ('yellow', [1, 3])]
417
418Setting the :attr:`default_factory` to :class:`int` makes the
419:class:`defaultdict` useful for counting (like a bag or multiset in other
Christian Heimesfe337bf2008-03-23 21:54:12 +0000420languages):
Georg Brandl116aa622007-08-15 14:28:22 +0000421
422 >>> s = 'mississippi'
423 >>> d = defaultdict(int)
424 >>> for k in s:
425 ... d[k] += 1
426 ...
427 >>> d.items()
428 [('i', 4), ('p', 2), ('s', 4), ('m', 1)]
429
430When a letter is first encountered, it is missing from the mapping, so the
431:attr:`default_factory` function calls :func:`int` to supply a default count of
432zero. The increment operation then builds up the count for each letter.
433
434The function :func:`int` which always returns zero is just a special case of
435constant functions. A faster and more flexible way to create constant functions
436is to use a lambda function which can supply any constant value (not just
Christian Heimesfe337bf2008-03-23 21:54:12 +0000437zero):
Georg Brandl116aa622007-08-15 14:28:22 +0000438
439 >>> def constant_factory(value):
440 ... return lambda: value
441 >>> d = defaultdict(constant_factory('<missing>'))
442 >>> d.update(name='John', action='ran')
443 >>> '%(name)s %(action)s to %(object)s' % d
444 'John ran to <missing>'
445
446Setting the :attr:`default_factory` to :class:`set` makes the
Christian Heimesfe337bf2008-03-23 21:54:12 +0000447:class:`defaultdict` useful for building a dictionary of sets:
Georg Brandl116aa622007-08-15 14:28:22 +0000448
449 >>> s = [('red', 1), ('blue', 2), ('red', 3), ('blue', 4), ('red', 1), ('blue', 4)]
450 >>> d = defaultdict(set)
451 >>> for k, v in s:
452 ... d[k].add(v)
453 ...
454 >>> d.items()
455 [('blue', set([2, 4])), ('red', set([1, 3]))]
456
457
458.. _named-tuple-factory:
459
Guido van Rossum3d392eb2007-11-16 00:35:22 +0000460:func:`namedtuple` Factory Function for Tuples with Named Fields
Christian Heimes790c8232008-01-07 21:14:23 +0000461----------------------------------------------------------------
Georg Brandl116aa622007-08-15 14:28:22 +0000462
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000463Named tuples assign meaning to each position in a tuple and allow for more readable,
464self-documenting code. They can be used wherever regular tuples are used, and
465they add the ability to access fields by name instead of position index.
Georg Brandl116aa622007-08-15 14:28:22 +0000466
Guido van Rossum3d392eb2007-11-16 00:35:22 +0000467.. function:: namedtuple(typename, fieldnames, [verbose])
Georg Brandl116aa622007-08-15 14:28:22 +0000468
469 Returns a new tuple subclass named *typename*. The new subclass is used to
Christian Heimesc3f30c42008-02-22 16:37:40 +0000470 create tuple-like objects that have fields accessible by attribute lookup as
Georg Brandl116aa622007-08-15 14:28:22 +0000471 well as being indexable and iterable. Instances of the subclass also have a
472 helpful docstring (with typename and fieldnames) and a helpful :meth:`__repr__`
473 method which lists the tuple contents in a ``name=value`` format.
474
Georg Brandl9afde1c2007-11-01 20:32:30 +0000475 The *fieldnames* are a single string with each fieldname separated by whitespace
Christian Heimes25bb7832008-01-11 16:17:00 +0000476 and/or commas, for example ``'x y'`` or ``'x, y'``. Alternatively, *fieldnames*
477 can be a sequence of strings such as ``['x', 'y']``.
Georg Brandl9afde1c2007-11-01 20:32:30 +0000478
479 Any valid Python identifier may be used for a fieldname except for names
Christian Heimes0449f632007-12-15 01:27:15 +0000480 starting with an underscore. Valid identifiers consist of letters, digits,
481 and underscores but do not start with a digit or underscore and cannot be
Georg Brandlf6945182008-02-01 11:56:49 +0000482 a :mod:`keyword` such as *class*, *for*, *return*, *global*, *pass*,
Georg Brandl9afde1c2007-11-01 20:32:30 +0000483 or *raise*.
Georg Brandl116aa622007-08-15 14:28:22 +0000484
Christian Heimes25bb7832008-01-11 16:17:00 +0000485 If *verbose* is true, the class definition is printed just before being built.
Georg Brandl116aa622007-08-15 14:28:22 +0000486
Georg Brandl9afde1c2007-11-01 20:32:30 +0000487 Named tuple instances do not have per-instance dictionaries, so they are
Thomas Wouters8ce81f72007-09-20 18:22:40 +0000488 lightweight and require no more memory than regular tuples.
Georg Brandl116aa622007-08-15 14:28:22 +0000489
Christian Heimesfe337bf2008-03-23 21:54:12 +0000490Example:
491
492.. doctest::
493 :options: +NORMALIZE_WHITESPACE
Georg Brandl116aa622007-08-15 14:28:22 +0000494
Guido van Rossum3d392eb2007-11-16 00:35:22 +0000495 >>> Point = namedtuple('Point', 'x y', verbose=True)
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000496 class Point(tuple):
497 'Point(x, y)'
Christian Heimesfe337bf2008-03-23 21:54:12 +0000498 <BLANKLINE>
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000499 __slots__ = ()
Christian Heimesfe337bf2008-03-23 21:54:12 +0000500 <BLANKLINE>
Christian Heimesfaf2f632008-01-06 16:59:19 +0000501 _fields = ('x', 'y')
Christian Heimesfe337bf2008-03-23 21:54:12 +0000502 <BLANKLINE>
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000503 def __new__(cls, x, y):
504 return tuple.__new__(cls, (x, y))
Christian Heimesfe337bf2008-03-23 21:54:12 +0000505 <BLANKLINE>
Christian Heimesfaf2f632008-01-06 16:59:19 +0000506 @classmethod
Christian Heimesfe337bf2008-03-23 21:54:12 +0000507 def _make(cls, iterable, new=tuple.__new__, len=len):
Christian Heimesfaf2f632008-01-06 16:59:19 +0000508 'Make a new Point object from a sequence or iterable'
Christian Heimesfe337bf2008-03-23 21:54:12 +0000509 result = new(cls, iterable)
Christian Heimesfaf2f632008-01-06 16:59:19 +0000510 if len(result) != 2:
511 raise TypeError('Expected 2 arguments, got %d' % len(result))
512 return result
Christian Heimesfe337bf2008-03-23 21:54:12 +0000513 <BLANKLINE>
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000514 def __repr__(self):
515 return 'Point(x=%r, y=%r)' % self
Christian Heimesfe337bf2008-03-23 21:54:12 +0000516 <BLANKLINE>
Christian Heimes99170a52007-12-19 02:07:34 +0000517 def _asdict(t):
Christian Heimes0449f632007-12-15 01:27:15 +0000518 'Return a new dict which maps field names to their values'
Christian Heimes99170a52007-12-19 02:07:34 +0000519 return {'x': t[0], 'y': t[1]}
Christian Heimesfe337bf2008-03-23 21:54:12 +0000520 <BLANKLINE>
Christian Heimes0449f632007-12-15 01:27:15 +0000521 def _replace(self, **kwds):
Guido van Rossum3d392eb2007-11-16 00:35:22 +0000522 'Return a new Point object replacing specified fields with new values'
Christian Heimesfaf2f632008-01-06 16:59:19 +0000523 result = self._make(map(kwds.pop, ('x', 'y'), self))
524 if kwds:
525 raise ValueError('Got unexpected field names: %r' % kwds.keys())
526 return result
Georg Brandlc28e1fa2008-06-10 19:20:26 +0000527 <BLANKLINE>
Benjamin Peterson41181742008-07-02 20:22:54 +0000528 def __getnewargs__(self):
529 return tuple(self)
Christian Heimesfe337bf2008-03-23 21:54:12 +0000530 <BLANKLINE>
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000531 x = property(itemgetter(0))
532 y = property(itemgetter(1))
Georg Brandl116aa622007-08-15 14:28:22 +0000533
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000534 >>> p = Point(11, y=22) # instantiate with positional or keyword arguments
Christian Heimes99170a52007-12-19 02:07:34 +0000535 >>> p[0] + p[1] # indexable like the plain tuple (11, 22)
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000536 33
537 >>> x, y = p # unpack like a regular tuple
538 >>> x, y
539 (11, 22)
Christian Heimesc3f30c42008-02-22 16:37:40 +0000540 >>> p.x + p.y # fields also accessible by name
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000541 33
542 >>> p # readable __repr__ with a name=value style
543 Point(x=11, y=22)
Georg Brandl116aa622007-08-15 14:28:22 +0000544
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000545Named tuples are especially useful for assigning field names to result tuples returned
546by the :mod:`csv` or :mod:`sqlite3` modules::
547
Guido van Rossum3d392eb2007-11-16 00:35:22 +0000548 EmployeeRecord = namedtuple('EmployeeRecord', 'name, age, title, department, paygrade')
Georg Brandl9afde1c2007-11-01 20:32:30 +0000549
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000550 import csv
Christian Heimesfaf2f632008-01-06 16:59:19 +0000551 for emp in map(EmployeeRecord._make, csv.reader(open("employees.csv", "rb"))):
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000552 print(emp.name, emp.title)
553
Georg Brandl9afde1c2007-11-01 20:32:30 +0000554 import sqlite3
555 conn = sqlite3.connect('/companydata')
556 cursor = conn.cursor()
557 cursor.execute('SELECT name, age, title, department, paygrade FROM employees')
Christian Heimesfaf2f632008-01-06 16:59:19 +0000558 for emp in map(EmployeeRecord._make, cursor.fetchall()):
Christian Heimes00412232008-01-10 16:02:19 +0000559 print(emp.name, emp.title)
Georg Brandl9afde1c2007-11-01 20:32:30 +0000560
Christian Heimes99170a52007-12-19 02:07:34 +0000561In addition to the methods inherited from tuples, named tuples support
Christian Heimes2380ac72008-01-09 00:17:24 +0000562three additional methods and one attribute. To prevent conflicts with
563field names, the method and attribute names start with an underscore.
Christian Heimes99170a52007-12-19 02:07:34 +0000564
Christian Heimes790c8232008-01-07 21:14:23 +0000565.. method:: somenamedtuple._make(iterable)
Christian Heimes99170a52007-12-19 02:07:34 +0000566
Christian Heimesfaf2f632008-01-06 16:59:19 +0000567 Class method that makes a new instance from an existing sequence or iterable.
Christian Heimes99170a52007-12-19 02:07:34 +0000568
Christian Heimesfe337bf2008-03-23 21:54:12 +0000569.. doctest::
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000570
Christian Heimesfaf2f632008-01-06 16:59:19 +0000571 >>> t = [11, 22]
572 >>> Point._make(t)
573 Point(x=11, y=22)
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000574
Christian Heimes790c8232008-01-07 21:14:23 +0000575.. method:: somenamedtuple._asdict()
Georg Brandl9afde1c2007-11-01 20:32:30 +0000576
Christian Heimesfe337bf2008-03-23 21:54:12 +0000577 Return a new dict which maps field names to their corresponding values::
Georg Brandl9afde1c2007-11-01 20:32:30 +0000578
Christian Heimes0449f632007-12-15 01:27:15 +0000579 >>> p._asdict()
Georg Brandl9afde1c2007-11-01 20:32:30 +0000580 {'x': 11, 'y': 22}
Christian Heimesfe337bf2008-03-23 21:54:12 +0000581
Christian Heimes790c8232008-01-07 21:14:23 +0000582.. method:: somenamedtuple._replace(kwargs)
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000583
Christian Heimesfe337bf2008-03-23 21:54:12 +0000584 Return a new instance of the named tuple replacing specified fields with new
585 values:
Thomas Wouters8ce81f72007-09-20 18:22:40 +0000586
587::
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000588
589 >>> p = Point(x=11, y=22)
Christian Heimes0449f632007-12-15 01:27:15 +0000590 >>> p._replace(x=33)
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000591 Point(x=33, y=22)
592
Guido van Rossum3d392eb2007-11-16 00:35:22 +0000593 >>> for partnum, record in inventory.items():
Christian Heimes454f37b2008-01-10 00:10:02 +0000594 ... inventory[partnum] = record._replace(price=newprices[partnum], timestamp=time.now())
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000595
Christian Heimes790c8232008-01-07 21:14:23 +0000596.. attribute:: somenamedtuple._fields
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000597
Christian Heimes2380ac72008-01-09 00:17:24 +0000598 Tuple of strings listing the field names. Useful for introspection
Georg Brandl9afde1c2007-11-01 20:32:30 +0000599 and for creating new named tuple types from existing named tuples.
Thomas Wouters8ce81f72007-09-20 18:22:40 +0000600
Christian Heimesfe337bf2008-03-23 21:54:12 +0000601.. doctest::
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000602
Christian Heimes0449f632007-12-15 01:27:15 +0000603 >>> p._fields # view the field names
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000604 ('x', 'y')
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000605
Guido van Rossum3d392eb2007-11-16 00:35:22 +0000606 >>> Color = namedtuple('Color', 'red green blue')
Christian Heimes0449f632007-12-15 01:27:15 +0000607 >>> Pixel = namedtuple('Pixel', Point._fields + Color._fields)
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000608 >>> Pixel(11, 22, 128, 255, 0)
Christian Heimes454f37b2008-01-10 00:10:02 +0000609 Pixel(x=11, y=22, red=128, green=255, blue=0)
Georg Brandl116aa622007-08-15 14:28:22 +0000610
Christian Heimes0449f632007-12-15 01:27:15 +0000611To retrieve a field whose name is stored in a string, use the :func:`getattr`
Christian Heimesfe337bf2008-03-23 21:54:12 +0000612function:
Christian Heimes0449f632007-12-15 01:27:15 +0000613
614 >>> getattr(p, 'x')
615 11
616
Christian Heimesfe337bf2008-03-23 21:54:12 +0000617To convert a dictionary to a named tuple, use the double-star-operator [#]_:
Christian Heimes99170a52007-12-19 02:07:34 +0000618
619 >>> d = {'x': 11, 'y': 22}
620 >>> Point(**d)
621 Point(x=11, y=22)
622
Guido van Rossum3d392eb2007-11-16 00:35:22 +0000623Since a named tuple is a regular Python class, it is easy to add or change
Christian Heimes043d6f62008-01-07 17:19:16 +0000624functionality with a subclass. Here is how to add a calculated field and
Christian Heimesfe337bf2008-03-23 21:54:12 +0000625a fixed-width print format:
Guido van Rossum3d392eb2007-11-16 00:35:22 +0000626
Christian Heimes043d6f62008-01-07 17:19:16 +0000627 >>> class Point(namedtuple('Point', 'x y')):
Christian Heimes25bb7832008-01-11 16:17:00 +0000628 ... __slots__ = ()
Christian Heimes454f37b2008-01-10 00:10:02 +0000629 ... @property
630 ... def hypot(self):
631 ... return (self.x ** 2 + self.y ** 2) ** 0.5
632 ... def __str__(self):
Christian Heimes25bb7832008-01-11 16:17:00 +0000633 ... return 'Point: x=%6.3f y=%6.3f hypot=%6.3f' % (self.x, self.y, self.hypot)
Guido van Rossum3d392eb2007-11-16 00:35:22 +0000634
Georg Brandl0df79792008-10-04 18:33:26 +0000635 >>> for p in Point(3, 4), Point(14, 5/7):
Christian Heimes00412232008-01-10 16:02:19 +0000636 ... print(p)
Christian Heimes25bb7832008-01-11 16:17:00 +0000637 Point: x= 3.000 y= 4.000 hypot= 5.000
638 Point: x=14.000 y= 0.714 hypot=14.018
Christian Heimes043d6f62008-01-07 17:19:16 +0000639
Christian Heimesaf98da12008-01-27 15:18:18 +0000640The subclass shown above sets ``__slots__`` to an empty tuple. This keeps
Christian Heimes679db4a2008-01-18 09:56:22 +0000641keep memory requirements low by preventing the creation of instance dictionaries.
642
Christian Heimes2380ac72008-01-09 00:17:24 +0000643
644Subclassing is not useful for adding new, stored fields. Instead, simply
Christian Heimesfe337bf2008-03-23 21:54:12 +0000645create a new named tuple type from the :attr:`_fields` attribute:
Christian Heimes2380ac72008-01-09 00:17:24 +0000646
Christian Heimes25bb7832008-01-11 16:17:00 +0000647 >>> Point3D = namedtuple('Point3D', Point._fields + ('z',))
Christian Heimes2380ac72008-01-09 00:17:24 +0000648
649Default values can be implemented by using :meth:`_replace` to
Christian Heimesfe337bf2008-03-23 21:54:12 +0000650customize a prototype instance:
Guido van Rossum3d392eb2007-11-16 00:35:22 +0000651
652 >>> Account = namedtuple('Account', 'owner balance transaction_count')
Christian Heimes587c2bf2008-01-19 16:21:02 +0000653 >>> default_account = Account('<owner name>', 0.0, 0)
654 >>> johns_account = default_account._replace(owner='John')
Guido van Rossum3d392eb2007-11-16 00:35:22 +0000655
Christian Heimese4ca8152008-05-08 17:18:53 +0000656Enumerated constants can be implemented with named tuples, but it is simpler
657and more efficient to use a simple class declaration:
658
659 >>> Status = namedtuple('Status', 'open pending closed')._make(range(3))
660 >>> Status.open, Status.pending, Status.closed
661 (0, 1, 2)
662 >>> class Status:
663 ... open, pending, closed = range(3)
664
Thomas Wouters47b49bf2007-08-30 22:15:33 +0000665.. rubric:: Footnotes
666
Christian Heimes99170a52007-12-19 02:07:34 +0000667.. [#] For information on the double-star-operator see
Thomas Wouters47b49bf2007-08-30 22:15:33 +0000668 :ref:`tut-unpacking-arguments` and :ref:`calls`.
Raymond Hettingere4c96ad2008-02-06 01:23:58 +0000669
670
671
672:class:`UserDict` objects
Mark Summerfield8f2d0062008-02-06 13:30:44 +0000673-------------------------
Raymond Hettingere4c96ad2008-02-06 01:23:58 +0000674
675The class, :class:`UserDict` acts as a wrapper around dictionary objects.
676The need for this class has been partially supplanted by the ability to
677subclass directly from :class:`dict`; however, this class can be easier
678to work with because the underlying dictionary is accessible as an
679attribute.
680
681.. class:: UserDict([initialdata])
682
683 Class that simulates a dictionary. The instance's contents are kept in a
684 regular dictionary, which is accessible via the :attr:`data` attribute of
685 :class:`UserDict` instances. If *initialdata* is provided, :attr:`data` is
686 initialized with its contents; note that a reference to *initialdata* will not
687 be kept, allowing it be used for other purposes.
688
689In addition to supporting the methods and operations of mappings,
Raymond Hettingerebcee3f2008-02-06 19:54:00 +0000690:class:`UserDict` instances provide the following attribute:
Raymond Hettingere4c96ad2008-02-06 01:23:58 +0000691
692.. attribute:: UserDict.data
693
694 A real dictionary used to store the contents of the :class:`UserDict` class.
Raymond Hettinger53dbe392008-02-12 20:03:09 +0000695
696
697
698:class:`UserList` objects
699-------------------------
700
701This class acts as a wrapper around list objects. It is a useful base class
702for your own list-like classes which can inherit from them and override
703existing methods or add new ones. In this way, one can add new behaviors to
704lists.
705
706The need for this class has been partially supplanted by the ability to
707subclass directly from :class:`list`; however, this class can be easier
708to work with because the underlying list is accessible as an attribute.
709
710.. class:: UserList([list])
711
712 Class that simulates a list. The instance's contents are kept in a regular
713 list, which is accessible via the :attr:`data` attribute of :class:`UserList`
714 instances. The instance's contents are initially set to a copy of *list*,
715 defaulting to the empty list ``[]``. *list* can be any iterable, for
716 example a real Python list or a :class:`UserList` object.
717
718In addition to supporting the methods and operations of mutable sequences,
719:class:`UserList` instances provide the following attribute:
720
721.. attribute:: UserList.data
722
723 A real :class:`list` object used to store the contents of the
724 :class:`UserList` class.
725
726**Subclassing requirements:** Subclasses of :class:`UserList` are expect to
727offer a constructor which can be called with either no arguments or one
728argument. List operations which return a new sequence attempt to create an
729instance of the actual implementation class. To do so, it assumes that the
730constructor can be called with a single parameter, which is a sequence object
731used as a data source.
732
733If a derived class does not wish to comply with this requirement, all of the
734special methods supported by this class will need to be overridden; please
735consult the sources for information about the methods which need to be provided
736in that case.
Raymond Hettingerb3a65f82008-02-21 22:11:37 +0000737
738:class:`UserString` objects
Christian Heimesc3f30c42008-02-22 16:37:40 +0000739---------------------------
Raymond Hettingerb3a65f82008-02-21 22:11:37 +0000740
741The class, :class:`UserString` acts as a wrapper around string objects.
742The need for this class has been partially supplanted by the ability to
743subclass directly from :class:`str`; however, this class can be easier
744to work with because the underlying string is accessible as an
745attribute.
746
747.. class:: UserString([sequence])
748
749 Class that simulates a string or a Unicode string object. The instance's
750 content is kept in a regular string object, which is accessible via the
751 :attr:`data` attribute of :class:`UserString` instances. The instance's
752 contents are initially set to a copy of *sequence*. The *sequence* can
753 be an instance of :class:`bytes`, :class:`str`, :class:`UserString` (or a
754 subclass) or an arbitrary sequence which can be converted into a string using
755 the built-in :func:`str` function.