blob: b12bd7ae31ce1217de52b4a8a4f133c1b2487b98 [file] [log] [blame]
Georg Brandl116aa622007-08-15 14:28:22 +00001
Raymond Hettinger53dbe392008-02-12 20:03:09 +00002:mod:`collections` --- Container datatypes
3==========================================
Georg Brandl116aa622007-08-15 14:28:22 +00004
5.. module:: collections
Raymond Hettinger53dbe392008-02-12 20:03:09 +00006 :synopsis: Container datatypes
Georg Brandl116aa622007-08-15 14:28:22 +00007.. moduleauthor:: Raymond Hettinger <python@rcn.com>
8.. sectionauthor:: Raymond Hettinger <python@rcn.com>
9
Christian Heimesfe337bf2008-03-23 21:54:12 +000010.. testsetup:: *
11
12 from collections import *
13 import itertools
14 __name__ = '<doctest>'
Georg Brandl116aa622007-08-15 14:28:22 +000015
Georg Brandl116aa622007-08-15 14:28:22 +000016This module implements high-performance container datatypes. Currently,
17there are two datatypes, :class:`deque` and :class:`defaultdict`, and
Mark Summerfield71316b02008-02-14 16:28:00 +000018one datatype factory function, :func:`namedtuple`. This module also
19provides the :class:`UserDict` and :class:`UserList` classes which may
20be useful when inheriting directly from :class:`dict` or
21:class:`list` isn't convenient.
Christian Heimes0bd4e112008-02-12 22:59:25 +000022
Raymond Hettingerebcee3f2008-02-06 19:54:00 +000023The specialized containers provided in this module provide alternatives
Christian Heimesfe337bf2008-03-23 21:54:12 +000024to Python's general purpose built-in containers, :class:`dict`,
Raymond Hettingerebcee3f2008-02-06 19:54:00 +000025:class:`list`, :class:`set`, and :class:`tuple`.
Georg Brandl116aa622007-08-15 14:28:22 +000026
Mark Summerfield08898b42007-09-05 08:43:04 +000027In addition to containers, the collections module provides some ABCs
Christian Heimesfe337bf2008-03-23 21:54:12 +000028(abstract base classes) that can be used to test whether a class
Raymond Hettingerebcee3f2008-02-06 19:54:00 +000029provides a particular interface, for example, is it hashable or
Mark Summerfield71316b02008-02-14 16:28:00 +000030a mapping, and some of them can also be used as mixin classes.
Raymond Hettingerebcee3f2008-02-06 19:54:00 +000031
32ABCs - abstract base classes
33----------------------------
34
35The collections module offers the following ABCs:
Mark Summerfield08898b42007-09-05 08:43:04 +000036
Georg Brandl86b2fb92008-07-16 03:43:04 +000037========================= ===================== ====================== ====================================================
38ABC Inherits Abstract Methods Mixin Methods
39========================= ===================== ====================== ====================================================
40:class:`Container` ``__contains__``
41:class:`Hashable` ``__hash__``
42:class:`Iterable` ``__iter__``
43:class:`Iterator` :class:`Iterable` ``__next__`` ``__iter__``
44:class:`Sized` ``__len__``
45:class:`Callable` ``__call__``
46
47:class:`Sequence` :class:`Sized`, ``__getitem__`` ``__contains__``. ``__iter__``, ``__reversed__``.
48 :class:`Iterable`, and ``__len__`` ``index``, and ``count``
49 :class:`Container`
50
Benjamin Peterson4469d0c2008-11-30 22:46:23 +000051:class:`MutableSequence` :class:`Sequence` ``__getitem__`` Inherited Sequence methods and
Georg Brandl86b2fb92008-07-16 03:43:04 +000052 ``__delitem__``, ``append``, ``reverse``, ``extend``, ``pop``,
53 ``insert``, ``remove``, and ``__iadd__``
54 and ``__len__``
55
56:class:`Set` :class:`Sized`, ``__len__``, ``__le__``, ``__lt__``, ``__eq__``, ``__ne__``,
57 :class:`Iterable`, ``__iter__``, and ``__gt__``, ``__ge__``, ``__and__``, ``__or__``
58 :class:`Container` ``__contains__`` ``__sub__``, ``__xor__``, and ``isdisjoint``
59
60:class:`MutableSet` :class:`Set` ``add`` and Inherited Set methods and
61 ``discard`` ``clear``, ``pop``, ``remove``, ``__ior__``,
62 ``__iand__``, ``__ixor__``, and ``__isub__``
63
64:class:`Mapping` :class:`Sized`, ``__getitem__``, ``__contains__``, ``keys``, ``items``, ``values``,
65 :class:`Iterable`, ``__len__``. and ``get``, ``__eq__``, and ``__ne__``
66 :class:`Container` ``__iter__``
67
68:class:`MutableMapping` :class:`Mapping` ``__getitem__`` Inherited Mapping methods and
69 ``__setitem__``, ``pop``, ``popitem``, ``clear``, ``update``,
70 ``__delitem__``, and ``setdefault``
71 ``__iter__``, and
72 ``__len__``
73
74:class:`MappingView` :class:`Sized` ``__len__``
75:class:`KeysView` :class:`MappingView`, ``__contains__``,
76 :class:`Set` ``__iter__``
77:class:`ItemsView` :class:`MappingView`, ``__contains__``,
78 :class:`Set` ``__iter__``
79:class:`ValuesView` :class:`MappingView` ``__contains__``, ``__iter__``
80========================= ===================== ====================== ====================================================
Mark Summerfield08898b42007-09-05 08:43:04 +000081
Mark Summerfield08898b42007-09-05 08:43:04 +000082These ABCs allow us to ask classes or instances if they provide
83particular functionality, for example::
84
Mark Summerfield08898b42007-09-05 08:43:04 +000085 size = None
Raymond Hettingerebcee3f2008-02-06 19:54:00 +000086 if isinstance(myvar, collections.Sized):
Mark Summerfield08898b42007-09-05 08:43:04 +000087 size = len(myvar)
88
Raymond Hettingerebcee3f2008-02-06 19:54:00 +000089Several of the ABCs are also useful as mixins that make it easier to develop
90classes supporting container APIs. For example, to write a class supporting
91the full :class:`Set` API, it only necessary to supply the three underlying
92abstract methods: :meth:`__contains__`, :meth:`__iter__`, and :meth:`__len__`.
93The ABC supplies the remaining methods such as :meth:`__and__` and
94:meth:`isdisjoint` ::
95
96 class ListBasedSet(collections.Set):
Raymond Hettingerc1b6a4a2008-02-08 23:46:23 +000097 ''' Alternate set implementation favoring space over speed
98 and not requiring the set elements to be hashable. '''
Raymond Hettingerebcee3f2008-02-06 19:54:00 +000099 def __init__(self, iterable):
Raymond Hettingerc1b6a4a2008-02-08 23:46:23 +0000100 self.elements = lst = []
101 for value in iterable:
102 if value not in lst:
103 lst.append(value)
Raymond Hettingerebcee3f2008-02-06 19:54:00 +0000104 def __iter__(self):
105 return iter(self.elements)
106 def __contains__(self, value):
107 return value in self.elements
108 def __len__(self):
109 return len(self.elements)
110
111 s1 = ListBasedSet('abcdef')
112 s2 = ListBasedSet('defghi')
113 overlap = s1 & s2 # The __and__() method is supported automatically
114
Raymond Hettinger7aebb642008-02-09 03:25:08 +0000115Notes on using :class:`Set` and :class:`MutableSet` as a mixin:
116
Christian Heimesfe337bf2008-03-23 21:54:12 +0000117(1)
Raymond Hettinger7aebb642008-02-09 03:25:08 +0000118 Since some set operations create new sets, the default mixin methods need
Christian Heimesfe337bf2008-03-23 21:54:12 +0000119 a way to create new instances from an iterable. The class constructor is
120 assumed to have a signature in the form ``ClassName(iterable)``.
Benjamin Peterson2b7411d2008-05-26 17:36:47 +0000121 That assumption is factored-out to an internal classmethod called
Raymond Hettinger7aebb642008-02-09 03:25:08 +0000122 :meth:`_from_iterable` which calls ``cls(iterable)`` to produce a new set.
123 If the :class:`Set` mixin is being used in a class with a different
Christian Heimesfe337bf2008-03-23 21:54:12 +0000124 constructor signature, you will need to override :meth:`from_iterable`
125 with a classmethod that can construct new instances from
Raymond Hettinger7aebb642008-02-09 03:25:08 +0000126 an iterable argument.
127
128(2)
129 To override the comparisons (presumably for speed, as the
130 semantics are fixed), redefine :meth:`__le__` and
131 then the other operations will automatically follow suit.
Raymond Hettingerebcee3f2008-02-06 19:54:00 +0000132
Raymond Hettinger0dbdab22008-02-09 03:48:16 +0000133(3)
134 The :class:`Set` mixin provides a :meth:`_hash` method to compute a hash value
135 for the set; however, :meth:`__hash__` is not defined because not all sets
136 are hashable or immutable. To add set hashabilty using mixins,
137 inherit from both :meth:`Set` and :meth:`Hashable`, then define
138 ``__hash__ = Set._hash``.
139
Mark Summerfield08898b42007-09-05 08:43:04 +0000140(For more about ABCs, see the :mod:`abc` module and :pep:`3119`.)
141
142
Georg Brandl116aa622007-08-15 14:28:22 +0000143.. _deque-objects:
144
145:class:`deque` objects
146----------------------
147
148
Georg Brandl9afde1c2007-11-01 20:32:30 +0000149.. class:: deque([iterable[, maxlen]])
Georg Brandl116aa622007-08-15 14:28:22 +0000150
151 Returns a new deque object initialized left-to-right (using :meth:`append`) with
152 data from *iterable*. If *iterable* is not specified, the new deque is empty.
153
154 Deques are a generalization of stacks and queues (the name is pronounced "deck"
155 and is short for "double-ended queue"). Deques support thread-safe, memory
156 efficient appends and pops from either side of the deque with approximately the
157 same O(1) performance in either direction.
158
159 Though :class:`list` objects support similar operations, they are optimized for
160 fast fixed-length operations and incur O(n) memory movement costs for
161 ``pop(0)`` and ``insert(0, v)`` operations which change both the size and
162 position of the underlying data representation.
163
Georg Brandl116aa622007-08-15 14:28:22 +0000164
Georg Brandl9afde1c2007-11-01 20:32:30 +0000165 If *maxlen* is not specified or is *None*, deques may grow to an
166 arbitrary length. Otherwise, the deque is bounded to the specified maximum
167 length. Once a bounded length deque is full, when new items are added, a
168 corresponding number of items are discarded from the opposite end. Bounded
169 length deques provide functionality similar to the ``tail`` filter in
170 Unix. They are also useful for tracking transactions and other pools of data
171 where only the most recent activity is of interest.
172
Georg Brandl9afde1c2007-11-01 20:32:30 +0000173
Benjamin Petersone41251e2008-04-25 01:59:09 +0000174 Deque objects support the following methods:
Georg Brandl116aa622007-08-15 14:28:22 +0000175
Benjamin Petersone41251e2008-04-25 01:59:09 +0000176 .. method:: append(x)
Georg Brandl116aa622007-08-15 14:28:22 +0000177
Benjamin Petersone41251e2008-04-25 01:59:09 +0000178 Add *x* to the right side of the deque.
Georg Brandl116aa622007-08-15 14:28:22 +0000179
180
Benjamin Petersone41251e2008-04-25 01:59:09 +0000181 .. method:: appendleft(x)
Georg Brandl116aa622007-08-15 14:28:22 +0000182
Benjamin Petersone41251e2008-04-25 01:59:09 +0000183 Add *x* to the left side of the deque.
Georg Brandl116aa622007-08-15 14:28:22 +0000184
185
Benjamin Petersone41251e2008-04-25 01:59:09 +0000186 .. method:: clear()
Georg Brandl116aa622007-08-15 14:28:22 +0000187
Benjamin Petersone41251e2008-04-25 01:59:09 +0000188 Remove all elements from the deque leaving it with length 0.
Georg Brandl116aa622007-08-15 14:28:22 +0000189
190
Benjamin Petersone41251e2008-04-25 01:59:09 +0000191 .. method:: extend(iterable)
Georg Brandl116aa622007-08-15 14:28:22 +0000192
Benjamin Petersone41251e2008-04-25 01:59:09 +0000193 Extend the right side of the deque by appending elements from the iterable
194 argument.
Georg Brandl116aa622007-08-15 14:28:22 +0000195
196
Benjamin Petersone41251e2008-04-25 01:59:09 +0000197 .. method:: extendleft(iterable)
Georg Brandl116aa622007-08-15 14:28:22 +0000198
Benjamin Petersone41251e2008-04-25 01:59:09 +0000199 Extend the left side of the deque by appending elements from *iterable*.
200 Note, the series of left appends results in reversing the order of
201 elements in the iterable argument.
Georg Brandl116aa622007-08-15 14:28:22 +0000202
203
Benjamin Petersone41251e2008-04-25 01:59:09 +0000204 .. method:: pop()
Georg Brandl116aa622007-08-15 14:28:22 +0000205
Benjamin Petersone41251e2008-04-25 01:59:09 +0000206 Remove and return an element from the right side of the deque. If no
207 elements are present, raises an :exc:`IndexError`.
Georg Brandl116aa622007-08-15 14:28:22 +0000208
209
Benjamin Petersone41251e2008-04-25 01:59:09 +0000210 .. method:: popleft()
Georg Brandl116aa622007-08-15 14:28:22 +0000211
Benjamin Petersone41251e2008-04-25 01:59:09 +0000212 Remove and return an element from the left side of the deque. If no
213 elements are present, raises an :exc:`IndexError`.
Georg Brandl116aa622007-08-15 14:28:22 +0000214
215
Benjamin Petersone41251e2008-04-25 01:59:09 +0000216 .. method:: remove(value)
Georg Brandl116aa622007-08-15 14:28:22 +0000217
Benjamin Petersone41251e2008-04-25 01:59:09 +0000218 Removed the first occurrence of *value*. If not found, raises a
219 :exc:`ValueError`.
Georg Brandl116aa622007-08-15 14:28:22 +0000220
Georg Brandl116aa622007-08-15 14:28:22 +0000221
Benjamin Petersone41251e2008-04-25 01:59:09 +0000222 .. method:: rotate(n)
Georg Brandl116aa622007-08-15 14:28:22 +0000223
Benjamin Petersone41251e2008-04-25 01:59:09 +0000224 Rotate the deque *n* steps to the right. If *n* is negative, rotate to
225 the left. Rotating one step to the right is equivalent to:
226 ``d.appendleft(d.pop())``.
227
Georg Brandl116aa622007-08-15 14:28:22 +0000228
229In addition to the above, deques support iteration, pickling, ``len(d)``,
230``reversed(d)``, ``copy.copy(d)``, ``copy.deepcopy(d)``, membership testing with
Benjamin Peterson206e3072008-10-19 14:07:49 +0000231the :keyword:`in` operator, and subscript references such as ``d[-1]``. Indexed
232access is O(1) at both ends but slows to O(n) in the middle. For fast random
233access, use lists instead.
Georg Brandl116aa622007-08-15 14:28:22 +0000234
Christian Heimesfe337bf2008-03-23 21:54:12 +0000235Example:
236
237.. doctest::
Georg Brandl116aa622007-08-15 14:28:22 +0000238
239 >>> from collections import deque
240 >>> d = deque('ghi') # make a new deque with three items
241 >>> for elem in d: # iterate over the deque's elements
Neal Norwitz752abd02008-05-13 04:55:24 +0000242 ... print(elem.upper())
Georg Brandl116aa622007-08-15 14:28:22 +0000243 G
244 H
245 I
246
247 >>> d.append('j') # add a new entry to the right side
248 >>> d.appendleft('f') # add a new entry to the left side
249 >>> d # show the representation of the deque
250 deque(['f', 'g', 'h', 'i', 'j'])
251
252 >>> d.pop() # return and remove the rightmost item
253 'j'
254 >>> d.popleft() # return and remove the leftmost item
255 'f'
256 >>> list(d) # list the contents of the deque
257 ['g', 'h', 'i']
258 >>> d[0] # peek at leftmost item
259 'g'
260 >>> d[-1] # peek at rightmost item
261 'i'
262
263 >>> list(reversed(d)) # list the contents of a deque in reverse
264 ['i', 'h', 'g']
265 >>> 'h' in d # search the deque
266 True
267 >>> d.extend('jkl') # add multiple elements at once
268 >>> d
269 deque(['g', 'h', 'i', 'j', 'k', 'l'])
270 >>> d.rotate(1) # right rotation
271 >>> d
272 deque(['l', 'g', 'h', 'i', 'j', 'k'])
273 >>> d.rotate(-1) # left rotation
274 >>> d
275 deque(['g', 'h', 'i', 'j', 'k', 'l'])
276
277 >>> deque(reversed(d)) # make a new deque in reverse order
278 deque(['l', 'k', 'j', 'i', 'h', 'g'])
279 >>> d.clear() # empty the deque
280 >>> d.pop() # cannot pop from an empty deque
281 Traceback (most recent call last):
282 File "<pyshell#6>", line 1, in -toplevel-
283 d.pop()
284 IndexError: pop from an empty deque
285
286 >>> d.extendleft('abc') # extendleft() reverses the input order
287 >>> d
288 deque(['c', 'b', 'a'])
289
290
291.. _deque-recipes:
292
Georg Brandl9afde1c2007-11-01 20:32:30 +0000293:class:`deque` Recipes
294^^^^^^^^^^^^^^^^^^^^^^
Georg Brandl116aa622007-08-15 14:28:22 +0000295
296This section shows various approaches to working with deques.
297
298The :meth:`rotate` method provides a way to implement :class:`deque` slicing and
299deletion. For example, a pure python implementation of ``del d[n]`` relies on
300the :meth:`rotate` method to position elements to be popped::
301
302 def delete_nth(d, n):
303 d.rotate(-n)
304 d.popleft()
305 d.rotate(n)
306
307To implement :class:`deque` slicing, use a similar approach applying
308:meth:`rotate` to bring a target element to the left side of the deque. Remove
309old entries with :meth:`popleft`, add new entries with :meth:`extend`, and then
310reverse the rotation.
Georg Brandl116aa622007-08-15 14:28:22 +0000311With minor variations on that approach, it is easy to implement Forth style
312stack manipulations such as ``dup``, ``drop``, ``swap``, ``over``, ``pick``,
313``rot``, and ``roll``.
314
Georg Brandl116aa622007-08-15 14:28:22 +0000315Multi-pass data reduction algorithms can be succinctly expressed and efficiently
316coded by extracting elements with multiple calls to :meth:`popleft`, applying
Georg Brandl9afde1c2007-11-01 20:32:30 +0000317a reduction function, and calling :meth:`append` to add the result back to the
318deque.
Georg Brandl116aa622007-08-15 14:28:22 +0000319
320For example, building a balanced binary tree of nested lists entails reducing
Christian Heimesfe337bf2008-03-23 21:54:12 +0000321two adjacent nodes into one by grouping them in a list:
Georg Brandl116aa622007-08-15 14:28:22 +0000322
323 >>> def maketree(iterable):
324 ... d = deque(iterable)
325 ... while len(d) > 1:
326 ... pair = [d.popleft(), d.popleft()]
327 ... d.append(pair)
328 ... return list(d)
329 ...
Georg Brandl6911e3c2007-09-04 07:15:32 +0000330 >>> print(maketree('abcdefgh'))
Georg Brandl116aa622007-08-15 14:28:22 +0000331 [[[['a', 'b'], ['c', 'd']], [['e', 'f'], ['g', 'h']]]]
332
Georg Brandl9afde1c2007-11-01 20:32:30 +0000333Bounded length deques provide functionality similar to the ``tail`` filter
334in Unix::
Georg Brandl116aa622007-08-15 14:28:22 +0000335
Georg Brandl9afde1c2007-11-01 20:32:30 +0000336 def tail(filename, n=10):
337 'Return the last n lines of a file'
338 return deque(open(filename), n)
Georg Brandl116aa622007-08-15 14:28:22 +0000339
340.. _defaultdict-objects:
341
342:class:`defaultdict` objects
343----------------------------
344
345
346.. class:: defaultdict([default_factory[, ...]])
347
348 Returns a new dictionary-like object. :class:`defaultdict` is a subclass of the
349 builtin :class:`dict` class. It overrides one method and adds one writable
350 instance variable. The remaining functionality is the same as for the
351 :class:`dict` class and is not documented here.
352
353 The first argument provides the initial value for the :attr:`default_factory`
354 attribute; it defaults to ``None``. All remaining arguments are treated the same
355 as if they were passed to the :class:`dict` constructor, including keyword
356 arguments.
357
Georg Brandl116aa622007-08-15 14:28:22 +0000358
Benjamin Petersone41251e2008-04-25 01:59:09 +0000359 :class:`defaultdict` objects support the following method in addition to the
360 standard :class:`dict` operations:
Georg Brandl116aa622007-08-15 14:28:22 +0000361
Benjamin Petersone41251e2008-04-25 01:59:09 +0000362 .. method:: defaultdict.__missing__(key)
Georg Brandl116aa622007-08-15 14:28:22 +0000363
Benjamin Peterson5478b472008-09-17 22:25:09 +0000364 If the :attr:`default_factory` attribute is ``None``, this raises a
Benjamin Petersone41251e2008-04-25 01:59:09 +0000365 :exc:`KeyError` exception with the *key* as argument.
Georg Brandl116aa622007-08-15 14:28:22 +0000366
Benjamin Petersone41251e2008-04-25 01:59:09 +0000367 If :attr:`default_factory` is not ``None``, it is called without arguments
368 to provide a default value for the given *key*, this value is inserted in
369 the dictionary for the *key*, and returned.
Georg Brandl116aa622007-08-15 14:28:22 +0000370
Benjamin Petersone41251e2008-04-25 01:59:09 +0000371 If calling :attr:`default_factory` raises an exception this exception is
372 propagated unchanged.
Georg Brandl116aa622007-08-15 14:28:22 +0000373
Benjamin Petersone41251e2008-04-25 01:59:09 +0000374 This method is called by the :meth:`__getitem__` method of the
375 :class:`dict` class when the requested key is not found; whatever it
376 returns or raises is then returned or raised by :meth:`__getitem__`.
Georg Brandl116aa622007-08-15 14:28:22 +0000377
378
Benjamin Petersone41251e2008-04-25 01:59:09 +0000379 :class:`defaultdict` objects support the following instance variable:
Georg Brandl116aa622007-08-15 14:28:22 +0000380
Benjamin Petersone41251e2008-04-25 01:59:09 +0000381
382 .. attribute:: defaultdict.default_factory
383
384 This attribute is used by the :meth:`__missing__` method; it is
385 initialized from the first argument to the constructor, if present, or to
386 ``None``, if absent.
Georg Brandl116aa622007-08-15 14:28:22 +0000387
388
389.. _defaultdict-examples:
390
391:class:`defaultdict` Examples
392^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
393
394Using :class:`list` as the :attr:`default_factory`, it is easy to group a
Christian Heimesfe337bf2008-03-23 21:54:12 +0000395sequence of key-value pairs into a dictionary of lists:
Georg Brandl116aa622007-08-15 14:28:22 +0000396
397 >>> s = [('yellow', 1), ('blue', 2), ('yellow', 3), ('blue', 4), ('red', 1)]
398 >>> d = defaultdict(list)
399 >>> for k, v in s:
400 ... d[k].append(v)
401 ...
402 >>> d.items()
403 [('blue', [2, 4]), ('red', [1]), ('yellow', [1, 3])]
404
405When each key is encountered for the first time, it is not already in the
406mapping; so an entry is automatically created using the :attr:`default_factory`
407function which returns an empty :class:`list`. The :meth:`list.append`
408operation then attaches the value to the new list. When keys are encountered
409again, the look-up proceeds normally (returning the list for that key) and the
410:meth:`list.append` operation adds another value to the list. This technique is
Christian Heimesfe337bf2008-03-23 21:54:12 +0000411simpler and faster than an equivalent technique using :meth:`dict.setdefault`:
Georg Brandl116aa622007-08-15 14:28:22 +0000412
413 >>> d = {}
414 >>> for k, v in s:
415 ... d.setdefault(k, []).append(v)
416 ...
417 >>> d.items()
418 [('blue', [2, 4]), ('red', [1]), ('yellow', [1, 3])]
419
420Setting the :attr:`default_factory` to :class:`int` makes the
421:class:`defaultdict` useful for counting (like a bag or multiset in other
Christian Heimesfe337bf2008-03-23 21:54:12 +0000422languages):
Georg Brandl116aa622007-08-15 14:28:22 +0000423
424 >>> s = 'mississippi'
425 >>> d = defaultdict(int)
426 >>> for k in s:
427 ... d[k] += 1
428 ...
429 >>> d.items()
430 [('i', 4), ('p', 2), ('s', 4), ('m', 1)]
431
432When a letter is first encountered, it is missing from the mapping, so the
433:attr:`default_factory` function calls :func:`int` to supply a default count of
434zero. The increment operation then builds up the count for each letter.
435
436The function :func:`int` which always returns zero is just a special case of
437constant functions. A faster and more flexible way to create constant functions
438is to use a lambda function which can supply any constant value (not just
Christian Heimesfe337bf2008-03-23 21:54:12 +0000439zero):
Georg Brandl116aa622007-08-15 14:28:22 +0000440
441 >>> def constant_factory(value):
442 ... return lambda: value
443 >>> d = defaultdict(constant_factory('<missing>'))
444 >>> d.update(name='John', action='ran')
445 >>> '%(name)s %(action)s to %(object)s' % d
446 'John ran to <missing>'
447
448Setting the :attr:`default_factory` to :class:`set` makes the
Christian Heimesfe337bf2008-03-23 21:54:12 +0000449:class:`defaultdict` useful for building a dictionary of sets:
Georg Brandl116aa622007-08-15 14:28:22 +0000450
451 >>> s = [('red', 1), ('blue', 2), ('red', 3), ('blue', 4), ('red', 1), ('blue', 4)]
452 >>> d = defaultdict(set)
453 >>> for k, v in s:
454 ... d[k].add(v)
455 ...
456 >>> d.items()
457 [('blue', set([2, 4])), ('red', set([1, 3]))]
458
459
460.. _named-tuple-factory:
461
Guido van Rossum3d392eb2007-11-16 00:35:22 +0000462:func:`namedtuple` Factory Function for Tuples with Named Fields
Christian Heimes790c8232008-01-07 21:14:23 +0000463----------------------------------------------------------------
Georg Brandl116aa622007-08-15 14:28:22 +0000464
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000465Named tuples assign meaning to each position in a tuple and allow for more readable,
466self-documenting code. They can be used wherever regular tuples are used, and
467they add the ability to access fields by name instead of position index.
Georg Brandl116aa622007-08-15 14:28:22 +0000468
Benjamin Peterson4469d0c2008-11-30 22:46:23 +0000469.. function:: namedtuple(typename, field_names, [verbose])
Georg Brandl116aa622007-08-15 14:28:22 +0000470
471 Returns a new tuple subclass named *typename*. The new subclass is used to
Christian Heimesc3f30c42008-02-22 16:37:40 +0000472 create tuple-like objects that have fields accessible by attribute lookup as
Georg Brandl116aa622007-08-15 14:28:22 +0000473 well as being indexable and iterable. Instances of the subclass also have a
Benjamin Peterson4469d0c2008-11-30 22:46:23 +0000474 helpful docstring (with typename and field_names) and a helpful :meth:`__repr__`
Georg Brandl116aa622007-08-15 14:28:22 +0000475 method which lists the tuple contents in a ``name=value`` format.
476
Benjamin Peterson4469d0c2008-11-30 22:46:23 +0000477 The *field_names* are a single string with each fieldname separated by whitespace
478 and/or commas, for example ``'x y'`` or ``'x, y'``. Alternatively, *field_names*
Christian Heimes25bb7832008-01-11 16:17:00 +0000479 can be a sequence of strings such as ``['x', 'y']``.
Georg Brandl9afde1c2007-11-01 20:32:30 +0000480
481 Any valid Python identifier may be used for a fieldname except for names
Christian Heimes0449f632007-12-15 01:27:15 +0000482 starting with an underscore. Valid identifiers consist of letters, digits,
483 and underscores but do not start with a digit or underscore and cannot be
Georg Brandlf6945182008-02-01 11:56:49 +0000484 a :mod:`keyword` such as *class*, *for*, *return*, *global*, *pass*,
Georg Brandl9afde1c2007-11-01 20:32:30 +0000485 or *raise*.
Georg Brandl116aa622007-08-15 14:28:22 +0000486
Christian Heimes25bb7832008-01-11 16:17:00 +0000487 If *verbose* is true, the class definition is printed just before being built.
Georg Brandl116aa622007-08-15 14:28:22 +0000488
Georg Brandl9afde1c2007-11-01 20:32:30 +0000489 Named tuple instances do not have per-instance dictionaries, so they are
Thomas Wouters8ce81f72007-09-20 18:22:40 +0000490 lightweight and require no more memory than regular tuples.
Georg Brandl116aa622007-08-15 14:28:22 +0000491
Christian Heimesfe337bf2008-03-23 21:54:12 +0000492Example:
493
494.. doctest::
495 :options: +NORMALIZE_WHITESPACE
Georg Brandl116aa622007-08-15 14:28:22 +0000496
Guido van Rossum3d392eb2007-11-16 00:35:22 +0000497 >>> Point = namedtuple('Point', 'x y', verbose=True)
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000498 class Point(tuple):
499 'Point(x, y)'
Christian Heimesfe337bf2008-03-23 21:54:12 +0000500 <BLANKLINE>
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000501 __slots__ = ()
Christian Heimesfe337bf2008-03-23 21:54:12 +0000502 <BLANKLINE>
Christian Heimesfaf2f632008-01-06 16:59:19 +0000503 _fields = ('x', 'y')
Christian Heimesfe337bf2008-03-23 21:54:12 +0000504 <BLANKLINE>
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000505 def __new__(cls, x, y):
506 return tuple.__new__(cls, (x, y))
Christian Heimesfe337bf2008-03-23 21:54:12 +0000507 <BLANKLINE>
Christian Heimesfaf2f632008-01-06 16:59:19 +0000508 @classmethod
Christian Heimesfe337bf2008-03-23 21:54:12 +0000509 def _make(cls, iterable, new=tuple.__new__, len=len):
Christian Heimesfaf2f632008-01-06 16:59:19 +0000510 'Make a new Point object from a sequence or iterable'
Christian Heimesfe337bf2008-03-23 21:54:12 +0000511 result = new(cls, iterable)
Christian Heimesfaf2f632008-01-06 16:59:19 +0000512 if len(result) != 2:
513 raise TypeError('Expected 2 arguments, got %d' % len(result))
514 return result
Christian Heimesfe337bf2008-03-23 21:54:12 +0000515 <BLANKLINE>
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000516 def __repr__(self):
517 return 'Point(x=%r, y=%r)' % self
Christian Heimesfe337bf2008-03-23 21:54:12 +0000518 <BLANKLINE>
Christian Heimes99170a52007-12-19 02:07:34 +0000519 def _asdict(t):
Christian Heimes0449f632007-12-15 01:27:15 +0000520 'Return a new dict which maps field names to their values'
Christian Heimes99170a52007-12-19 02:07:34 +0000521 return {'x': t[0], 'y': t[1]}
Christian Heimesfe337bf2008-03-23 21:54:12 +0000522 <BLANKLINE>
Christian Heimes0449f632007-12-15 01:27:15 +0000523 def _replace(self, **kwds):
Guido van Rossum3d392eb2007-11-16 00:35:22 +0000524 'Return a new Point object replacing specified fields with new values'
Christian Heimesfaf2f632008-01-06 16:59:19 +0000525 result = self._make(map(kwds.pop, ('x', 'y'), self))
526 if kwds:
527 raise ValueError('Got unexpected field names: %r' % kwds.keys())
528 return result
Georg Brandlc28e1fa2008-06-10 19:20:26 +0000529 <BLANKLINE>
Benjamin Peterson41181742008-07-02 20:22:54 +0000530 def __getnewargs__(self):
531 return tuple(self)
Christian Heimesfe337bf2008-03-23 21:54:12 +0000532 <BLANKLINE>
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000533 x = property(itemgetter(0))
534 y = property(itemgetter(1))
Georg Brandl116aa622007-08-15 14:28:22 +0000535
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000536 >>> p = Point(11, y=22) # instantiate with positional or keyword arguments
Christian Heimes99170a52007-12-19 02:07:34 +0000537 >>> p[0] + p[1] # indexable like the plain tuple (11, 22)
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000538 33
539 >>> x, y = p # unpack like a regular tuple
540 >>> x, y
541 (11, 22)
Christian Heimesc3f30c42008-02-22 16:37:40 +0000542 >>> p.x + p.y # fields also accessible by name
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000543 33
544 >>> p # readable __repr__ with a name=value style
545 Point(x=11, y=22)
Georg Brandl116aa622007-08-15 14:28:22 +0000546
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000547Named tuples are especially useful for assigning field names to result tuples returned
548by the :mod:`csv` or :mod:`sqlite3` modules::
549
Guido van Rossum3d392eb2007-11-16 00:35:22 +0000550 EmployeeRecord = namedtuple('EmployeeRecord', 'name, age, title, department, paygrade')
Georg Brandl9afde1c2007-11-01 20:32:30 +0000551
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000552 import csv
Christian Heimesfaf2f632008-01-06 16:59:19 +0000553 for emp in map(EmployeeRecord._make, csv.reader(open("employees.csv", "rb"))):
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000554 print(emp.name, emp.title)
555
Georg Brandl9afde1c2007-11-01 20:32:30 +0000556 import sqlite3
557 conn = sqlite3.connect('/companydata')
558 cursor = conn.cursor()
559 cursor.execute('SELECT name, age, title, department, paygrade FROM employees')
Christian Heimesfaf2f632008-01-06 16:59:19 +0000560 for emp in map(EmployeeRecord._make, cursor.fetchall()):
Christian Heimes00412232008-01-10 16:02:19 +0000561 print(emp.name, emp.title)
Georg Brandl9afde1c2007-11-01 20:32:30 +0000562
Christian Heimes99170a52007-12-19 02:07:34 +0000563In addition to the methods inherited from tuples, named tuples support
Christian Heimes2380ac72008-01-09 00:17:24 +0000564three additional methods and one attribute. To prevent conflicts with
565field names, the method and attribute names start with an underscore.
Christian Heimes99170a52007-12-19 02:07:34 +0000566
Christian Heimes790c8232008-01-07 21:14:23 +0000567.. method:: somenamedtuple._make(iterable)
Christian Heimes99170a52007-12-19 02:07:34 +0000568
Christian Heimesfaf2f632008-01-06 16:59:19 +0000569 Class method that makes a new instance from an existing sequence or iterable.
Christian Heimes99170a52007-12-19 02:07:34 +0000570
Christian Heimesfe337bf2008-03-23 21:54:12 +0000571.. doctest::
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000572
Christian Heimesfaf2f632008-01-06 16:59:19 +0000573 >>> t = [11, 22]
574 >>> Point._make(t)
575 Point(x=11, y=22)
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000576
Christian Heimes790c8232008-01-07 21:14:23 +0000577.. method:: somenamedtuple._asdict()
Georg Brandl9afde1c2007-11-01 20:32:30 +0000578
Christian Heimesfe337bf2008-03-23 21:54:12 +0000579 Return a new dict which maps field names to their corresponding values::
Georg Brandl9afde1c2007-11-01 20:32:30 +0000580
Christian Heimes0449f632007-12-15 01:27:15 +0000581 >>> p._asdict()
Georg Brandl9afde1c2007-11-01 20:32:30 +0000582 {'x': 11, 'y': 22}
Christian Heimesfe337bf2008-03-23 21:54:12 +0000583
Christian Heimes790c8232008-01-07 21:14:23 +0000584.. method:: somenamedtuple._replace(kwargs)
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000585
Christian Heimesfe337bf2008-03-23 21:54:12 +0000586 Return a new instance of the named tuple replacing specified fields with new
587 values:
Thomas Wouters8ce81f72007-09-20 18:22:40 +0000588
589::
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000590
591 >>> p = Point(x=11, y=22)
Christian Heimes0449f632007-12-15 01:27:15 +0000592 >>> p._replace(x=33)
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000593 Point(x=33, y=22)
594
Guido van Rossum3d392eb2007-11-16 00:35:22 +0000595 >>> for partnum, record in inventory.items():
Christian Heimes454f37b2008-01-10 00:10:02 +0000596 ... inventory[partnum] = record._replace(price=newprices[partnum], timestamp=time.now())
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000597
Christian Heimes790c8232008-01-07 21:14:23 +0000598.. attribute:: somenamedtuple._fields
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000599
Christian Heimes2380ac72008-01-09 00:17:24 +0000600 Tuple of strings listing the field names. Useful for introspection
Georg Brandl9afde1c2007-11-01 20:32:30 +0000601 and for creating new named tuple types from existing named tuples.
Thomas Wouters8ce81f72007-09-20 18:22:40 +0000602
Christian Heimesfe337bf2008-03-23 21:54:12 +0000603.. doctest::
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000604
Christian Heimes0449f632007-12-15 01:27:15 +0000605 >>> p._fields # view the field names
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000606 ('x', 'y')
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000607
Guido van Rossum3d392eb2007-11-16 00:35:22 +0000608 >>> Color = namedtuple('Color', 'red green blue')
Christian Heimes0449f632007-12-15 01:27:15 +0000609 >>> Pixel = namedtuple('Pixel', Point._fields + Color._fields)
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000610 >>> Pixel(11, 22, 128, 255, 0)
Christian Heimes454f37b2008-01-10 00:10:02 +0000611 Pixel(x=11, y=22, red=128, green=255, blue=0)
Georg Brandl116aa622007-08-15 14:28:22 +0000612
Christian Heimes0449f632007-12-15 01:27:15 +0000613To retrieve a field whose name is stored in a string, use the :func:`getattr`
Christian Heimesfe337bf2008-03-23 21:54:12 +0000614function:
Christian Heimes0449f632007-12-15 01:27:15 +0000615
616 >>> getattr(p, 'x')
617 11
618
Christian Heimesfe337bf2008-03-23 21:54:12 +0000619To convert a dictionary to a named tuple, use the double-star-operator [#]_:
Christian Heimes99170a52007-12-19 02:07:34 +0000620
621 >>> d = {'x': 11, 'y': 22}
622 >>> Point(**d)
623 Point(x=11, y=22)
624
Guido van Rossum3d392eb2007-11-16 00:35:22 +0000625Since a named tuple is a regular Python class, it is easy to add or change
Christian Heimes043d6f62008-01-07 17:19:16 +0000626functionality with a subclass. Here is how to add a calculated field and
Christian Heimesfe337bf2008-03-23 21:54:12 +0000627a fixed-width print format:
Guido van Rossum3d392eb2007-11-16 00:35:22 +0000628
Christian Heimes043d6f62008-01-07 17:19:16 +0000629 >>> class Point(namedtuple('Point', 'x y')):
Christian Heimes25bb7832008-01-11 16:17:00 +0000630 ... __slots__ = ()
Christian Heimes454f37b2008-01-10 00:10:02 +0000631 ... @property
632 ... def hypot(self):
633 ... return (self.x ** 2 + self.y ** 2) ** 0.5
634 ... def __str__(self):
Christian Heimes25bb7832008-01-11 16:17:00 +0000635 ... return 'Point: x=%6.3f y=%6.3f hypot=%6.3f' % (self.x, self.y, self.hypot)
Guido van Rossum3d392eb2007-11-16 00:35:22 +0000636
Georg Brandl0df79792008-10-04 18:33:26 +0000637 >>> for p in Point(3, 4), Point(14, 5/7):
Christian Heimes00412232008-01-10 16:02:19 +0000638 ... print(p)
Christian Heimes25bb7832008-01-11 16:17:00 +0000639 Point: x= 3.000 y= 4.000 hypot= 5.000
640 Point: x=14.000 y= 0.714 hypot=14.018
Christian Heimes043d6f62008-01-07 17:19:16 +0000641
Christian Heimesaf98da12008-01-27 15:18:18 +0000642The subclass shown above sets ``__slots__`` to an empty tuple. This keeps
Christian Heimes679db4a2008-01-18 09:56:22 +0000643keep memory requirements low by preventing the creation of instance dictionaries.
644
Christian Heimes2380ac72008-01-09 00:17:24 +0000645
646Subclassing is not useful for adding new, stored fields. Instead, simply
Christian Heimesfe337bf2008-03-23 21:54:12 +0000647create a new named tuple type from the :attr:`_fields` attribute:
Christian Heimes2380ac72008-01-09 00:17:24 +0000648
Christian Heimes25bb7832008-01-11 16:17:00 +0000649 >>> Point3D = namedtuple('Point3D', Point._fields + ('z',))
Christian Heimes2380ac72008-01-09 00:17:24 +0000650
651Default values can be implemented by using :meth:`_replace` to
Christian Heimesfe337bf2008-03-23 21:54:12 +0000652customize a prototype instance:
Guido van Rossum3d392eb2007-11-16 00:35:22 +0000653
654 >>> Account = namedtuple('Account', 'owner balance transaction_count')
Christian Heimes587c2bf2008-01-19 16:21:02 +0000655 >>> default_account = Account('<owner name>', 0.0, 0)
656 >>> johns_account = default_account._replace(owner='John')
Guido van Rossum3d392eb2007-11-16 00:35:22 +0000657
Christian Heimese4ca8152008-05-08 17:18:53 +0000658Enumerated constants can be implemented with named tuples, but it is simpler
659and more efficient to use a simple class declaration:
660
661 >>> Status = namedtuple('Status', 'open pending closed')._make(range(3))
662 >>> Status.open, Status.pending, Status.closed
663 (0, 1, 2)
664 >>> class Status:
665 ... open, pending, closed = range(3)
666
Thomas Wouters47b49bf2007-08-30 22:15:33 +0000667.. rubric:: Footnotes
668
Christian Heimes99170a52007-12-19 02:07:34 +0000669.. [#] For information on the double-star-operator see
Thomas Wouters47b49bf2007-08-30 22:15:33 +0000670 :ref:`tut-unpacking-arguments` and :ref:`calls`.
Raymond Hettingere4c96ad2008-02-06 01:23:58 +0000671
672
673
674:class:`UserDict` objects
Mark Summerfield8f2d0062008-02-06 13:30:44 +0000675-------------------------
Raymond Hettingere4c96ad2008-02-06 01:23:58 +0000676
677The class, :class:`UserDict` acts as a wrapper around dictionary objects.
678The need for this class has been partially supplanted by the ability to
679subclass directly from :class:`dict`; however, this class can be easier
680to work with because the underlying dictionary is accessible as an
681attribute.
682
683.. class:: UserDict([initialdata])
684
685 Class that simulates a dictionary. The instance's contents are kept in a
686 regular dictionary, which is accessible via the :attr:`data` attribute of
687 :class:`UserDict` instances. If *initialdata* is provided, :attr:`data` is
688 initialized with its contents; note that a reference to *initialdata* will not
689 be kept, allowing it be used for other purposes.
690
691In addition to supporting the methods and operations of mappings,
Raymond Hettingerebcee3f2008-02-06 19:54:00 +0000692:class:`UserDict` instances provide the following attribute:
Raymond Hettingere4c96ad2008-02-06 01:23:58 +0000693
694.. attribute:: UserDict.data
695
696 A real dictionary used to store the contents of the :class:`UserDict` class.
Raymond Hettinger53dbe392008-02-12 20:03:09 +0000697
698
699
700:class:`UserList` objects
701-------------------------
702
703This class acts as a wrapper around list objects. It is a useful base class
704for your own list-like classes which can inherit from them and override
705existing methods or add new ones. In this way, one can add new behaviors to
706lists.
707
708The need for this class has been partially supplanted by the ability to
709subclass directly from :class:`list`; however, this class can be easier
710to work with because the underlying list is accessible as an attribute.
711
712.. class:: UserList([list])
713
714 Class that simulates a list. The instance's contents are kept in a regular
715 list, which is accessible via the :attr:`data` attribute of :class:`UserList`
716 instances. The instance's contents are initially set to a copy of *list*,
717 defaulting to the empty list ``[]``. *list* can be any iterable, for
718 example a real Python list or a :class:`UserList` object.
719
720In addition to supporting the methods and operations of mutable sequences,
721:class:`UserList` instances provide the following attribute:
722
723.. attribute:: UserList.data
724
725 A real :class:`list` object used to store the contents of the
726 :class:`UserList` class.
727
728**Subclassing requirements:** Subclasses of :class:`UserList` are expect to
729offer a constructor which can be called with either no arguments or one
730argument. List operations which return a new sequence attempt to create an
731instance of the actual implementation class. To do so, it assumes that the
732constructor can be called with a single parameter, which is a sequence object
733used as a data source.
734
735If a derived class does not wish to comply with this requirement, all of the
736special methods supported by this class will need to be overridden; please
737consult the sources for information about the methods which need to be provided
738in that case.
Raymond Hettingerb3a65f82008-02-21 22:11:37 +0000739
740:class:`UserString` objects
Christian Heimesc3f30c42008-02-22 16:37:40 +0000741---------------------------
Raymond Hettingerb3a65f82008-02-21 22:11:37 +0000742
743The class, :class:`UserString` acts as a wrapper around string objects.
744The need for this class has been partially supplanted by the ability to
745subclass directly from :class:`str`; however, this class can be easier
746to work with because the underlying string is accessible as an
747attribute.
748
749.. class:: UserString([sequence])
750
751 Class that simulates a string or a Unicode string object. The instance's
752 content is kept in a regular string object, which is accessible via the
753 :attr:`data` attribute of :class:`UserString` instances. The instance's
754 contents are initially set to a copy of *sequence*. The *sequence* can
755 be an instance of :class:`bytes`, :class:`str`, :class:`UserString` (or a
756 subclass) or an arbitrary sequence which can be converted into a string using
757 the built-in :func:`str` function.