blob: f41abd670e6323b3af422f34c1a5ef72c1ee364a [file] [log] [blame]
Georg Brandl116aa622007-08-15 14:28:22 +00001
Raymond Hettinger53dbe392008-02-12 20:03:09 +00002:mod:`collections` --- Container datatypes
3==========================================
Georg Brandl116aa622007-08-15 14:28:22 +00004
5.. module:: collections
Raymond Hettinger53dbe392008-02-12 20:03:09 +00006 :synopsis: Container datatypes
Georg Brandl116aa622007-08-15 14:28:22 +00007.. moduleauthor:: Raymond Hettinger <python@rcn.com>
8.. sectionauthor:: Raymond Hettinger <python@rcn.com>
9
Christian Heimesfe337bf2008-03-23 21:54:12 +000010.. testsetup:: *
11
12 from collections import *
13 import itertools
14 __name__ = '<doctest>'
Georg Brandl116aa622007-08-15 14:28:22 +000015
Georg Brandl116aa622007-08-15 14:28:22 +000016This module implements high-performance container datatypes. Currently,
17there are two datatypes, :class:`deque` and :class:`defaultdict`, and
Mark Summerfield71316b02008-02-14 16:28:00 +000018one datatype factory function, :func:`namedtuple`. This module also
19provides the :class:`UserDict` and :class:`UserList` classes which may
20be useful when inheriting directly from :class:`dict` or
21:class:`list` isn't convenient.
Christian Heimes0bd4e112008-02-12 22:59:25 +000022
Raymond Hettingerebcee3f2008-02-06 19:54:00 +000023The specialized containers provided in this module provide alternatives
Christian Heimesfe337bf2008-03-23 21:54:12 +000024to Python's general purpose built-in containers, :class:`dict`,
Raymond Hettingerebcee3f2008-02-06 19:54:00 +000025:class:`list`, :class:`set`, and :class:`tuple`.
Georg Brandl116aa622007-08-15 14:28:22 +000026
Mark Summerfield08898b42007-09-05 08:43:04 +000027In addition to containers, the collections module provides some ABCs
Christian Heimesfe337bf2008-03-23 21:54:12 +000028(abstract base classes) that can be used to test whether a class
Raymond Hettingerebcee3f2008-02-06 19:54:00 +000029provides a particular interface, for example, is it hashable or
Mark Summerfield71316b02008-02-14 16:28:00 +000030a mapping, and some of them can also be used as mixin classes.
Raymond Hettingerebcee3f2008-02-06 19:54:00 +000031
32ABCs - abstract base classes
33----------------------------
34
35The collections module offers the following ABCs:
Mark Summerfield08898b42007-09-05 08:43:04 +000036
Georg Brandl86b2fb92008-07-16 03:43:04 +000037========================= ===================== ====================== ====================================================
38ABC Inherits Abstract Methods Mixin Methods
39========================= ===================== ====================== ====================================================
40:class:`Container` ``__contains__``
41:class:`Hashable` ``__hash__``
42:class:`Iterable` ``__iter__``
43:class:`Iterator` :class:`Iterable` ``__next__`` ``__iter__``
44:class:`Sized` ``__len__``
45:class:`Callable` ``__call__``
Georg Brandl48310cd2009-01-03 21:18:54 +000046
Georg Brandl86b2fb92008-07-16 03:43:04 +000047:class:`Sequence` :class:`Sized`, ``__getitem__`` ``__contains__``. ``__iter__``, ``__reversed__``.
48 :class:`Iterable`, and ``__len__`` ``index``, and ``count``
Georg Brandl48310cd2009-01-03 21:18:54 +000049 :class:`Container`
50
Benjamin Peterson4469d0c2008-11-30 22:46:23 +000051:class:`MutableSequence` :class:`Sequence` ``__getitem__`` Inherited Sequence methods and
Georg Brandl86b2fb92008-07-16 03:43:04 +000052 ``__delitem__``, ``append``, ``reverse``, ``extend``, ``pop``,
53 ``insert``, ``remove``, and ``__iadd__``
54 and ``__len__``
Georg Brandl48310cd2009-01-03 21:18:54 +000055
Georg Brandl86b2fb92008-07-16 03:43:04 +000056:class:`Set` :class:`Sized`, ``__len__``, ``__le__``, ``__lt__``, ``__eq__``, ``__ne__``,
57 :class:`Iterable`, ``__iter__``, and ``__gt__``, ``__ge__``, ``__and__``, ``__or__``
58 :class:`Container` ``__contains__`` ``__sub__``, ``__xor__``, and ``isdisjoint``
Georg Brandl48310cd2009-01-03 21:18:54 +000059
Georg Brandl86b2fb92008-07-16 03:43:04 +000060:class:`MutableSet` :class:`Set` ``add`` and Inherited Set methods and
61 ``discard`` ``clear``, ``pop``, ``remove``, ``__ior__``,
62 ``__iand__``, ``__ixor__``, and ``__isub__``
Georg Brandl48310cd2009-01-03 21:18:54 +000063
Georg Brandl86b2fb92008-07-16 03:43:04 +000064:class:`Mapping` :class:`Sized`, ``__getitem__``, ``__contains__``, ``keys``, ``items``, ``values``,
65 :class:`Iterable`, ``__len__``. and ``get``, ``__eq__``, and ``__ne__``
66 :class:`Container` ``__iter__``
Georg Brandl48310cd2009-01-03 21:18:54 +000067
Georg Brandl86b2fb92008-07-16 03:43:04 +000068:class:`MutableMapping` :class:`Mapping` ``__getitem__`` Inherited Mapping methods and
69 ``__setitem__``, ``pop``, ``popitem``, ``clear``, ``update``,
70 ``__delitem__``, and ``setdefault``
71 ``__iter__``, and
72 ``__len__``
Georg Brandl48310cd2009-01-03 21:18:54 +000073
Georg Brandl86b2fb92008-07-16 03:43:04 +000074:class:`MappingView` :class:`Sized` ``__len__``
75:class:`KeysView` :class:`MappingView`, ``__contains__``,
76 :class:`Set` ``__iter__``
77:class:`ItemsView` :class:`MappingView`, ``__contains__``,
78 :class:`Set` ``__iter__``
79:class:`ValuesView` :class:`MappingView` ``__contains__``, ``__iter__``
80========================= ===================== ====================== ====================================================
Mark Summerfield08898b42007-09-05 08:43:04 +000081
Mark Summerfield08898b42007-09-05 08:43:04 +000082These ABCs allow us to ask classes or instances if they provide
83particular functionality, for example::
84
Mark Summerfield08898b42007-09-05 08:43:04 +000085 size = None
Raymond Hettingerebcee3f2008-02-06 19:54:00 +000086 if isinstance(myvar, collections.Sized):
Mark Summerfield08898b42007-09-05 08:43:04 +000087 size = len(myvar)
88
Raymond Hettingerebcee3f2008-02-06 19:54:00 +000089Several of the ABCs are also useful as mixins that make it easier to develop
90classes supporting container APIs. For example, to write a class supporting
91the full :class:`Set` API, it only necessary to supply the three underlying
92abstract methods: :meth:`__contains__`, :meth:`__iter__`, and :meth:`__len__`.
93The ABC supplies the remaining methods such as :meth:`__and__` and
94:meth:`isdisjoint` ::
95
96 class ListBasedSet(collections.Set):
Raymond Hettingerc1b6a4a2008-02-08 23:46:23 +000097 ''' Alternate set implementation favoring space over speed
98 and not requiring the set elements to be hashable. '''
Raymond Hettingerebcee3f2008-02-06 19:54:00 +000099 def __init__(self, iterable):
Raymond Hettingerc1b6a4a2008-02-08 23:46:23 +0000100 self.elements = lst = []
101 for value in iterable:
102 if value not in lst:
103 lst.append(value)
Raymond Hettingerebcee3f2008-02-06 19:54:00 +0000104 def __iter__(self):
105 return iter(self.elements)
106 def __contains__(self, value):
107 return value in self.elements
108 def __len__(self):
109 return len(self.elements)
110
111 s1 = ListBasedSet('abcdef')
112 s2 = ListBasedSet('defghi')
113 overlap = s1 & s2 # The __and__() method is supported automatically
114
Raymond Hettinger7aebb642008-02-09 03:25:08 +0000115Notes on using :class:`Set` and :class:`MutableSet` as a mixin:
116
Christian Heimesfe337bf2008-03-23 21:54:12 +0000117(1)
Raymond Hettinger7aebb642008-02-09 03:25:08 +0000118 Since some set operations create new sets, the default mixin methods need
Christian Heimesfe337bf2008-03-23 21:54:12 +0000119 a way to create new instances from an iterable. The class constructor is
120 assumed to have a signature in the form ``ClassName(iterable)``.
Benjamin Peterson2b7411d2008-05-26 17:36:47 +0000121 That assumption is factored-out to an internal classmethod called
Raymond Hettinger7aebb642008-02-09 03:25:08 +0000122 :meth:`_from_iterable` which calls ``cls(iterable)`` to produce a new set.
123 If the :class:`Set` mixin is being used in a class with a different
Christian Heimesfe337bf2008-03-23 21:54:12 +0000124 constructor signature, you will need to override :meth:`from_iterable`
125 with a classmethod that can construct new instances from
Raymond Hettinger7aebb642008-02-09 03:25:08 +0000126 an iterable argument.
127
128(2)
129 To override the comparisons (presumably for speed, as the
130 semantics are fixed), redefine :meth:`__le__` and
131 then the other operations will automatically follow suit.
Raymond Hettingerebcee3f2008-02-06 19:54:00 +0000132
Raymond Hettinger0dbdab22008-02-09 03:48:16 +0000133(3)
134 The :class:`Set` mixin provides a :meth:`_hash` method to compute a hash value
135 for the set; however, :meth:`__hash__` is not defined because not all sets
136 are hashable or immutable. To add set hashabilty using mixins,
137 inherit from both :meth:`Set` and :meth:`Hashable`, then define
138 ``__hash__ = Set._hash``.
139
Mark Summerfield08898b42007-09-05 08:43:04 +0000140(For more about ABCs, see the :mod:`abc` module and :pep:`3119`.)
141
142
Georg Brandl116aa622007-08-15 14:28:22 +0000143.. _deque-objects:
144
145:class:`deque` objects
146----------------------
147
148
Georg Brandl9afde1c2007-11-01 20:32:30 +0000149.. class:: deque([iterable[, maxlen]])
Georg Brandl116aa622007-08-15 14:28:22 +0000150
151 Returns a new deque object initialized left-to-right (using :meth:`append`) with
152 data from *iterable*. If *iterable* is not specified, the new deque is empty.
153
154 Deques are a generalization of stacks and queues (the name is pronounced "deck"
155 and is short for "double-ended queue"). Deques support thread-safe, memory
156 efficient appends and pops from either side of the deque with approximately the
157 same O(1) performance in either direction.
158
159 Though :class:`list` objects support similar operations, they are optimized for
160 fast fixed-length operations and incur O(n) memory movement costs for
161 ``pop(0)`` and ``insert(0, v)`` operations which change both the size and
162 position of the underlying data representation.
163
Georg Brandl116aa622007-08-15 14:28:22 +0000164
Georg Brandl9afde1c2007-11-01 20:32:30 +0000165 If *maxlen* is not specified or is *None*, deques may grow to an
166 arbitrary length. Otherwise, the deque is bounded to the specified maximum
167 length. Once a bounded length deque is full, when new items are added, a
168 corresponding number of items are discarded from the opposite end. Bounded
169 length deques provide functionality similar to the ``tail`` filter in
170 Unix. They are also useful for tracking transactions and other pools of data
171 where only the most recent activity is of interest.
172
Georg Brandl9afde1c2007-11-01 20:32:30 +0000173
Benjamin Petersone41251e2008-04-25 01:59:09 +0000174 Deque objects support the following methods:
Georg Brandl116aa622007-08-15 14:28:22 +0000175
Benjamin Petersone41251e2008-04-25 01:59:09 +0000176 .. method:: append(x)
Georg Brandl116aa622007-08-15 14:28:22 +0000177
Benjamin Petersone41251e2008-04-25 01:59:09 +0000178 Add *x* to the right side of the deque.
Georg Brandl116aa622007-08-15 14:28:22 +0000179
180
Benjamin Petersone41251e2008-04-25 01:59:09 +0000181 .. method:: appendleft(x)
Georg Brandl116aa622007-08-15 14:28:22 +0000182
Benjamin Petersone41251e2008-04-25 01:59:09 +0000183 Add *x* to the left side of the deque.
Georg Brandl116aa622007-08-15 14:28:22 +0000184
185
Benjamin Petersone41251e2008-04-25 01:59:09 +0000186 .. method:: clear()
Georg Brandl116aa622007-08-15 14:28:22 +0000187
Benjamin Petersone41251e2008-04-25 01:59:09 +0000188 Remove all elements from the deque leaving it with length 0.
Georg Brandl116aa622007-08-15 14:28:22 +0000189
190
Benjamin Petersone41251e2008-04-25 01:59:09 +0000191 .. method:: extend(iterable)
Georg Brandl116aa622007-08-15 14:28:22 +0000192
Benjamin Petersone41251e2008-04-25 01:59:09 +0000193 Extend the right side of the deque by appending elements from the iterable
194 argument.
Georg Brandl116aa622007-08-15 14:28:22 +0000195
196
Benjamin Petersone41251e2008-04-25 01:59:09 +0000197 .. method:: extendleft(iterable)
Georg Brandl116aa622007-08-15 14:28:22 +0000198
Benjamin Petersone41251e2008-04-25 01:59:09 +0000199 Extend the left side of the deque by appending elements from *iterable*.
200 Note, the series of left appends results in reversing the order of
201 elements in the iterable argument.
Georg Brandl116aa622007-08-15 14:28:22 +0000202
203
Benjamin Petersone41251e2008-04-25 01:59:09 +0000204 .. method:: pop()
Georg Brandl116aa622007-08-15 14:28:22 +0000205
Benjamin Petersone41251e2008-04-25 01:59:09 +0000206 Remove and return an element from the right side of the deque. If no
207 elements are present, raises an :exc:`IndexError`.
Georg Brandl116aa622007-08-15 14:28:22 +0000208
209
Benjamin Petersone41251e2008-04-25 01:59:09 +0000210 .. method:: popleft()
Georg Brandl116aa622007-08-15 14:28:22 +0000211
Benjamin Petersone41251e2008-04-25 01:59:09 +0000212 Remove and return an element from the left side of the deque. If no
213 elements are present, raises an :exc:`IndexError`.
Georg Brandl116aa622007-08-15 14:28:22 +0000214
215
Benjamin Petersone41251e2008-04-25 01:59:09 +0000216 .. method:: remove(value)
Georg Brandl116aa622007-08-15 14:28:22 +0000217
Benjamin Petersone41251e2008-04-25 01:59:09 +0000218 Removed the first occurrence of *value*. If not found, raises a
219 :exc:`ValueError`.
Georg Brandl116aa622007-08-15 14:28:22 +0000220
Georg Brandl116aa622007-08-15 14:28:22 +0000221
Benjamin Petersone41251e2008-04-25 01:59:09 +0000222 .. method:: rotate(n)
Georg Brandl116aa622007-08-15 14:28:22 +0000223
Benjamin Petersone41251e2008-04-25 01:59:09 +0000224 Rotate the deque *n* steps to the right. If *n* is negative, rotate to
225 the left. Rotating one step to the right is equivalent to:
226 ``d.appendleft(d.pop())``.
227
Georg Brandl116aa622007-08-15 14:28:22 +0000228
229In addition to the above, deques support iteration, pickling, ``len(d)``,
230``reversed(d)``, ``copy.copy(d)``, ``copy.deepcopy(d)``, membership testing with
Benjamin Peterson206e3072008-10-19 14:07:49 +0000231the :keyword:`in` operator, and subscript references such as ``d[-1]``. Indexed
232access is O(1) at both ends but slows to O(n) in the middle. For fast random
233access, use lists instead.
Georg Brandl116aa622007-08-15 14:28:22 +0000234
Christian Heimesfe337bf2008-03-23 21:54:12 +0000235Example:
236
237.. doctest::
Georg Brandl116aa622007-08-15 14:28:22 +0000238
239 >>> from collections import deque
240 >>> d = deque('ghi') # make a new deque with three items
241 >>> for elem in d: # iterate over the deque's elements
Neal Norwitz752abd02008-05-13 04:55:24 +0000242 ... print(elem.upper())
Georg Brandl116aa622007-08-15 14:28:22 +0000243 G
244 H
245 I
246
247 >>> d.append('j') # add a new entry to the right side
248 >>> d.appendleft('f') # add a new entry to the left side
249 >>> d # show the representation of the deque
250 deque(['f', 'g', 'h', 'i', 'j'])
251
252 >>> d.pop() # return and remove the rightmost item
253 'j'
254 >>> d.popleft() # return and remove the leftmost item
255 'f'
256 >>> list(d) # list the contents of the deque
257 ['g', 'h', 'i']
258 >>> d[0] # peek at leftmost item
259 'g'
260 >>> d[-1] # peek at rightmost item
261 'i'
262
263 >>> list(reversed(d)) # list the contents of a deque in reverse
264 ['i', 'h', 'g']
265 >>> 'h' in d # search the deque
266 True
267 >>> d.extend('jkl') # add multiple elements at once
268 >>> d
269 deque(['g', 'h', 'i', 'j', 'k', 'l'])
270 >>> d.rotate(1) # right rotation
271 >>> d
272 deque(['l', 'g', 'h', 'i', 'j', 'k'])
273 >>> d.rotate(-1) # left rotation
274 >>> d
275 deque(['g', 'h', 'i', 'j', 'k', 'l'])
276
277 >>> deque(reversed(d)) # make a new deque in reverse order
278 deque(['l', 'k', 'j', 'i', 'h', 'g'])
279 >>> d.clear() # empty the deque
280 >>> d.pop() # cannot pop from an empty deque
281 Traceback (most recent call last):
282 File "<pyshell#6>", line 1, in -toplevel-
283 d.pop()
284 IndexError: pop from an empty deque
285
286 >>> d.extendleft('abc') # extendleft() reverses the input order
287 >>> d
288 deque(['c', 'b', 'a'])
289
290
291.. _deque-recipes:
292
Georg Brandl9afde1c2007-11-01 20:32:30 +0000293:class:`deque` Recipes
294^^^^^^^^^^^^^^^^^^^^^^
Georg Brandl116aa622007-08-15 14:28:22 +0000295
296This section shows various approaches to working with deques.
297
298The :meth:`rotate` method provides a way to implement :class:`deque` slicing and
299deletion. For example, a pure python implementation of ``del d[n]`` relies on
300the :meth:`rotate` method to position elements to be popped::
301
302 def delete_nth(d, n):
303 d.rotate(-n)
304 d.popleft()
305 d.rotate(n)
306
307To implement :class:`deque` slicing, use a similar approach applying
308:meth:`rotate` to bring a target element to the left side of the deque. Remove
309old entries with :meth:`popleft`, add new entries with :meth:`extend`, and then
310reverse the rotation.
Georg Brandl116aa622007-08-15 14:28:22 +0000311With minor variations on that approach, it is easy to implement Forth style
312stack manipulations such as ``dup``, ``drop``, ``swap``, ``over``, ``pick``,
313``rot``, and ``roll``.
314
Georg Brandl116aa622007-08-15 14:28:22 +0000315Multi-pass data reduction algorithms can be succinctly expressed and efficiently
316coded by extracting elements with multiple calls to :meth:`popleft`, applying
Georg Brandl9afde1c2007-11-01 20:32:30 +0000317a reduction function, and calling :meth:`append` to add the result back to the
318deque.
Georg Brandl116aa622007-08-15 14:28:22 +0000319
320For example, building a balanced binary tree of nested lists entails reducing
Christian Heimesfe337bf2008-03-23 21:54:12 +0000321two adjacent nodes into one by grouping them in a list:
Georg Brandl116aa622007-08-15 14:28:22 +0000322
323 >>> def maketree(iterable):
324 ... d = deque(iterable)
325 ... while len(d) > 1:
326 ... pair = [d.popleft(), d.popleft()]
327 ... d.append(pair)
328 ... return list(d)
329 ...
Georg Brandl6911e3c2007-09-04 07:15:32 +0000330 >>> print(maketree('abcdefgh'))
Georg Brandl116aa622007-08-15 14:28:22 +0000331 [[[['a', 'b'], ['c', 'd']], [['e', 'f'], ['g', 'h']]]]
332
Georg Brandl9afde1c2007-11-01 20:32:30 +0000333Bounded length deques provide functionality similar to the ``tail`` filter
334in Unix::
Georg Brandl116aa622007-08-15 14:28:22 +0000335
Georg Brandl9afde1c2007-11-01 20:32:30 +0000336 def tail(filename, n=10):
337 'Return the last n lines of a file'
338 return deque(open(filename), n)
Georg Brandl116aa622007-08-15 14:28:22 +0000339
340.. _defaultdict-objects:
341
342:class:`defaultdict` objects
343----------------------------
344
345
346.. class:: defaultdict([default_factory[, ...]])
347
348 Returns a new dictionary-like object. :class:`defaultdict` is a subclass of the
349 builtin :class:`dict` class. It overrides one method and adds one writable
350 instance variable. The remaining functionality is the same as for the
351 :class:`dict` class and is not documented here.
352
353 The first argument provides the initial value for the :attr:`default_factory`
354 attribute; it defaults to ``None``. All remaining arguments are treated the same
355 as if they were passed to the :class:`dict` constructor, including keyword
356 arguments.
357
Georg Brandl116aa622007-08-15 14:28:22 +0000358
Benjamin Petersone41251e2008-04-25 01:59:09 +0000359 :class:`defaultdict` objects support the following method in addition to the
360 standard :class:`dict` operations:
Georg Brandl116aa622007-08-15 14:28:22 +0000361
Benjamin Petersone41251e2008-04-25 01:59:09 +0000362 .. method:: defaultdict.__missing__(key)
Georg Brandl116aa622007-08-15 14:28:22 +0000363
Benjamin Peterson5478b472008-09-17 22:25:09 +0000364 If the :attr:`default_factory` attribute is ``None``, this raises a
Benjamin Petersone41251e2008-04-25 01:59:09 +0000365 :exc:`KeyError` exception with the *key* as argument.
Georg Brandl116aa622007-08-15 14:28:22 +0000366
Benjamin Petersone41251e2008-04-25 01:59:09 +0000367 If :attr:`default_factory` is not ``None``, it is called without arguments
368 to provide a default value for the given *key*, this value is inserted in
369 the dictionary for the *key*, and returned.
Georg Brandl116aa622007-08-15 14:28:22 +0000370
Benjamin Petersone41251e2008-04-25 01:59:09 +0000371 If calling :attr:`default_factory` raises an exception this exception is
372 propagated unchanged.
Georg Brandl116aa622007-08-15 14:28:22 +0000373
Benjamin Petersone41251e2008-04-25 01:59:09 +0000374 This method is called by the :meth:`__getitem__` method of the
375 :class:`dict` class when the requested key is not found; whatever it
376 returns or raises is then returned or raised by :meth:`__getitem__`.
Georg Brandl116aa622007-08-15 14:28:22 +0000377
378
Benjamin Petersone41251e2008-04-25 01:59:09 +0000379 :class:`defaultdict` objects support the following instance variable:
Georg Brandl116aa622007-08-15 14:28:22 +0000380
Benjamin Petersone41251e2008-04-25 01:59:09 +0000381
382 .. attribute:: defaultdict.default_factory
383
384 This attribute is used by the :meth:`__missing__` method; it is
385 initialized from the first argument to the constructor, if present, or to
386 ``None``, if absent.
Georg Brandl116aa622007-08-15 14:28:22 +0000387
388
389.. _defaultdict-examples:
390
391:class:`defaultdict` Examples
392^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
393
394Using :class:`list` as the :attr:`default_factory`, it is easy to group a
Christian Heimesfe337bf2008-03-23 21:54:12 +0000395sequence of key-value pairs into a dictionary of lists:
Georg Brandl116aa622007-08-15 14:28:22 +0000396
397 >>> s = [('yellow', 1), ('blue', 2), ('yellow', 3), ('blue', 4), ('red', 1)]
398 >>> d = defaultdict(list)
399 >>> for k, v in s:
400 ... d[k].append(v)
401 ...
402 >>> d.items()
403 [('blue', [2, 4]), ('red', [1]), ('yellow', [1, 3])]
404
405When each key is encountered for the first time, it is not already in the
406mapping; so an entry is automatically created using the :attr:`default_factory`
407function which returns an empty :class:`list`. The :meth:`list.append`
408operation then attaches the value to the new list. When keys are encountered
409again, the look-up proceeds normally (returning the list for that key) and the
410:meth:`list.append` operation adds another value to the list. This technique is
Christian Heimesfe337bf2008-03-23 21:54:12 +0000411simpler and faster than an equivalent technique using :meth:`dict.setdefault`:
Georg Brandl116aa622007-08-15 14:28:22 +0000412
413 >>> d = {}
414 >>> for k, v in s:
415 ... d.setdefault(k, []).append(v)
416 ...
417 >>> d.items()
418 [('blue', [2, 4]), ('red', [1]), ('yellow', [1, 3])]
419
420Setting the :attr:`default_factory` to :class:`int` makes the
421:class:`defaultdict` useful for counting (like a bag or multiset in other
Christian Heimesfe337bf2008-03-23 21:54:12 +0000422languages):
Georg Brandl116aa622007-08-15 14:28:22 +0000423
424 >>> s = 'mississippi'
425 >>> d = defaultdict(int)
426 >>> for k in s:
427 ... d[k] += 1
428 ...
429 >>> d.items()
430 [('i', 4), ('p', 2), ('s', 4), ('m', 1)]
431
432When a letter is first encountered, it is missing from the mapping, so the
433:attr:`default_factory` function calls :func:`int` to supply a default count of
434zero. The increment operation then builds up the count for each letter.
435
436The function :func:`int` which always returns zero is just a special case of
437constant functions. A faster and more flexible way to create constant functions
438is to use a lambda function which can supply any constant value (not just
Christian Heimesfe337bf2008-03-23 21:54:12 +0000439zero):
Georg Brandl116aa622007-08-15 14:28:22 +0000440
441 >>> def constant_factory(value):
442 ... return lambda: value
443 >>> d = defaultdict(constant_factory('<missing>'))
444 >>> d.update(name='John', action='ran')
445 >>> '%(name)s %(action)s to %(object)s' % d
446 'John ran to <missing>'
447
448Setting the :attr:`default_factory` to :class:`set` makes the
Christian Heimesfe337bf2008-03-23 21:54:12 +0000449:class:`defaultdict` useful for building a dictionary of sets:
Georg Brandl116aa622007-08-15 14:28:22 +0000450
451 >>> s = [('red', 1), ('blue', 2), ('red', 3), ('blue', 4), ('red', 1), ('blue', 4)]
452 >>> d = defaultdict(set)
453 >>> for k, v in s:
454 ... d[k].add(v)
455 ...
456 >>> d.items()
457 [('blue', set([2, 4])), ('red', set([1, 3]))]
458
459
460.. _named-tuple-factory:
461
Guido van Rossum3d392eb2007-11-16 00:35:22 +0000462:func:`namedtuple` Factory Function for Tuples with Named Fields
Christian Heimes790c8232008-01-07 21:14:23 +0000463----------------------------------------------------------------
Georg Brandl116aa622007-08-15 14:28:22 +0000464
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000465Named tuples assign meaning to each position in a tuple and allow for more readable,
466self-documenting code. They can be used wherever regular tuples are used, and
467they add the ability to access fields by name instead of position index.
Georg Brandl116aa622007-08-15 14:28:22 +0000468
Benjamin Peterson4469d0c2008-11-30 22:46:23 +0000469.. function:: namedtuple(typename, field_names, [verbose])
Georg Brandl116aa622007-08-15 14:28:22 +0000470
471 Returns a new tuple subclass named *typename*. The new subclass is used to
Christian Heimesc3f30c42008-02-22 16:37:40 +0000472 create tuple-like objects that have fields accessible by attribute lookup as
Georg Brandl116aa622007-08-15 14:28:22 +0000473 well as being indexable and iterable. Instances of the subclass also have a
Benjamin Peterson4469d0c2008-11-30 22:46:23 +0000474 helpful docstring (with typename and field_names) and a helpful :meth:`__repr__`
Georg Brandl116aa622007-08-15 14:28:22 +0000475 method which lists the tuple contents in a ``name=value`` format.
476
Benjamin Peterson4469d0c2008-11-30 22:46:23 +0000477 The *field_names* are a single string with each fieldname separated by whitespace
478 and/or commas, for example ``'x y'`` or ``'x, y'``. Alternatively, *field_names*
Christian Heimes25bb7832008-01-11 16:17:00 +0000479 can be a sequence of strings such as ``['x', 'y']``.
Georg Brandl9afde1c2007-11-01 20:32:30 +0000480
481 Any valid Python identifier may be used for a fieldname except for names
Christian Heimes0449f632007-12-15 01:27:15 +0000482 starting with an underscore. Valid identifiers consist of letters, digits,
483 and underscores but do not start with a digit or underscore and cannot be
Georg Brandlf6945182008-02-01 11:56:49 +0000484 a :mod:`keyword` such as *class*, *for*, *return*, *global*, *pass*,
Georg Brandl9afde1c2007-11-01 20:32:30 +0000485 or *raise*.
Georg Brandl116aa622007-08-15 14:28:22 +0000486
Christian Heimes25bb7832008-01-11 16:17:00 +0000487 If *verbose* is true, the class definition is printed just before being built.
Georg Brandl116aa622007-08-15 14:28:22 +0000488
Georg Brandl9afde1c2007-11-01 20:32:30 +0000489 Named tuple instances do not have per-instance dictionaries, so they are
Thomas Wouters8ce81f72007-09-20 18:22:40 +0000490 lightweight and require no more memory than regular tuples.
Georg Brandl116aa622007-08-15 14:28:22 +0000491
Christian Heimesfe337bf2008-03-23 21:54:12 +0000492Example:
493
494.. doctest::
495 :options: +NORMALIZE_WHITESPACE
Georg Brandl116aa622007-08-15 14:28:22 +0000496
Guido van Rossum3d392eb2007-11-16 00:35:22 +0000497 >>> Point = namedtuple('Point', 'x y', verbose=True)
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000498 class Point(tuple):
499 'Point(x, y)'
Christian Heimesfe337bf2008-03-23 21:54:12 +0000500 <BLANKLINE>
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000501 __slots__ = ()
Christian Heimesfe337bf2008-03-23 21:54:12 +0000502 <BLANKLINE>
Christian Heimesfaf2f632008-01-06 16:59:19 +0000503 _fields = ('x', 'y')
Christian Heimesfe337bf2008-03-23 21:54:12 +0000504 <BLANKLINE>
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000505 def __new__(cls, x, y):
506 return tuple.__new__(cls, (x, y))
Christian Heimesfe337bf2008-03-23 21:54:12 +0000507 <BLANKLINE>
Christian Heimesfaf2f632008-01-06 16:59:19 +0000508 @classmethod
Christian Heimesfe337bf2008-03-23 21:54:12 +0000509 def _make(cls, iterable, new=tuple.__new__, len=len):
Christian Heimesfaf2f632008-01-06 16:59:19 +0000510 'Make a new Point object from a sequence or iterable'
Christian Heimesfe337bf2008-03-23 21:54:12 +0000511 result = new(cls, iterable)
Christian Heimesfaf2f632008-01-06 16:59:19 +0000512 if len(result) != 2:
513 raise TypeError('Expected 2 arguments, got %d' % len(result))
514 return result
Christian Heimesfe337bf2008-03-23 21:54:12 +0000515 <BLANKLINE>
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000516 def __repr__(self):
517 return 'Point(x=%r, y=%r)' % self
Christian Heimesfe337bf2008-03-23 21:54:12 +0000518 <BLANKLINE>
Christian Heimes99170a52007-12-19 02:07:34 +0000519 def _asdict(t):
Christian Heimes0449f632007-12-15 01:27:15 +0000520 'Return a new dict which maps field names to their values'
Christian Heimes99170a52007-12-19 02:07:34 +0000521 return {'x': t[0], 'y': t[1]}
Christian Heimesfe337bf2008-03-23 21:54:12 +0000522 <BLANKLINE>
Christian Heimes0449f632007-12-15 01:27:15 +0000523 def _replace(self, **kwds):
Guido van Rossum3d392eb2007-11-16 00:35:22 +0000524 'Return a new Point object replacing specified fields with new values'
Christian Heimesfaf2f632008-01-06 16:59:19 +0000525 result = self._make(map(kwds.pop, ('x', 'y'), self))
526 if kwds:
527 raise ValueError('Got unexpected field names: %r' % kwds.keys())
528 return result
Georg Brandl48310cd2009-01-03 21:18:54 +0000529 <BLANKLINE>
530 def __getnewargs__(self):
Benjamin Peterson41181742008-07-02 20:22:54 +0000531 return tuple(self)
Christian Heimesfe337bf2008-03-23 21:54:12 +0000532 <BLANKLINE>
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000533 x = property(itemgetter(0))
534 y = property(itemgetter(1))
Georg Brandl116aa622007-08-15 14:28:22 +0000535
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000536 >>> p = Point(11, y=22) # instantiate with positional or keyword arguments
Christian Heimes99170a52007-12-19 02:07:34 +0000537 >>> p[0] + p[1] # indexable like the plain tuple (11, 22)
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000538 33
539 >>> x, y = p # unpack like a regular tuple
540 >>> x, y
541 (11, 22)
Christian Heimesc3f30c42008-02-22 16:37:40 +0000542 >>> p.x + p.y # fields also accessible by name
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000543 33
544 >>> p # readable __repr__ with a name=value style
545 Point(x=11, y=22)
Georg Brandl116aa622007-08-15 14:28:22 +0000546
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000547Named tuples are especially useful for assigning field names to result tuples returned
548by the :mod:`csv` or :mod:`sqlite3` modules::
549
Guido van Rossum3d392eb2007-11-16 00:35:22 +0000550 EmployeeRecord = namedtuple('EmployeeRecord', 'name, age, title, department, paygrade')
Georg Brandl9afde1c2007-11-01 20:32:30 +0000551
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000552 import csv
Christian Heimesfaf2f632008-01-06 16:59:19 +0000553 for emp in map(EmployeeRecord._make, csv.reader(open("employees.csv", "rb"))):
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000554 print(emp.name, emp.title)
555
Georg Brandl9afde1c2007-11-01 20:32:30 +0000556 import sqlite3
557 conn = sqlite3.connect('/companydata')
558 cursor = conn.cursor()
559 cursor.execute('SELECT name, age, title, department, paygrade FROM employees')
Christian Heimesfaf2f632008-01-06 16:59:19 +0000560 for emp in map(EmployeeRecord._make, cursor.fetchall()):
Christian Heimes00412232008-01-10 16:02:19 +0000561 print(emp.name, emp.title)
Georg Brandl9afde1c2007-11-01 20:32:30 +0000562
Christian Heimes99170a52007-12-19 02:07:34 +0000563In addition to the methods inherited from tuples, named tuples support
Christian Heimes2380ac72008-01-09 00:17:24 +0000564three additional methods and one attribute. To prevent conflicts with
565field names, the method and attribute names start with an underscore.
Christian Heimes99170a52007-12-19 02:07:34 +0000566
Christian Heimes790c8232008-01-07 21:14:23 +0000567.. method:: somenamedtuple._make(iterable)
Christian Heimes99170a52007-12-19 02:07:34 +0000568
Christian Heimesfaf2f632008-01-06 16:59:19 +0000569 Class method that makes a new instance from an existing sequence or iterable.
Christian Heimes99170a52007-12-19 02:07:34 +0000570
Christian Heimesfe337bf2008-03-23 21:54:12 +0000571.. doctest::
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000572
Christian Heimesfaf2f632008-01-06 16:59:19 +0000573 >>> t = [11, 22]
574 >>> Point._make(t)
575 Point(x=11, y=22)
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000576
Christian Heimes790c8232008-01-07 21:14:23 +0000577.. method:: somenamedtuple._asdict()
Georg Brandl9afde1c2007-11-01 20:32:30 +0000578
Christian Heimesfe337bf2008-03-23 21:54:12 +0000579 Return a new dict which maps field names to their corresponding values::
Georg Brandl9afde1c2007-11-01 20:32:30 +0000580
Christian Heimes0449f632007-12-15 01:27:15 +0000581 >>> p._asdict()
Georg Brandl9afde1c2007-11-01 20:32:30 +0000582 {'x': 11, 'y': 22}
Christian Heimesfe337bf2008-03-23 21:54:12 +0000583
Christian Heimes790c8232008-01-07 21:14:23 +0000584.. method:: somenamedtuple._replace(kwargs)
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000585
Christian Heimesfe337bf2008-03-23 21:54:12 +0000586 Return a new instance of the named tuple replacing specified fields with new
587 values:
Thomas Wouters8ce81f72007-09-20 18:22:40 +0000588
589::
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000590
591 >>> p = Point(x=11, y=22)
Christian Heimes0449f632007-12-15 01:27:15 +0000592 >>> p._replace(x=33)
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000593 Point(x=33, y=22)
594
Guido van Rossum3d392eb2007-11-16 00:35:22 +0000595 >>> for partnum, record in inventory.items():
Christian Heimes454f37b2008-01-10 00:10:02 +0000596 ... inventory[partnum] = record._replace(price=newprices[partnum], timestamp=time.now())
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000597
Christian Heimes790c8232008-01-07 21:14:23 +0000598.. attribute:: somenamedtuple._fields
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000599
Christian Heimes2380ac72008-01-09 00:17:24 +0000600 Tuple of strings listing the field names. Useful for introspection
Georg Brandl9afde1c2007-11-01 20:32:30 +0000601 and for creating new named tuple types from existing named tuples.
Thomas Wouters8ce81f72007-09-20 18:22:40 +0000602
Christian Heimesfe337bf2008-03-23 21:54:12 +0000603.. doctest::
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000604
Christian Heimes0449f632007-12-15 01:27:15 +0000605 >>> p._fields # view the field names
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000606 ('x', 'y')
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000607
Guido van Rossum3d392eb2007-11-16 00:35:22 +0000608 >>> Color = namedtuple('Color', 'red green blue')
Christian Heimes0449f632007-12-15 01:27:15 +0000609 >>> Pixel = namedtuple('Pixel', Point._fields + Color._fields)
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000610 >>> Pixel(11, 22, 128, 255, 0)
Christian Heimes454f37b2008-01-10 00:10:02 +0000611 Pixel(x=11, y=22, red=128, green=255, blue=0)
Georg Brandl116aa622007-08-15 14:28:22 +0000612
Christian Heimes0449f632007-12-15 01:27:15 +0000613To retrieve a field whose name is stored in a string, use the :func:`getattr`
Christian Heimesfe337bf2008-03-23 21:54:12 +0000614function:
Christian Heimes0449f632007-12-15 01:27:15 +0000615
616 >>> getattr(p, 'x')
617 11
618
Christian Heimesfe337bf2008-03-23 21:54:12 +0000619To convert a dictionary to a named tuple, use the double-star-operator [#]_:
Christian Heimes99170a52007-12-19 02:07:34 +0000620
621 >>> d = {'x': 11, 'y': 22}
622 >>> Point(**d)
623 Point(x=11, y=22)
624
Guido van Rossum3d392eb2007-11-16 00:35:22 +0000625Since a named tuple is a regular Python class, it is easy to add or change
Christian Heimes043d6f62008-01-07 17:19:16 +0000626functionality with a subclass. Here is how to add a calculated field and
Christian Heimesfe337bf2008-03-23 21:54:12 +0000627a fixed-width print format:
Guido van Rossum3d392eb2007-11-16 00:35:22 +0000628
Christian Heimes043d6f62008-01-07 17:19:16 +0000629 >>> class Point(namedtuple('Point', 'x y')):
Christian Heimes25bb7832008-01-11 16:17:00 +0000630 ... __slots__ = ()
Christian Heimes454f37b2008-01-10 00:10:02 +0000631 ... @property
632 ... def hypot(self):
633 ... return (self.x ** 2 + self.y ** 2) ** 0.5
634 ... def __str__(self):
Christian Heimes25bb7832008-01-11 16:17:00 +0000635 ... return 'Point: x=%6.3f y=%6.3f hypot=%6.3f' % (self.x, self.y, self.hypot)
Guido van Rossum3d392eb2007-11-16 00:35:22 +0000636
Georg Brandl0df79792008-10-04 18:33:26 +0000637 >>> for p in Point(3, 4), Point(14, 5/7):
Christian Heimes00412232008-01-10 16:02:19 +0000638 ... print(p)
Christian Heimes25bb7832008-01-11 16:17:00 +0000639 Point: x= 3.000 y= 4.000 hypot= 5.000
640 Point: x=14.000 y= 0.714 hypot=14.018
Christian Heimes043d6f62008-01-07 17:19:16 +0000641
Christian Heimesaf98da12008-01-27 15:18:18 +0000642The subclass shown above sets ``__slots__`` to an empty tuple. This keeps
Christian Heimes679db4a2008-01-18 09:56:22 +0000643keep memory requirements low by preventing the creation of instance dictionaries.
644
Christian Heimes2380ac72008-01-09 00:17:24 +0000645
646Subclassing is not useful for adding new, stored fields. Instead, simply
Christian Heimesfe337bf2008-03-23 21:54:12 +0000647create a new named tuple type from the :attr:`_fields` attribute:
Christian Heimes2380ac72008-01-09 00:17:24 +0000648
Christian Heimes25bb7832008-01-11 16:17:00 +0000649 >>> Point3D = namedtuple('Point3D', Point._fields + ('z',))
Christian Heimes2380ac72008-01-09 00:17:24 +0000650
651Default values can be implemented by using :meth:`_replace` to
Christian Heimesfe337bf2008-03-23 21:54:12 +0000652customize a prototype instance:
Guido van Rossum3d392eb2007-11-16 00:35:22 +0000653
654 >>> Account = namedtuple('Account', 'owner balance transaction_count')
Christian Heimes587c2bf2008-01-19 16:21:02 +0000655 >>> default_account = Account('<owner name>', 0.0, 0)
656 >>> johns_account = default_account._replace(owner='John')
Guido van Rossum3d392eb2007-11-16 00:35:22 +0000657
Christian Heimese4ca8152008-05-08 17:18:53 +0000658Enumerated constants can be implemented with named tuples, but it is simpler
659and more efficient to use a simple class declaration:
660
661 >>> Status = namedtuple('Status', 'open pending closed')._make(range(3))
662 >>> Status.open, Status.pending, Status.closed
663 (0, 1, 2)
664 >>> class Status:
665 ... open, pending, closed = range(3)
666
Thomas Wouters47b49bf2007-08-30 22:15:33 +0000667.. rubric:: Footnotes
668
Christian Heimes99170a52007-12-19 02:07:34 +0000669.. [#] For information on the double-star-operator see
Thomas Wouters47b49bf2007-08-30 22:15:33 +0000670 :ref:`tut-unpacking-arguments` and :ref:`calls`.
Raymond Hettingere4c96ad2008-02-06 01:23:58 +0000671
672
673
674:class:`UserDict` objects
Mark Summerfield8f2d0062008-02-06 13:30:44 +0000675-------------------------
Raymond Hettingere4c96ad2008-02-06 01:23:58 +0000676
Georg Brandl48310cd2009-01-03 21:18:54 +0000677The class, :class:`UserDict` acts as a wrapper around dictionary objects.
678The need for this class has been partially supplanted by the ability to
Raymond Hettingere4c96ad2008-02-06 01:23:58 +0000679subclass directly from :class:`dict`; however, this class can be easier
680to work with because the underlying dictionary is accessible as an
681attribute.
682
683.. class:: UserDict([initialdata])
684
685 Class that simulates a dictionary. The instance's contents are kept in a
686 regular dictionary, which is accessible via the :attr:`data` attribute of
687 :class:`UserDict` instances. If *initialdata* is provided, :attr:`data` is
688 initialized with its contents; note that a reference to *initialdata* will not
689 be kept, allowing it be used for other purposes.
690
Georg Brandl48310cd2009-01-03 21:18:54 +0000691In addition to supporting the methods and operations of mappings,
Raymond Hettingerebcee3f2008-02-06 19:54:00 +0000692:class:`UserDict` instances provide the following attribute:
Raymond Hettingere4c96ad2008-02-06 01:23:58 +0000693
694.. attribute:: UserDict.data
695
696 A real dictionary used to store the contents of the :class:`UserDict` class.
Raymond Hettinger53dbe392008-02-12 20:03:09 +0000697
698
699
700:class:`UserList` objects
701-------------------------
702
703This class acts as a wrapper around list objects. It is a useful base class
Georg Brandl48310cd2009-01-03 21:18:54 +0000704for your own list-like classes which can inherit from them and override
Raymond Hettinger53dbe392008-02-12 20:03:09 +0000705existing methods or add new ones. In this way, one can add new behaviors to
706lists.
707
Georg Brandl48310cd2009-01-03 21:18:54 +0000708The need for this class has been partially supplanted by the ability to
Raymond Hettinger53dbe392008-02-12 20:03:09 +0000709subclass directly from :class:`list`; however, this class can be easier
710to work with because the underlying list is accessible as an attribute.
711
712.. class:: UserList([list])
713
714 Class that simulates a list. The instance's contents are kept in a regular
715 list, which is accessible via the :attr:`data` attribute of :class:`UserList`
716 instances. The instance's contents are initially set to a copy of *list*,
717 defaulting to the empty list ``[]``. *list* can be any iterable, for
718 example a real Python list or a :class:`UserList` object.
719
Georg Brandl48310cd2009-01-03 21:18:54 +0000720In addition to supporting the methods and operations of mutable sequences,
Raymond Hettinger53dbe392008-02-12 20:03:09 +0000721:class:`UserList` instances provide the following attribute:
722
723.. attribute:: UserList.data
724
Georg Brandl48310cd2009-01-03 21:18:54 +0000725 A real :class:`list` object used to store the contents of the
Raymond Hettinger53dbe392008-02-12 20:03:09 +0000726 :class:`UserList` class.
727
728**Subclassing requirements:** Subclasses of :class:`UserList` are expect to
729offer a constructor which can be called with either no arguments or one
730argument. List operations which return a new sequence attempt to create an
731instance of the actual implementation class. To do so, it assumes that the
732constructor can be called with a single parameter, which is a sequence object
733used as a data source.
734
735If a derived class does not wish to comply with this requirement, all of the
736special methods supported by this class will need to be overridden; please
737consult the sources for information about the methods which need to be provided
738in that case.
Raymond Hettingerb3a65f82008-02-21 22:11:37 +0000739
740:class:`UserString` objects
Christian Heimesc3f30c42008-02-22 16:37:40 +0000741---------------------------
Raymond Hettingerb3a65f82008-02-21 22:11:37 +0000742
Georg Brandl48310cd2009-01-03 21:18:54 +0000743The class, :class:`UserString` acts as a wrapper around string objects.
744The need for this class has been partially supplanted by the ability to
Raymond Hettingerb3a65f82008-02-21 22:11:37 +0000745subclass directly from :class:`str`; however, this class can be easier
746to work with because the underlying string is accessible as an
747attribute.
748
749.. class:: UserString([sequence])
750
751 Class that simulates a string or a Unicode string object. The instance's
Georg Brandl48310cd2009-01-03 21:18:54 +0000752 content is kept in a regular string object, which is accessible via the
753 :attr:`data` attribute of :class:`UserString` instances. The instance's
Raymond Hettingerb3a65f82008-02-21 22:11:37 +0000754 contents are initially set to a copy of *sequence*. The *sequence* can
755 be an instance of :class:`bytes`, :class:`str`, :class:`UserString` (or a
756 subclass) or an arbitrary sequence which can be converted into a string using
757 the built-in :func:`str` function.