blob: d2d4f4750fcd5dfa72db81cb7d409ba31af60f85 [file] [log] [blame]
Georg Brandl116aa622007-08-15 14:28:22 +00001
Raymond Hettinger53dbe392008-02-12 20:03:09 +00002:mod:`collections` --- Container datatypes
3==========================================
Georg Brandl116aa622007-08-15 14:28:22 +00004
5.. module:: collections
Raymond Hettinger53dbe392008-02-12 20:03:09 +00006 :synopsis: Container datatypes
Georg Brandl116aa622007-08-15 14:28:22 +00007.. moduleauthor:: Raymond Hettinger <python@rcn.com>
8.. sectionauthor:: Raymond Hettinger <python@rcn.com>
9
Christian Heimesfe337bf2008-03-23 21:54:12 +000010.. testsetup:: *
11
12 from collections import *
13 import itertools
14 __name__ = '<doctest>'
Georg Brandl116aa622007-08-15 14:28:22 +000015
Georg Brandl116aa622007-08-15 14:28:22 +000016This module implements high-performance container datatypes. Currently,
17there are two datatypes, :class:`deque` and :class:`defaultdict`, and
Mark Summerfield71316b02008-02-14 16:28:00 +000018one datatype factory function, :func:`namedtuple`. This module also
19provides the :class:`UserDict` and :class:`UserList` classes which may
20be useful when inheriting directly from :class:`dict` or
21:class:`list` isn't convenient.
Christian Heimes0bd4e112008-02-12 22:59:25 +000022
Raymond Hettingerebcee3f2008-02-06 19:54:00 +000023The specialized containers provided in this module provide alternatives
Christian Heimesfe337bf2008-03-23 21:54:12 +000024to Python's general purpose built-in containers, :class:`dict`,
Raymond Hettingerebcee3f2008-02-06 19:54:00 +000025:class:`list`, :class:`set`, and :class:`tuple`.
Raymond Hettingerebcee3f2008-02-06 19:54:00 +000026Besides the containers provided here, the optional :mod:`bsddb`
Christian Heimesfe337bf2008-03-23 21:54:12 +000027module offers the ability to create in-memory or file based ordered
Raymond Hettingerebcee3f2008-02-06 19:54:00 +000028dictionaries with string keys using the :meth:`bsddb.btopen` method.
Georg Brandl116aa622007-08-15 14:28:22 +000029
Mark Summerfield08898b42007-09-05 08:43:04 +000030In addition to containers, the collections module provides some ABCs
Christian Heimesfe337bf2008-03-23 21:54:12 +000031(abstract base classes) that can be used to test whether a class
Raymond Hettingerebcee3f2008-02-06 19:54:00 +000032provides a particular interface, for example, is it hashable or
Mark Summerfield71316b02008-02-14 16:28:00 +000033a mapping, and some of them can also be used as mixin classes.
Raymond Hettingerebcee3f2008-02-06 19:54:00 +000034
35ABCs - abstract base classes
36----------------------------
37
38The collections module offers the following ABCs:
Mark Summerfield08898b42007-09-05 08:43:04 +000039
Georg Brandl86b2fb92008-07-16 03:43:04 +000040========================= ===================== ====================== ====================================================
41ABC Inherits Abstract Methods Mixin Methods
42========================= ===================== ====================== ====================================================
43:class:`Container` ``__contains__``
44:class:`Hashable` ``__hash__``
45:class:`Iterable` ``__iter__``
46:class:`Iterator` :class:`Iterable` ``__next__`` ``__iter__``
47:class:`Sized` ``__len__``
48:class:`Callable` ``__call__``
49
50:class:`Sequence` :class:`Sized`, ``__getitem__`` ``__contains__``. ``__iter__``, ``__reversed__``.
51 :class:`Iterable`, and ``__len__`` ``index``, and ``count``
52 :class:`Container`
53
54:class:`MutableSequnce` :class:`Sequence` ``__getitem__`` Inherited Sequence methods and
55 ``__delitem__``, ``append``, ``reverse``, ``extend``, ``pop``,
56 ``insert``, ``remove``, and ``__iadd__``
57 and ``__len__``
58
59:class:`Set` :class:`Sized`, ``__len__``, ``__le__``, ``__lt__``, ``__eq__``, ``__ne__``,
60 :class:`Iterable`, ``__iter__``, and ``__gt__``, ``__ge__``, ``__and__``, ``__or__``
61 :class:`Container` ``__contains__`` ``__sub__``, ``__xor__``, and ``isdisjoint``
62
63:class:`MutableSet` :class:`Set` ``add`` and Inherited Set methods and
64 ``discard`` ``clear``, ``pop``, ``remove``, ``__ior__``,
65 ``__iand__``, ``__ixor__``, and ``__isub__``
66
67:class:`Mapping` :class:`Sized`, ``__getitem__``, ``__contains__``, ``keys``, ``items``, ``values``,
68 :class:`Iterable`, ``__len__``. and ``get``, ``__eq__``, and ``__ne__``
69 :class:`Container` ``__iter__``
70
71:class:`MutableMapping` :class:`Mapping` ``__getitem__`` Inherited Mapping methods and
72 ``__setitem__``, ``pop``, ``popitem``, ``clear``, ``update``,
73 ``__delitem__``, and ``setdefault``
74 ``__iter__``, and
75 ``__len__``
76
77:class:`MappingView` :class:`Sized` ``__len__``
78:class:`KeysView` :class:`MappingView`, ``__contains__``,
79 :class:`Set` ``__iter__``
80:class:`ItemsView` :class:`MappingView`, ``__contains__``,
81 :class:`Set` ``__iter__``
82:class:`ValuesView` :class:`MappingView` ``__contains__``, ``__iter__``
83========================= ===================== ====================== ====================================================
Mark Summerfield08898b42007-09-05 08:43:04 +000084
Mark Summerfield08898b42007-09-05 08:43:04 +000085These ABCs allow us to ask classes or instances if they provide
86particular functionality, for example::
87
Mark Summerfield08898b42007-09-05 08:43:04 +000088 size = None
Raymond Hettingerebcee3f2008-02-06 19:54:00 +000089 if isinstance(myvar, collections.Sized):
Mark Summerfield08898b42007-09-05 08:43:04 +000090 size = len(myvar)
91
Raymond Hettingerebcee3f2008-02-06 19:54:00 +000092Several of the ABCs are also useful as mixins that make it easier to develop
93classes supporting container APIs. For example, to write a class supporting
94the full :class:`Set` API, it only necessary to supply the three underlying
95abstract methods: :meth:`__contains__`, :meth:`__iter__`, and :meth:`__len__`.
96The ABC supplies the remaining methods such as :meth:`__and__` and
97:meth:`isdisjoint` ::
98
99 class ListBasedSet(collections.Set):
Raymond Hettingerc1b6a4a2008-02-08 23:46:23 +0000100 ''' Alternate set implementation favoring space over speed
101 and not requiring the set elements to be hashable. '''
Raymond Hettingerebcee3f2008-02-06 19:54:00 +0000102 def __init__(self, iterable):
Raymond Hettingerc1b6a4a2008-02-08 23:46:23 +0000103 self.elements = lst = []
104 for value in iterable:
105 if value not in lst:
106 lst.append(value)
Raymond Hettingerebcee3f2008-02-06 19:54:00 +0000107 def __iter__(self):
108 return iter(self.elements)
109 def __contains__(self, value):
110 return value in self.elements
111 def __len__(self):
112 return len(self.elements)
113
114 s1 = ListBasedSet('abcdef')
115 s2 = ListBasedSet('defghi')
116 overlap = s1 & s2 # The __and__() method is supported automatically
117
Raymond Hettinger7aebb642008-02-09 03:25:08 +0000118Notes on using :class:`Set` and :class:`MutableSet` as a mixin:
119
Christian Heimesfe337bf2008-03-23 21:54:12 +0000120(1)
Raymond Hettinger7aebb642008-02-09 03:25:08 +0000121 Since some set operations create new sets, the default mixin methods need
Christian Heimesfe337bf2008-03-23 21:54:12 +0000122 a way to create new instances from an iterable. The class constructor is
123 assumed to have a signature in the form ``ClassName(iterable)``.
Benjamin Peterson2b7411d2008-05-26 17:36:47 +0000124 That assumption is factored-out to an internal classmethod called
Raymond Hettinger7aebb642008-02-09 03:25:08 +0000125 :meth:`_from_iterable` which calls ``cls(iterable)`` to produce a new set.
126 If the :class:`Set` mixin is being used in a class with a different
Christian Heimesfe337bf2008-03-23 21:54:12 +0000127 constructor signature, you will need to override :meth:`from_iterable`
128 with a classmethod that can construct new instances from
Raymond Hettinger7aebb642008-02-09 03:25:08 +0000129 an iterable argument.
130
131(2)
132 To override the comparisons (presumably for speed, as the
133 semantics are fixed), redefine :meth:`__le__` and
134 then the other operations will automatically follow suit.
Raymond Hettingerebcee3f2008-02-06 19:54:00 +0000135
Raymond Hettinger0dbdab22008-02-09 03:48:16 +0000136(3)
137 The :class:`Set` mixin provides a :meth:`_hash` method to compute a hash value
138 for the set; however, :meth:`__hash__` is not defined because not all sets
139 are hashable or immutable. To add set hashabilty using mixins,
140 inherit from both :meth:`Set` and :meth:`Hashable`, then define
141 ``__hash__ = Set._hash``.
142
Mark Summerfield08898b42007-09-05 08:43:04 +0000143(For more about ABCs, see the :mod:`abc` module and :pep:`3119`.)
144
145
Georg Brandl116aa622007-08-15 14:28:22 +0000146.. _deque-objects:
147
148:class:`deque` objects
149----------------------
150
151
Georg Brandl9afde1c2007-11-01 20:32:30 +0000152.. class:: deque([iterable[, maxlen]])
Georg Brandl116aa622007-08-15 14:28:22 +0000153
154 Returns a new deque object initialized left-to-right (using :meth:`append`) with
155 data from *iterable*. If *iterable* is not specified, the new deque is empty.
156
157 Deques are a generalization of stacks and queues (the name is pronounced "deck"
158 and is short for "double-ended queue"). Deques support thread-safe, memory
159 efficient appends and pops from either side of the deque with approximately the
160 same O(1) performance in either direction.
161
162 Though :class:`list` objects support similar operations, they are optimized for
163 fast fixed-length operations and incur O(n) memory movement costs for
164 ``pop(0)`` and ``insert(0, v)`` operations which change both the size and
165 position of the underlying data representation.
166
Georg Brandl116aa622007-08-15 14:28:22 +0000167
Georg Brandl9afde1c2007-11-01 20:32:30 +0000168 If *maxlen* is not specified or is *None*, deques may grow to an
169 arbitrary length. Otherwise, the deque is bounded to the specified maximum
170 length. Once a bounded length deque is full, when new items are added, a
171 corresponding number of items are discarded from the opposite end. Bounded
172 length deques provide functionality similar to the ``tail`` filter in
173 Unix. They are also useful for tracking transactions and other pools of data
174 where only the most recent activity is of interest.
175
Georg Brandl9afde1c2007-11-01 20:32:30 +0000176
Benjamin Petersone41251e2008-04-25 01:59:09 +0000177 Deque objects support the following methods:
Georg Brandl116aa622007-08-15 14:28:22 +0000178
Benjamin Petersone41251e2008-04-25 01:59:09 +0000179 .. method:: append(x)
Georg Brandl116aa622007-08-15 14:28:22 +0000180
Benjamin Petersone41251e2008-04-25 01:59:09 +0000181 Add *x* to the right side of the deque.
Georg Brandl116aa622007-08-15 14:28:22 +0000182
183
Benjamin Petersone41251e2008-04-25 01:59:09 +0000184 .. method:: appendleft(x)
Georg Brandl116aa622007-08-15 14:28:22 +0000185
Benjamin Petersone41251e2008-04-25 01:59:09 +0000186 Add *x* to the left side of the deque.
Georg Brandl116aa622007-08-15 14:28:22 +0000187
188
Benjamin Petersone41251e2008-04-25 01:59:09 +0000189 .. method:: clear()
Georg Brandl116aa622007-08-15 14:28:22 +0000190
Benjamin Petersone41251e2008-04-25 01:59:09 +0000191 Remove all elements from the deque leaving it with length 0.
Georg Brandl116aa622007-08-15 14:28:22 +0000192
193
Benjamin Petersone41251e2008-04-25 01:59:09 +0000194 .. method:: extend(iterable)
Georg Brandl116aa622007-08-15 14:28:22 +0000195
Benjamin Petersone41251e2008-04-25 01:59:09 +0000196 Extend the right side of the deque by appending elements from the iterable
197 argument.
Georg Brandl116aa622007-08-15 14:28:22 +0000198
199
Benjamin Petersone41251e2008-04-25 01:59:09 +0000200 .. method:: extendleft(iterable)
Georg Brandl116aa622007-08-15 14:28:22 +0000201
Benjamin Petersone41251e2008-04-25 01:59:09 +0000202 Extend the left side of the deque by appending elements from *iterable*.
203 Note, the series of left appends results in reversing the order of
204 elements in the iterable argument.
Georg Brandl116aa622007-08-15 14:28:22 +0000205
206
Benjamin Petersone41251e2008-04-25 01:59:09 +0000207 .. method:: pop()
Georg Brandl116aa622007-08-15 14:28:22 +0000208
Benjamin Petersone41251e2008-04-25 01:59:09 +0000209 Remove and return an element from the right side of the deque. If no
210 elements are present, raises an :exc:`IndexError`.
Georg Brandl116aa622007-08-15 14:28:22 +0000211
212
Benjamin Petersone41251e2008-04-25 01:59:09 +0000213 .. method:: popleft()
Georg Brandl116aa622007-08-15 14:28:22 +0000214
Benjamin Petersone41251e2008-04-25 01:59:09 +0000215 Remove and return an element from the left side of the deque. If no
216 elements are present, raises an :exc:`IndexError`.
Georg Brandl116aa622007-08-15 14:28:22 +0000217
218
Benjamin Petersone41251e2008-04-25 01:59:09 +0000219 .. method:: remove(value)
Georg Brandl116aa622007-08-15 14:28:22 +0000220
Benjamin Petersone41251e2008-04-25 01:59:09 +0000221 Removed the first occurrence of *value*. If not found, raises a
222 :exc:`ValueError`.
Georg Brandl116aa622007-08-15 14:28:22 +0000223
Georg Brandl116aa622007-08-15 14:28:22 +0000224
Benjamin Petersone41251e2008-04-25 01:59:09 +0000225 .. method:: rotate(n)
Georg Brandl116aa622007-08-15 14:28:22 +0000226
Benjamin Petersone41251e2008-04-25 01:59:09 +0000227 Rotate the deque *n* steps to the right. If *n* is negative, rotate to
228 the left. Rotating one step to the right is equivalent to:
229 ``d.appendleft(d.pop())``.
230
Georg Brandl116aa622007-08-15 14:28:22 +0000231
232In addition to the above, deques support iteration, pickling, ``len(d)``,
233``reversed(d)``, ``copy.copy(d)``, ``copy.deepcopy(d)``, membership testing with
234the :keyword:`in` operator, and subscript references such as ``d[-1]``.
235
Christian Heimesfe337bf2008-03-23 21:54:12 +0000236Example:
237
238.. doctest::
Georg Brandl116aa622007-08-15 14:28:22 +0000239
240 >>> from collections import deque
241 >>> d = deque('ghi') # make a new deque with three items
242 >>> for elem in d: # iterate over the deque's elements
Neal Norwitz752abd02008-05-13 04:55:24 +0000243 ... print(elem.upper())
Georg Brandl116aa622007-08-15 14:28:22 +0000244 G
245 H
246 I
247
248 >>> d.append('j') # add a new entry to the right side
249 >>> d.appendleft('f') # add a new entry to the left side
250 >>> d # show the representation of the deque
251 deque(['f', 'g', 'h', 'i', 'j'])
252
253 >>> d.pop() # return and remove the rightmost item
254 'j'
255 >>> d.popleft() # return and remove the leftmost item
256 'f'
257 >>> list(d) # list the contents of the deque
258 ['g', 'h', 'i']
259 >>> d[0] # peek at leftmost item
260 'g'
261 >>> d[-1] # peek at rightmost item
262 'i'
263
264 >>> list(reversed(d)) # list the contents of a deque in reverse
265 ['i', 'h', 'g']
266 >>> 'h' in d # search the deque
267 True
268 >>> d.extend('jkl') # add multiple elements at once
269 >>> d
270 deque(['g', 'h', 'i', 'j', 'k', 'l'])
271 >>> d.rotate(1) # right rotation
272 >>> d
273 deque(['l', 'g', 'h', 'i', 'j', 'k'])
274 >>> d.rotate(-1) # left rotation
275 >>> d
276 deque(['g', 'h', 'i', 'j', 'k', 'l'])
277
278 >>> deque(reversed(d)) # make a new deque in reverse order
279 deque(['l', 'k', 'j', 'i', 'h', 'g'])
280 >>> d.clear() # empty the deque
281 >>> d.pop() # cannot pop from an empty deque
282 Traceback (most recent call last):
283 File "<pyshell#6>", line 1, in -toplevel-
284 d.pop()
285 IndexError: pop from an empty deque
286
287 >>> d.extendleft('abc') # extendleft() reverses the input order
288 >>> d
289 deque(['c', 'b', 'a'])
290
291
292.. _deque-recipes:
293
Georg Brandl9afde1c2007-11-01 20:32:30 +0000294:class:`deque` Recipes
295^^^^^^^^^^^^^^^^^^^^^^
Georg Brandl116aa622007-08-15 14:28:22 +0000296
297This section shows various approaches to working with deques.
298
299The :meth:`rotate` method provides a way to implement :class:`deque` slicing and
300deletion. For example, a pure python implementation of ``del d[n]`` relies on
301the :meth:`rotate` method to position elements to be popped::
302
303 def delete_nth(d, n):
304 d.rotate(-n)
305 d.popleft()
306 d.rotate(n)
307
308To implement :class:`deque` slicing, use a similar approach applying
309:meth:`rotate` to bring a target element to the left side of the deque. Remove
310old entries with :meth:`popleft`, add new entries with :meth:`extend`, and then
311reverse the rotation.
Georg Brandl116aa622007-08-15 14:28:22 +0000312With minor variations on that approach, it is easy to implement Forth style
313stack manipulations such as ``dup``, ``drop``, ``swap``, ``over``, ``pick``,
314``rot``, and ``roll``.
315
Georg Brandl116aa622007-08-15 14:28:22 +0000316Multi-pass data reduction algorithms can be succinctly expressed and efficiently
317coded by extracting elements with multiple calls to :meth:`popleft`, applying
Georg Brandl9afde1c2007-11-01 20:32:30 +0000318a reduction function, and calling :meth:`append` to add the result back to the
319deque.
Georg Brandl116aa622007-08-15 14:28:22 +0000320
321For example, building a balanced binary tree of nested lists entails reducing
Christian Heimesfe337bf2008-03-23 21:54:12 +0000322two adjacent nodes into one by grouping them in a list:
Georg Brandl116aa622007-08-15 14:28:22 +0000323
324 >>> def maketree(iterable):
325 ... d = deque(iterable)
326 ... while len(d) > 1:
327 ... pair = [d.popleft(), d.popleft()]
328 ... d.append(pair)
329 ... return list(d)
330 ...
Georg Brandl6911e3c2007-09-04 07:15:32 +0000331 >>> print(maketree('abcdefgh'))
Georg Brandl116aa622007-08-15 14:28:22 +0000332 [[[['a', 'b'], ['c', 'd']], [['e', 'f'], ['g', 'h']]]]
333
Georg Brandl9afde1c2007-11-01 20:32:30 +0000334Bounded length deques provide functionality similar to the ``tail`` filter
335in Unix::
Georg Brandl116aa622007-08-15 14:28:22 +0000336
Georg Brandl9afde1c2007-11-01 20:32:30 +0000337 def tail(filename, n=10):
338 'Return the last n lines of a file'
339 return deque(open(filename), n)
Georg Brandl116aa622007-08-15 14:28:22 +0000340
341.. _defaultdict-objects:
342
343:class:`defaultdict` objects
344----------------------------
345
346
347.. class:: defaultdict([default_factory[, ...]])
348
349 Returns a new dictionary-like object. :class:`defaultdict` is a subclass of the
350 builtin :class:`dict` class. It overrides one method and adds one writable
351 instance variable. The remaining functionality is the same as for the
352 :class:`dict` class and is not documented here.
353
354 The first argument provides the initial value for the :attr:`default_factory`
355 attribute; it defaults to ``None``. All remaining arguments are treated the same
356 as if they were passed to the :class:`dict` constructor, including keyword
357 arguments.
358
Georg Brandl116aa622007-08-15 14:28:22 +0000359
Benjamin Petersone41251e2008-04-25 01:59:09 +0000360 :class:`defaultdict` objects support the following method in addition to the
361 standard :class:`dict` operations:
Georg Brandl116aa622007-08-15 14:28:22 +0000362
Benjamin Petersone41251e2008-04-25 01:59:09 +0000363 .. method:: defaultdict.__missing__(key)
Georg Brandl116aa622007-08-15 14:28:22 +0000364
Benjamin Petersone41251e2008-04-25 01:59:09 +0000365 If the :attr:`default_factory` attribute is ``None``, this raises an
366 :exc:`KeyError` exception with the *key* as argument.
Georg Brandl116aa622007-08-15 14:28:22 +0000367
Benjamin Petersone41251e2008-04-25 01:59:09 +0000368 If :attr:`default_factory` is not ``None``, it is called without arguments
369 to provide a default value for the given *key*, this value is inserted in
370 the dictionary for the *key*, and returned.
Georg Brandl116aa622007-08-15 14:28:22 +0000371
Benjamin Petersone41251e2008-04-25 01:59:09 +0000372 If calling :attr:`default_factory` raises an exception this exception is
373 propagated unchanged.
Georg Brandl116aa622007-08-15 14:28:22 +0000374
Benjamin Petersone41251e2008-04-25 01:59:09 +0000375 This method is called by the :meth:`__getitem__` method of the
376 :class:`dict` class when the requested key is not found; whatever it
377 returns or raises is then returned or raised by :meth:`__getitem__`.
Georg Brandl116aa622007-08-15 14:28:22 +0000378
379
Benjamin Petersone41251e2008-04-25 01:59:09 +0000380 :class:`defaultdict` objects support the following instance variable:
Georg Brandl116aa622007-08-15 14:28:22 +0000381
Benjamin Petersone41251e2008-04-25 01:59:09 +0000382
383 .. attribute:: defaultdict.default_factory
384
385 This attribute is used by the :meth:`__missing__` method; it is
386 initialized from the first argument to the constructor, if present, or to
387 ``None``, if absent.
Georg Brandl116aa622007-08-15 14:28:22 +0000388
389
390.. _defaultdict-examples:
391
392:class:`defaultdict` Examples
393^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
394
395Using :class:`list` as the :attr:`default_factory`, it is easy to group a
Christian Heimesfe337bf2008-03-23 21:54:12 +0000396sequence of key-value pairs into a dictionary of lists:
Georg Brandl116aa622007-08-15 14:28:22 +0000397
398 >>> s = [('yellow', 1), ('blue', 2), ('yellow', 3), ('blue', 4), ('red', 1)]
399 >>> d = defaultdict(list)
400 >>> for k, v in s:
401 ... d[k].append(v)
402 ...
403 >>> d.items()
404 [('blue', [2, 4]), ('red', [1]), ('yellow', [1, 3])]
405
406When each key is encountered for the first time, it is not already in the
407mapping; so an entry is automatically created using the :attr:`default_factory`
408function which returns an empty :class:`list`. The :meth:`list.append`
409operation then attaches the value to the new list. When keys are encountered
410again, the look-up proceeds normally (returning the list for that key) and the
411:meth:`list.append` operation adds another value to the list. This technique is
Christian Heimesfe337bf2008-03-23 21:54:12 +0000412simpler and faster than an equivalent technique using :meth:`dict.setdefault`:
Georg Brandl116aa622007-08-15 14:28:22 +0000413
414 >>> d = {}
415 >>> for k, v in s:
416 ... d.setdefault(k, []).append(v)
417 ...
418 >>> d.items()
419 [('blue', [2, 4]), ('red', [1]), ('yellow', [1, 3])]
420
421Setting the :attr:`default_factory` to :class:`int` makes the
422:class:`defaultdict` useful for counting (like a bag or multiset in other
Christian Heimesfe337bf2008-03-23 21:54:12 +0000423languages):
Georg Brandl116aa622007-08-15 14:28:22 +0000424
425 >>> s = 'mississippi'
426 >>> d = defaultdict(int)
427 >>> for k in s:
428 ... d[k] += 1
429 ...
430 >>> d.items()
431 [('i', 4), ('p', 2), ('s', 4), ('m', 1)]
432
433When a letter is first encountered, it is missing from the mapping, so the
434:attr:`default_factory` function calls :func:`int` to supply a default count of
435zero. The increment operation then builds up the count for each letter.
436
437The function :func:`int` which always returns zero is just a special case of
438constant functions. A faster and more flexible way to create constant functions
439is to use a lambda function which can supply any constant value (not just
Christian Heimesfe337bf2008-03-23 21:54:12 +0000440zero):
Georg Brandl116aa622007-08-15 14:28:22 +0000441
442 >>> def constant_factory(value):
443 ... return lambda: value
444 >>> d = defaultdict(constant_factory('<missing>'))
445 >>> d.update(name='John', action='ran')
446 >>> '%(name)s %(action)s to %(object)s' % d
447 'John ran to <missing>'
448
449Setting the :attr:`default_factory` to :class:`set` makes the
Christian Heimesfe337bf2008-03-23 21:54:12 +0000450:class:`defaultdict` useful for building a dictionary of sets:
Georg Brandl116aa622007-08-15 14:28:22 +0000451
452 >>> s = [('red', 1), ('blue', 2), ('red', 3), ('blue', 4), ('red', 1), ('blue', 4)]
453 >>> d = defaultdict(set)
454 >>> for k, v in s:
455 ... d[k].add(v)
456 ...
457 >>> d.items()
458 [('blue', set([2, 4])), ('red', set([1, 3]))]
459
460
461.. _named-tuple-factory:
462
Guido van Rossum3d392eb2007-11-16 00:35:22 +0000463:func:`namedtuple` Factory Function for Tuples with Named Fields
Christian Heimes790c8232008-01-07 21:14:23 +0000464----------------------------------------------------------------
Georg Brandl116aa622007-08-15 14:28:22 +0000465
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000466Named tuples assign meaning to each position in a tuple and allow for more readable,
467self-documenting code. They can be used wherever regular tuples are used, and
468they add the ability to access fields by name instead of position index.
Georg Brandl116aa622007-08-15 14:28:22 +0000469
Guido van Rossum3d392eb2007-11-16 00:35:22 +0000470.. function:: namedtuple(typename, fieldnames, [verbose])
Georg Brandl116aa622007-08-15 14:28:22 +0000471
472 Returns a new tuple subclass named *typename*. The new subclass is used to
Christian Heimesc3f30c42008-02-22 16:37:40 +0000473 create tuple-like objects that have fields accessible by attribute lookup as
Georg Brandl116aa622007-08-15 14:28:22 +0000474 well as being indexable and iterable. Instances of the subclass also have a
475 helpful docstring (with typename and fieldnames) and a helpful :meth:`__repr__`
476 method which lists the tuple contents in a ``name=value`` format.
477
Georg Brandl9afde1c2007-11-01 20:32:30 +0000478 The *fieldnames* are a single string with each fieldname separated by whitespace
Christian Heimes25bb7832008-01-11 16:17:00 +0000479 and/or commas, for example ``'x y'`` or ``'x, y'``. Alternatively, *fieldnames*
480 can be a sequence of strings such as ``['x', 'y']``.
Georg Brandl9afde1c2007-11-01 20:32:30 +0000481
482 Any valid Python identifier may be used for a fieldname except for names
Christian Heimes0449f632007-12-15 01:27:15 +0000483 starting with an underscore. Valid identifiers consist of letters, digits,
484 and underscores but do not start with a digit or underscore and cannot be
Georg Brandlf6945182008-02-01 11:56:49 +0000485 a :mod:`keyword` such as *class*, *for*, *return*, *global*, *pass*,
Georg Brandl9afde1c2007-11-01 20:32:30 +0000486 or *raise*.
Georg Brandl116aa622007-08-15 14:28:22 +0000487
Christian Heimes25bb7832008-01-11 16:17:00 +0000488 If *verbose* is true, the class definition is printed just before being built.
Georg Brandl116aa622007-08-15 14:28:22 +0000489
Georg Brandl9afde1c2007-11-01 20:32:30 +0000490 Named tuple instances do not have per-instance dictionaries, so they are
Thomas Wouters8ce81f72007-09-20 18:22:40 +0000491 lightweight and require no more memory than regular tuples.
Georg Brandl116aa622007-08-15 14:28:22 +0000492
Christian Heimesfe337bf2008-03-23 21:54:12 +0000493Example:
494
495.. doctest::
496 :options: +NORMALIZE_WHITESPACE
Georg Brandl116aa622007-08-15 14:28:22 +0000497
Guido van Rossum3d392eb2007-11-16 00:35:22 +0000498 >>> Point = namedtuple('Point', 'x y', verbose=True)
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000499 class Point(tuple):
500 'Point(x, y)'
Christian Heimesfe337bf2008-03-23 21:54:12 +0000501 <BLANKLINE>
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000502 __slots__ = ()
Christian Heimesfe337bf2008-03-23 21:54:12 +0000503 <BLANKLINE>
Christian Heimesfaf2f632008-01-06 16:59:19 +0000504 _fields = ('x', 'y')
Christian Heimesfe337bf2008-03-23 21:54:12 +0000505 <BLANKLINE>
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000506 def __new__(cls, x, y):
507 return tuple.__new__(cls, (x, y))
Christian Heimesfe337bf2008-03-23 21:54:12 +0000508 <BLANKLINE>
Christian Heimesfaf2f632008-01-06 16:59:19 +0000509 @classmethod
Christian Heimesfe337bf2008-03-23 21:54:12 +0000510 def _make(cls, iterable, new=tuple.__new__, len=len):
Christian Heimesfaf2f632008-01-06 16:59:19 +0000511 'Make a new Point object from a sequence or iterable'
Christian Heimesfe337bf2008-03-23 21:54:12 +0000512 result = new(cls, iterable)
Christian Heimesfaf2f632008-01-06 16:59:19 +0000513 if len(result) != 2:
514 raise TypeError('Expected 2 arguments, got %d' % len(result))
515 return result
Christian Heimesfe337bf2008-03-23 21:54:12 +0000516 <BLANKLINE>
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000517 def __repr__(self):
518 return 'Point(x=%r, y=%r)' % self
Christian Heimesfe337bf2008-03-23 21:54:12 +0000519 <BLANKLINE>
Christian Heimes99170a52007-12-19 02:07:34 +0000520 def _asdict(t):
Christian Heimes0449f632007-12-15 01:27:15 +0000521 'Return a new dict which maps field names to their values'
Christian Heimes99170a52007-12-19 02:07:34 +0000522 return {'x': t[0], 'y': t[1]}
Christian Heimesfe337bf2008-03-23 21:54:12 +0000523 <BLANKLINE>
Christian Heimes0449f632007-12-15 01:27:15 +0000524 def _replace(self, **kwds):
Guido van Rossum3d392eb2007-11-16 00:35:22 +0000525 'Return a new Point object replacing specified fields with new values'
Christian Heimesfaf2f632008-01-06 16:59:19 +0000526 result = self._make(map(kwds.pop, ('x', 'y'), self))
527 if kwds:
528 raise ValueError('Got unexpected field names: %r' % kwds.keys())
529 return result
Georg Brandlc28e1fa2008-06-10 19:20:26 +0000530 <BLANKLINE>
Benjamin Peterson41181742008-07-02 20:22:54 +0000531 def __getnewargs__(self):
532 return tuple(self)
Christian Heimesfe337bf2008-03-23 21:54:12 +0000533 <BLANKLINE>
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000534 x = property(itemgetter(0))
535 y = property(itemgetter(1))
Georg Brandl116aa622007-08-15 14:28:22 +0000536
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000537 >>> p = Point(11, y=22) # instantiate with positional or keyword arguments
Christian Heimes99170a52007-12-19 02:07:34 +0000538 >>> p[0] + p[1] # indexable like the plain tuple (11, 22)
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000539 33
540 >>> x, y = p # unpack like a regular tuple
541 >>> x, y
542 (11, 22)
Christian Heimesc3f30c42008-02-22 16:37:40 +0000543 >>> p.x + p.y # fields also accessible by name
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000544 33
545 >>> p # readable __repr__ with a name=value style
546 Point(x=11, y=22)
Georg Brandl116aa622007-08-15 14:28:22 +0000547
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000548Named tuples are especially useful for assigning field names to result tuples returned
549by the :mod:`csv` or :mod:`sqlite3` modules::
550
Guido van Rossum3d392eb2007-11-16 00:35:22 +0000551 EmployeeRecord = namedtuple('EmployeeRecord', 'name, age, title, department, paygrade')
Georg Brandl9afde1c2007-11-01 20:32:30 +0000552
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000553 import csv
Christian Heimesfaf2f632008-01-06 16:59:19 +0000554 for emp in map(EmployeeRecord._make, csv.reader(open("employees.csv", "rb"))):
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000555 print(emp.name, emp.title)
556
Georg Brandl9afde1c2007-11-01 20:32:30 +0000557 import sqlite3
558 conn = sqlite3.connect('/companydata')
559 cursor = conn.cursor()
560 cursor.execute('SELECT name, age, title, department, paygrade FROM employees')
Christian Heimesfaf2f632008-01-06 16:59:19 +0000561 for emp in map(EmployeeRecord._make, cursor.fetchall()):
Christian Heimes00412232008-01-10 16:02:19 +0000562 print(emp.name, emp.title)
Georg Brandl9afde1c2007-11-01 20:32:30 +0000563
Christian Heimes99170a52007-12-19 02:07:34 +0000564In addition to the methods inherited from tuples, named tuples support
Christian Heimes2380ac72008-01-09 00:17:24 +0000565three additional methods and one attribute. To prevent conflicts with
566field names, the method and attribute names start with an underscore.
Christian Heimes99170a52007-12-19 02:07:34 +0000567
Christian Heimes790c8232008-01-07 21:14:23 +0000568.. method:: somenamedtuple._make(iterable)
Christian Heimes99170a52007-12-19 02:07:34 +0000569
Christian Heimesfaf2f632008-01-06 16:59:19 +0000570 Class method that makes a new instance from an existing sequence or iterable.
Christian Heimes99170a52007-12-19 02:07:34 +0000571
Christian Heimesfe337bf2008-03-23 21:54:12 +0000572.. doctest::
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000573
Christian Heimesfaf2f632008-01-06 16:59:19 +0000574 >>> t = [11, 22]
575 >>> Point._make(t)
576 Point(x=11, y=22)
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000577
Christian Heimes790c8232008-01-07 21:14:23 +0000578.. method:: somenamedtuple._asdict()
Georg Brandl9afde1c2007-11-01 20:32:30 +0000579
Christian Heimesfe337bf2008-03-23 21:54:12 +0000580 Return a new dict which maps field names to their corresponding values::
Georg Brandl9afde1c2007-11-01 20:32:30 +0000581
Christian Heimes0449f632007-12-15 01:27:15 +0000582 >>> p._asdict()
Georg Brandl9afde1c2007-11-01 20:32:30 +0000583 {'x': 11, 'y': 22}
Christian Heimesfe337bf2008-03-23 21:54:12 +0000584
Christian Heimes790c8232008-01-07 21:14:23 +0000585.. method:: somenamedtuple._replace(kwargs)
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000586
Christian Heimesfe337bf2008-03-23 21:54:12 +0000587 Return a new instance of the named tuple replacing specified fields with new
588 values:
Thomas Wouters8ce81f72007-09-20 18:22:40 +0000589
590::
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000591
592 >>> p = Point(x=11, y=22)
Christian Heimes0449f632007-12-15 01:27:15 +0000593 >>> p._replace(x=33)
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000594 Point(x=33, y=22)
595
Guido van Rossum3d392eb2007-11-16 00:35:22 +0000596 >>> for partnum, record in inventory.items():
Christian Heimes454f37b2008-01-10 00:10:02 +0000597 ... inventory[partnum] = record._replace(price=newprices[partnum], timestamp=time.now())
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000598
Christian Heimes790c8232008-01-07 21:14:23 +0000599.. attribute:: somenamedtuple._fields
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000600
Christian Heimes2380ac72008-01-09 00:17:24 +0000601 Tuple of strings listing the field names. Useful for introspection
Georg Brandl9afde1c2007-11-01 20:32:30 +0000602 and for creating new named tuple types from existing named tuples.
Thomas Wouters8ce81f72007-09-20 18:22:40 +0000603
Christian Heimesfe337bf2008-03-23 21:54:12 +0000604.. doctest::
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000605
Christian Heimes0449f632007-12-15 01:27:15 +0000606 >>> p._fields # view the field names
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000607 ('x', 'y')
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000608
Guido van Rossum3d392eb2007-11-16 00:35:22 +0000609 >>> Color = namedtuple('Color', 'red green blue')
Christian Heimes0449f632007-12-15 01:27:15 +0000610 >>> Pixel = namedtuple('Pixel', Point._fields + Color._fields)
Thomas Wouters1b7f8912007-09-19 03:06:30 +0000611 >>> Pixel(11, 22, 128, 255, 0)
Christian Heimes454f37b2008-01-10 00:10:02 +0000612 Pixel(x=11, y=22, red=128, green=255, blue=0)
Georg Brandl116aa622007-08-15 14:28:22 +0000613
Christian Heimes0449f632007-12-15 01:27:15 +0000614To retrieve a field whose name is stored in a string, use the :func:`getattr`
Christian Heimesfe337bf2008-03-23 21:54:12 +0000615function:
Christian Heimes0449f632007-12-15 01:27:15 +0000616
617 >>> getattr(p, 'x')
618 11
619
Christian Heimesfe337bf2008-03-23 21:54:12 +0000620To convert a dictionary to a named tuple, use the double-star-operator [#]_:
Christian Heimes99170a52007-12-19 02:07:34 +0000621
622 >>> d = {'x': 11, 'y': 22}
623 >>> Point(**d)
624 Point(x=11, y=22)
625
Guido van Rossum3d392eb2007-11-16 00:35:22 +0000626Since a named tuple is a regular Python class, it is easy to add or change
Christian Heimes043d6f62008-01-07 17:19:16 +0000627functionality with a subclass. Here is how to add a calculated field and
Christian Heimesfe337bf2008-03-23 21:54:12 +0000628a fixed-width print format:
Guido van Rossum3d392eb2007-11-16 00:35:22 +0000629
Christian Heimes043d6f62008-01-07 17:19:16 +0000630 >>> class Point(namedtuple('Point', 'x y')):
Christian Heimes25bb7832008-01-11 16:17:00 +0000631 ... __slots__ = ()
Christian Heimes454f37b2008-01-10 00:10:02 +0000632 ... @property
633 ... def hypot(self):
634 ... return (self.x ** 2 + self.y ** 2) ** 0.5
635 ... def __str__(self):
Christian Heimes25bb7832008-01-11 16:17:00 +0000636 ... return 'Point: x=%6.3f y=%6.3f hypot=%6.3f' % (self.x, self.y, self.hypot)
Guido van Rossum3d392eb2007-11-16 00:35:22 +0000637
Christian Heimes25bb7832008-01-11 16:17:00 +0000638 >>> for p in Point(3, 4), Point(14, 5/7.):
Christian Heimes00412232008-01-10 16:02:19 +0000639 ... print(p)
Christian Heimes25bb7832008-01-11 16:17:00 +0000640 Point: x= 3.000 y= 4.000 hypot= 5.000
641 Point: x=14.000 y= 0.714 hypot=14.018
Christian Heimes043d6f62008-01-07 17:19:16 +0000642
Christian Heimesaf98da12008-01-27 15:18:18 +0000643The subclass shown above sets ``__slots__`` to an empty tuple. This keeps
Christian Heimes679db4a2008-01-18 09:56:22 +0000644keep memory requirements low by preventing the creation of instance dictionaries.
645
Christian Heimes2380ac72008-01-09 00:17:24 +0000646
647Subclassing is not useful for adding new, stored fields. Instead, simply
Christian Heimesfe337bf2008-03-23 21:54:12 +0000648create a new named tuple type from the :attr:`_fields` attribute:
Christian Heimes2380ac72008-01-09 00:17:24 +0000649
Christian Heimes25bb7832008-01-11 16:17:00 +0000650 >>> Point3D = namedtuple('Point3D', Point._fields + ('z',))
Christian Heimes2380ac72008-01-09 00:17:24 +0000651
652Default values can be implemented by using :meth:`_replace` to
Christian Heimesfe337bf2008-03-23 21:54:12 +0000653customize a prototype instance:
Guido van Rossum3d392eb2007-11-16 00:35:22 +0000654
655 >>> Account = namedtuple('Account', 'owner balance transaction_count')
Christian Heimes587c2bf2008-01-19 16:21:02 +0000656 >>> default_account = Account('<owner name>', 0.0, 0)
657 >>> johns_account = default_account._replace(owner='John')
Guido van Rossum3d392eb2007-11-16 00:35:22 +0000658
Christian Heimese4ca8152008-05-08 17:18:53 +0000659Enumerated constants can be implemented with named tuples, but it is simpler
660and more efficient to use a simple class declaration:
661
662 >>> Status = namedtuple('Status', 'open pending closed')._make(range(3))
663 >>> Status.open, Status.pending, Status.closed
664 (0, 1, 2)
665 >>> class Status:
666 ... open, pending, closed = range(3)
667
Thomas Wouters47b49bf2007-08-30 22:15:33 +0000668.. rubric:: Footnotes
669
Christian Heimes99170a52007-12-19 02:07:34 +0000670.. [#] For information on the double-star-operator see
Thomas Wouters47b49bf2007-08-30 22:15:33 +0000671 :ref:`tut-unpacking-arguments` and :ref:`calls`.
Raymond Hettingere4c96ad2008-02-06 01:23:58 +0000672
673
674
675:class:`UserDict` objects
Mark Summerfield8f2d0062008-02-06 13:30:44 +0000676-------------------------
Raymond Hettingere4c96ad2008-02-06 01:23:58 +0000677
678The class, :class:`UserDict` acts as a wrapper around dictionary objects.
679The need for this class has been partially supplanted by the ability to
680subclass directly from :class:`dict`; however, this class can be easier
681to work with because the underlying dictionary is accessible as an
682attribute.
683
684.. class:: UserDict([initialdata])
685
686 Class that simulates a dictionary. The instance's contents are kept in a
687 regular dictionary, which is accessible via the :attr:`data` attribute of
688 :class:`UserDict` instances. If *initialdata* is provided, :attr:`data` is
689 initialized with its contents; note that a reference to *initialdata* will not
690 be kept, allowing it be used for other purposes.
691
692In addition to supporting the methods and operations of mappings,
Raymond Hettingerebcee3f2008-02-06 19:54:00 +0000693:class:`UserDict` instances provide the following attribute:
Raymond Hettingere4c96ad2008-02-06 01:23:58 +0000694
695.. attribute:: UserDict.data
696
697 A real dictionary used to store the contents of the :class:`UserDict` class.
Raymond Hettinger53dbe392008-02-12 20:03:09 +0000698
699
700
701:class:`UserList` objects
702-------------------------
703
704This class acts as a wrapper around list objects. It is a useful base class
705for your own list-like classes which can inherit from them and override
706existing methods or add new ones. In this way, one can add new behaviors to
707lists.
708
709The need for this class has been partially supplanted by the ability to
710subclass directly from :class:`list`; however, this class can be easier
711to work with because the underlying list is accessible as an attribute.
712
713.. class:: UserList([list])
714
715 Class that simulates a list. The instance's contents are kept in a regular
716 list, which is accessible via the :attr:`data` attribute of :class:`UserList`
717 instances. The instance's contents are initially set to a copy of *list*,
718 defaulting to the empty list ``[]``. *list* can be any iterable, for
719 example a real Python list or a :class:`UserList` object.
720
721In addition to supporting the methods and operations of mutable sequences,
722:class:`UserList` instances provide the following attribute:
723
724.. attribute:: UserList.data
725
726 A real :class:`list` object used to store the contents of the
727 :class:`UserList` class.
728
729**Subclassing requirements:** Subclasses of :class:`UserList` are expect to
730offer a constructor which can be called with either no arguments or one
731argument. List operations which return a new sequence attempt to create an
732instance of the actual implementation class. To do so, it assumes that the
733constructor can be called with a single parameter, which is a sequence object
734used as a data source.
735
736If a derived class does not wish to comply with this requirement, all of the
737special methods supported by this class will need to be overridden; please
738consult the sources for information about the methods which need to be provided
739in that case.
Raymond Hettingerb3a65f82008-02-21 22:11:37 +0000740
741:class:`UserString` objects
Christian Heimesc3f30c42008-02-22 16:37:40 +0000742---------------------------
Raymond Hettingerb3a65f82008-02-21 22:11:37 +0000743
744The class, :class:`UserString` acts as a wrapper around string objects.
745The need for this class has been partially supplanted by the ability to
746subclass directly from :class:`str`; however, this class can be easier
747to work with because the underlying string is accessible as an
748attribute.
749
750.. class:: UserString([sequence])
751
752 Class that simulates a string or a Unicode string object. The instance's
753 content is kept in a regular string object, which is accessible via the
754 :attr:`data` attribute of :class:`UserString` instances. The instance's
755 contents are initially set to a copy of *sequence*. The *sequence* can
756 be an instance of :class:`bytes`, :class:`str`, :class:`UserString` (or a
757 subclass) or an arbitrary sequence which can be converted into a string using
758 the built-in :func:`str` function.