Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 1 | :mod:`shelve` --- Python object persistence |
| 2 | =========================================== |
| 3 | |
| 4 | .. module:: shelve |
| 5 | :synopsis: Python object persistence. |
| 6 | |
| 7 | |
| 8 | .. index:: module: pickle |
| 9 | |
Raymond Hettinger | 1048094 | 2011-01-10 03:26:08 +0000 | [diff] [blame] | 10 | **Source code:** :source:`Lib/shelve.py` |
| 11 | |
Raymond Hettinger | 4f707fd | 2011-01-10 19:54:11 +0000 | [diff] [blame] | 12 | -------------- |
| 13 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 14 | A "shelf" is a persistent, dictionary-like object. The difference with "dbm" |
| 15 | databases is that the values (not the keys!) in a shelf can be essentially |
| 16 | arbitrary Python objects --- anything that the :mod:`pickle` module can handle. |
| 17 | This includes most class instances, recursive data types, and objects containing |
| 18 | lots of shared sub-objects. The keys are ordinary strings. |
| 19 | |
| 20 | |
Georg Brandl | 1824415 | 2009-09-02 20:34:52 +0000 | [diff] [blame] | 21 | .. function:: open(filename, flag='c', protocol=None, writeback=False) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 22 | |
| 23 | Open a persistent dictionary. The filename specified is the base filename for |
| 24 | the underlying database. As a side-effect, an extension may be added to the |
| 25 | filename and more than one file may be created. By default, the underlying |
| 26 | database file is opened for reading and writing. The optional *flag* parameter |
Georg Brandl | 0a7ac7d | 2008-05-26 10:29:35 +0000 | [diff] [blame] | 27 | has the same interpretation as the *flag* parameter of :func:`dbm.open`. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 28 | |
Raymond Hettinger | 8560226 | 2009-02-03 04:19:10 +0000 | [diff] [blame] | 29 | By default, version 3 pickles are used to serialize values. The version of the |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 30 | pickle protocol can be specified with the *protocol* parameter. |
| 31 | |
R. David Murray | ff85bca | 2009-05-12 01:40:16 +0000 | [diff] [blame] | 32 | Because of Python semantics, a shelf cannot know when a mutable |
| 33 | persistent-dictionary entry is modified. By default modified objects are |
R. David Murray | ddb3ed0 | 2010-02-11 02:42:19 +0000 | [diff] [blame] | 34 | written *only* when assigned to the shelf (see :ref:`shelve-example`). If the |
| 35 | optional *writeback* parameter is set to *True*, all entries accessed are also |
| 36 | cached in memory, and written back on :meth:`~Shelf.sync` and |
| 37 | :meth:`~Shelf.close`; this can make it handier to mutate mutable entries in |
| 38 | the persistent dictionary, but, if many entries are accessed, it can consume |
| 39 | vast amounts of memory for the cache, and it can make the close operation |
| 40 | very slow since all accessed entries are written back (there is no way to |
| 41 | determine which accessed entries are mutable, nor which ones were actually |
| 42 | mutated). |
Benjamin Peterson | 4ac9ce4 | 2009-10-04 14:49:41 +0000 | [diff] [blame] | 43 | |
| 44 | .. note:: |
| 45 | |
| 46 | Do not rely on the shelf being closed automatically; always call |
Ezio Melotti | d23c0a8 | 2013-02-01 05:01:50 +0200 | [diff] [blame] | 47 | :meth:`~Shelf.close` explicitly when you don't need it any more, or |
| 48 | use :func:`shelve.open` as a context manager:: |
| 49 | |
| 50 | with shelve.open('spam') as db: |
| 51 | db['eggs'] = 'eggs' |
Benjamin Peterson | 4ac9ce4 | 2009-10-04 14:49:41 +0000 | [diff] [blame] | 52 | |
Georg Brandl | 7716ca6 | 2010-10-17 09:37:54 +0000 | [diff] [blame] | 53 | .. warning:: |
| 54 | |
| 55 | Because the :mod:`shelve` module is backed by :mod:`pickle`, it is insecure |
| 56 | to load a shelf from an untrusted source. Like with pickle, loading a shelf |
| 57 | can execute arbitrary code. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 58 | |
Benjamin Peterson | 25c95f1 | 2009-05-08 20:42:26 +0000 | [diff] [blame] | 59 | Shelf objects support all methods supported by dictionaries. This eases the |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 60 | transition from dictionary based scripts to those requiring persistent storage. |
| 61 | |
Benjamin Peterson | 4ac9ce4 | 2009-10-04 14:49:41 +0000 | [diff] [blame] | 62 | Two additional methods are supported: |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 63 | |
| 64 | .. method:: Shelf.sync() |
| 65 | |
Benjamin Peterson | 4ac9ce4 | 2009-10-04 14:49:41 +0000 | [diff] [blame] | 66 | Write back all entries in the cache if the shelf was opened with *writeback* |
| 67 | set to :const:`True`. Also empty the cache and synchronize the persistent |
| 68 | dictionary on disk, if feasible. This is called automatically when the shelf |
| 69 | is closed with :meth:`close`. |
| 70 | |
| 71 | .. method:: Shelf.close() |
| 72 | |
| 73 | Synchronize and close the persistent *dict* object. Operations on a closed |
| 74 | shelf will fail with a :exc:`ValueError`. |
| 75 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 76 | |
Raymond Hettinger | 65c9eb2 | 2009-04-04 05:39:52 +0000 | [diff] [blame] | 77 | .. seealso:: |
| 78 | |
| 79 | `Persistent dictionary recipe <http://code.activestate.com/recipes/576642/>`_ |
| 80 | with widely supported storage formats and having the speed of native |
| 81 | dictionaries. |
| 82 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 83 | |
| 84 | Restrictions |
| 85 | ------------ |
| 86 | |
| 87 | .. index:: |
Georg Brandl | 0a7ac7d | 2008-05-26 10:29:35 +0000 | [diff] [blame] | 88 | module: dbm.ndbm |
| 89 | module: dbm.gnu |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 90 | |
Benjamin Peterson | 9a46cab | 2008-09-08 02:49:30 +0000 | [diff] [blame] | 91 | * The choice of which database package will be used (such as :mod:`dbm.ndbm` or |
| 92 | :mod:`dbm.gnu`) depends on which interface is available. Therefore it is not |
| 93 | safe to open the database directly using :mod:`dbm`. The database is also |
| 94 | (unfortunately) subject to the limitations of :mod:`dbm`, if it is used --- |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 95 | this means that (the pickled representation of) the objects stored in the |
Benjamin Peterson | 9a46cab | 2008-09-08 02:49:30 +0000 | [diff] [blame] | 96 | database should be fairly small, and in rare cases key collisions may cause |
| 97 | the database to refuse updates. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 98 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 99 | * The :mod:`shelve` module does not support *concurrent* read/write access to |
| 100 | shelved objects. (Multiple simultaneous read accesses are safe.) When a |
| 101 | program has a shelf open for writing, no other program should have it open for |
| 102 | reading or writing. Unix file locking can be used to solve this, but this |
| 103 | differs across Unix versions and requires knowledge about the database |
| 104 | implementation used. |
| 105 | |
| 106 | |
Georg Brandl | 732324a | 2010-12-04 11:12:43 +0000 | [diff] [blame] | 107 | .. class:: Shelf(dict, protocol=None, writeback=False, keyencoding='utf-8') |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 108 | |
Serhiy Storchaka | bfdcd43 | 2013-10-13 23:09:14 +0300 | [diff] [blame] | 109 | A subclass of :class:`collections.abc.MutableMapping` which stores pickled |
| 110 | values in the *dict* object. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 111 | |
| 112 | By default, version 0 pickles are used to serialize values. The version of the |
| 113 | pickle protocol can be specified with the *protocol* parameter. See the |
| 114 | :mod:`pickle` documentation for a discussion of the pickle protocols. |
| 115 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 116 | If the *writeback* parameter is ``True``, the object will hold a cache of all |
| 117 | entries accessed and write them back to the *dict* at sync and close times. |
| 118 | This allows natural operations on mutable entries, but can consume much more |
| 119 | memory and make sync and close take a long time. |
| 120 | |
Georg Brandl | 732324a | 2010-12-04 11:12:43 +0000 | [diff] [blame] | 121 | The *keyencoding* parameter is the encoding used to encode keys before they |
| 122 | are used with the underlying dict. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 123 | |
R David Murray | 575fb31 | 2013-12-25 23:21:03 -0500 | [diff] [blame] | 124 | A :class:`Shelf` object can also be used as a context manager, in which |
| 125 | case it will be automatically closed when the :keyword:`with` block ends. |
Ezio Melotti | d23c0a8 | 2013-02-01 05:01:50 +0200 | [diff] [blame] | 126 | |
| 127 | .. versionchanged:: 3.2 |
| 128 | Added the *keyencoding* parameter; previously, keys were always encoded in |
Georg Brandl | 732324a | 2010-12-04 11:12:43 +0000 | [diff] [blame] | 129 | UTF-8. |
| 130 | |
Ezio Melotti | d23c0a8 | 2013-02-01 05:01:50 +0200 | [diff] [blame] | 131 | .. versionchanged:: 3.4 |
| 132 | Added context manager support. |
| 133 | |
Georg Brandl | 732324a | 2010-12-04 11:12:43 +0000 | [diff] [blame] | 134 | |
| 135 | .. class:: BsdDbShelf(dict, protocol=None, writeback=False, keyencoding='utf-8') |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 136 | |
Georg Brandl | 502d9a5 | 2009-07-26 15:02:41 +0000 | [diff] [blame] | 137 | A subclass of :class:`Shelf` which exposes :meth:`first`, :meth:`!next`, |
Georg Brandl | 1158a33 | 2009-06-04 09:30:30 +0000 | [diff] [blame] | 138 | :meth:`previous`, :meth:`last` and :meth:`set_location` which are available |
| 139 | in the third-party :mod:`bsddb` module from `pybsddb |
| 140 | <http://www.jcea.es/programacion/pybsddb.htm>`_ but not in other database |
| 141 | modules. The *dict* object passed to the constructor must support those |
| 142 | methods. This is generally accomplished by calling one of |
| 143 | :func:`bsddb.hashopen`, :func:`bsddb.btopen` or :func:`bsddb.rnopen`. The |
Georg Brandl | 732324a | 2010-12-04 11:12:43 +0000 | [diff] [blame] | 144 | optional *protocol*, *writeback*, and *keyencoding* parameters have the same |
| 145 | interpretation as for the :class:`Shelf` class. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 146 | |
| 147 | |
Georg Brandl | 1824415 | 2009-09-02 20:34:52 +0000 | [diff] [blame] | 148 | .. class:: DbfilenameShelf(filename, flag='c', protocol=None, writeback=False) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 149 | |
| 150 | A subclass of :class:`Shelf` which accepts a *filename* instead of a dict-like |
Georg Brandl | 0a7ac7d | 2008-05-26 10:29:35 +0000 | [diff] [blame] | 151 | object. The underlying file will be opened using :func:`dbm.open`. By |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 152 | default, the file will be created and opened for both read and write. The |
Georg Brandl | 502d9a5 | 2009-07-26 15:02:41 +0000 | [diff] [blame] | 153 | optional *flag* parameter has the same interpretation as for the :func:`.open` |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 154 | function. The optional *protocol* and *writeback* parameters have the same |
| 155 | interpretation as for the :class:`Shelf` class. |
| 156 | |
| 157 | |
R. David Murray | ff85bca | 2009-05-12 01:40:16 +0000 | [diff] [blame] | 158 | .. _shelve-example: |
| 159 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 160 | Example |
| 161 | ------- |
| 162 | |
| 163 | To summarize the interface (``key`` is a string, ``data`` is an arbitrary |
| 164 | object):: |
| 165 | |
| 166 | import shelve |
| 167 | |
| 168 | d = shelve.open(filename) # open -- file may get suffix added by low-level |
| 169 | # library |
| 170 | |
| 171 | d[key] = data # store data at key (overwrites old data if |
| 172 | # using an existing key) |
| 173 | data = d[key] # retrieve a COPY of data at key (raise KeyError if no |
| 174 | # such key) |
| 175 | del d[key] # delete data stored at key (raises KeyError |
| 176 | # if no such key) |
Ezio Melotti | 8f7649e | 2009-09-13 04:48:45 +0000 | [diff] [blame] | 177 | flag = key in d # true if the key exists |
| 178 | klist = list(d.keys()) # a list of all existing keys (slow!) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 179 | |
| 180 | # as d was opened WITHOUT writeback=True, beware: |
Antoine Pitrou | 631507d | 2011-02-07 23:10:33 +0000 | [diff] [blame] | 181 | d['xx'] = [0, 1, 2] # this works as expected, but... |
| 182 | d['xx'].append(3) # *this doesn't!* -- d['xx'] is STILL [0, 1, 2]! |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 183 | |
| 184 | # having opened d without writeback=True, you need to code carefully: |
| 185 | temp = d['xx'] # extracts the copy |
| 186 | temp.append(5) # mutates the copy |
| 187 | d['xx'] = temp # stores the copy right back, to persist it |
| 188 | |
| 189 | # or, d=shelve.open(filename,writeback=True) would let you just code |
| 190 | # d['xx'].append(5) and have it work as expected, BUT it would also |
| 191 | # consume more memory and make the d.close() operation slower. |
| 192 | |
| 193 | d.close() # close it |
| 194 | |
| 195 | |
| 196 | .. seealso:: |
| 197 | |
Georg Brandl | 0a7ac7d | 2008-05-26 10:29:35 +0000 | [diff] [blame] | 198 | Module :mod:`dbm` |
| 199 | Generic interface to ``dbm``-style databases. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 200 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 201 | Module :mod:`pickle` |
| 202 | Object serialization used by :mod:`shelve`. |
| 203 | |