Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 1 | :mod:`shelve` --- Python object persistence |
| 2 | =========================================== |
| 3 | |
| 4 | .. module:: shelve |
| 5 | :synopsis: Python object persistence. |
| 6 | |
Terry Jan Reedy | fa089b9 | 2016-06-11 15:02:54 -0400 | [diff] [blame] | 7 | **Source code:** :source:`Lib/shelve.py` |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 8 | |
| 9 | .. index:: module: pickle |
| 10 | |
Raymond Hettinger | 4f707fd | 2011-01-10 19:54:11 +0000 | [diff] [blame] | 11 | -------------- |
| 12 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 13 | A "shelf" is a persistent, dictionary-like object. The difference with "dbm" |
| 14 | databases is that the values (not the keys!) in a shelf can be essentially |
| 15 | arbitrary Python objects --- anything that the :mod:`pickle` module can handle. |
| 16 | This includes most class instances, recursive data types, and objects containing |
| 17 | lots of shared sub-objects. The keys are ordinary strings. |
| 18 | |
| 19 | |
Georg Brandl | 1824415 | 2009-09-02 20:34:52 +0000 | [diff] [blame] | 20 | .. function:: open(filename, flag='c', protocol=None, writeback=False) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 21 | |
| 22 | Open a persistent dictionary. The filename specified is the base filename for |
| 23 | the underlying database. As a side-effect, an extension may be added to the |
| 24 | filename and more than one file may be created. By default, the underlying |
| 25 | database file is opened for reading and writing. The optional *flag* parameter |
Georg Brandl | 0a7ac7d | 2008-05-26 10:29:35 +0000 | [diff] [blame] | 26 | has the same interpretation as the *flag* parameter of :func:`dbm.open`. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 27 | |
Raymond Hettinger | 8560226 | 2009-02-03 04:19:10 +0000 | [diff] [blame] | 28 | By default, version 3 pickles are used to serialize values. The version of the |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 29 | pickle protocol can be specified with the *protocol* parameter. |
| 30 | |
R. David Murray | ff85bca | 2009-05-12 01:40:16 +0000 | [diff] [blame] | 31 | Because of Python semantics, a shelf cannot know when a mutable |
| 32 | persistent-dictionary entry is modified. By default modified objects are |
R. David Murray | ddb3ed0 | 2010-02-11 02:42:19 +0000 | [diff] [blame] | 33 | written *only* when assigned to the shelf (see :ref:`shelve-example`). If the |
Serhiy Storchaka | a97cd2e | 2016-10-19 16:43:42 +0300 | [diff] [blame] | 34 | optional *writeback* parameter is set to ``True``, all entries accessed are also |
R. David Murray | ddb3ed0 | 2010-02-11 02:42:19 +0000 | [diff] [blame] | 35 | cached in memory, and written back on :meth:`~Shelf.sync` and |
| 36 | :meth:`~Shelf.close`; this can make it handier to mutate mutable entries in |
| 37 | the persistent dictionary, but, if many entries are accessed, it can consume |
| 38 | vast amounts of memory for the cache, and it can make the close operation |
| 39 | very slow since all accessed entries are written back (there is no way to |
| 40 | determine which accessed entries are mutable, nor which ones were actually |
| 41 | mutated). |
Benjamin Peterson | 4ac9ce4 | 2009-10-04 14:49:41 +0000 | [diff] [blame] | 42 | |
| 43 | .. note:: |
| 44 | |
| 45 | Do not rely on the shelf being closed automatically; always call |
Ezio Melotti | d23c0a8 | 2013-02-01 05:01:50 +0200 | [diff] [blame] | 46 | :meth:`~Shelf.close` explicitly when you don't need it any more, or |
| 47 | use :func:`shelve.open` as a context manager:: |
| 48 | |
| 49 | with shelve.open('spam') as db: |
| 50 | db['eggs'] = 'eggs' |
Benjamin Peterson | 4ac9ce4 | 2009-10-04 14:49:41 +0000 | [diff] [blame] | 51 | |
Georg Brandl | 7716ca6 | 2010-10-17 09:37:54 +0000 | [diff] [blame] | 52 | .. warning:: |
| 53 | |
| 54 | Because the :mod:`shelve` module is backed by :mod:`pickle`, it is insecure |
| 55 | to load a shelf from an untrusted source. Like with pickle, loading a shelf |
| 56 | can execute arbitrary code. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 57 | |
Benjamin Peterson | 25c95f1 | 2009-05-08 20:42:26 +0000 | [diff] [blame] | 58 | Shelf objects support all methods supported by dictionaries. This eases the |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 59 | transition from dictionary based scripts to those requiring persistent storage. |
| 60 | |
Benjamin Peterson | 4ac9ce4 | 2009-10-04 14:49:41 +0000 | [diff] [blame] | 61 | Two additional methods are supported: |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 62 | |
| 63 | .. method:: Shelf.sync() |
| 64 | |
Benjamin Peterson | 4ac9ce4 | 2009-10-04 14:49:41 +0000 | [diff] [blame] | 65 | Write back all entries in the cache if the shelf was opened with *writeback* |
| 66 | set to :const:`True`. Also empty the cache and synchronize the persistent |
| 67 | dictionary on disk, if feasible. This is called automatically when the shelf |
| 68 | is closed with :meth:`close`. |
| 69 | |
| 70 | .. method:: Shelf.close() |
| 71 | |
| 72 | Synchronize and close the persistent *dict* object. Operations on a closed |
| 73 | shelf will fail with a :exc:`ValueError`. |
| 74 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 75 | |
Raymond Hettinger | 65c9eb2 | 2009-04-04 05:39:52 +0000 | [diff] [blame] | 76 | .. seealso:: |
| 77 | |
Georg Brandl | 5d94134 | 2016-02-26 19:37:12 +0100 | [diff] [blame] | 78 | `Persistent dictionary recipe <https://code.activestate.com/recipes/576642/>`_ |
Raymond Hettinger | 65c9eb2 | 2009-04-04 05:39:52 +0000 | [diff] [blame] | 79 | with widely supported storage formats and having the speed of native |
| 80 | dictionaries. |
| 81 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 82 | |
| 83 | Restrictions |
| 84 | ------------ |
| 85 | |
| 86 | .. index:: |
Georg Brandl | 0a7ac7d | 2008-05-26 10:29:35 +0000 | [diff] [blame] | 87 | module: dbm.ndbm |
| 88 | module: dbm.gnu |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 89 | |
Benjamin Peterson | 9a46cab | 2008-09-08 02:49:30 +0000 | [diff] [blame] | 90 | * The choice of which database package will be used (such as :mod:`dbm.ndbm` or |
| 91 | :mod:`dbm.gnu`) depends on which interface is available. Therefore it is not |
| 92 | safe to open the database directly using :mod:`dbm`. The database is also |
| 93 | (unfortunately) subject to the limitations of :mod:`dbm`, if it is used --- |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 94 | this means that (the pickled representation of) the objects stored in the |
Benjamin Peterson | 9a46cab | 2008-09-08 02:49:30 +0000 | [diff] [blame] | 95 | database should be fairly small, and in rare cases key collisions may cause |
| 96 | the database to refuse updates. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 97 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 98 | * The :mod:`shelve` module does not support *concurrent* read/write access to |
| 99 | shelved objects. (Multiple simultaneous read accesses are safe.) When a |
| 100 | program has a shelf open for writing, no other program should have it open for |
| 101 | reading or writing. Unix file locking can be used to solve this, but this |
| 102 | differs across Unix versions and requires knowledge about the database |
| 103 | implementation used. |
| 104 | |
| 105 | |
Georg Brandl | 732324a | 2010-12-04 11:12:43 +0000 | [diff] [blame] | 106 | .. class:: Shelf(dict, protocol=None, writeback=False, keyencoding='utf-8') |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 107 | |
Serhiy Storchaka | bfdcd43 | 2013-10-13 23:09:14 +0300 | [diff] [blame] | 108 | A subclass of :class:`collections.abc.MutableMapping` which stores pickled |
| 109 | values in the *dict* object. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 110 | |
Berker Peksag | 8faca61 | 2016-07-01 12:33:00 +0300 | [diff] [blame] | 111 | By default, version 3 pickles are used to serialize values. The version of the |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 112 | pickle protocol can be specified with the *protocol* parameter. See the |
| 113 | :mod:`pickle` documentation for a discussion of the pickle protocols. |
| 114 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 115 | If the *writeback* parameter is ``True``, the object will hold a cache of all |
| 116 | entries accessed and write them back to the *dict* at sync and close times. |
| 117 | This allows natural operations on mutable entries, but can consume much more |
| 118 | memory and make sync and close take a long time. |
| 119 | |
Georg Brandl | 732324a | 2010-12-04 11:12:43 +0000 | [diff] [blame] | 120 | The *keyencoding* parameter is the encoding used to encode keys before they |
| 121 | are used with the underlying dict. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 122 | |
R David Murray | 575fb31 | 2013-12-25 23:21:03 -0500 | [diff] [blame] | 123 | A :class:`Shelf` object can also be used as a context manager, in which |
| 124 | case it will be automatically closed when the :keyword:`with` block ends. |
Ezio Melotti | d23c0a8 | 2013-02-01 05:01:50 +0200 | [diff] [blame] | 125 | |
| 126 | .. versionchanged:: 3.2 |
| 127 | Added the *keyencoding* parameter; previously, keys were always encoded in |
Georg Brandl | 732324a | 2010-12-04 11:12:43 +0000 | [diff] [blame] | 128 | UTF-8. |
| 129 | |
Ezio Melotti | d23c0a8 | 2013-02-01 05:01:50 +0200 | [diff] [blame] | 130 | .. versionchanged:: 3.4 |
| 131 | Added context manager support. |
| 132 | |
Georg Brandl | 732324a | 2010-12-04 11:12:43 +0000 | [diff] [blame] | 133 | |
| 134 | .. class:: BsdDbShelf(dict, protocol=None, writeback=False, keyencoding='utf-8') |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 135 | |
Georg Brandl | 502d9a5 | 2009-07-26 15:02:41 +0000 | [diff] [blame] | 136 | A subclass of :class:`Shelf` which exposes :meth:`first`, :meth:`!next`, |
Georg Brandl | 1158a33 | 2009-06-04 09:30:30 +0000 | [diff] [blame] | 137 | :meth:`previous`, :meth:`last` and :meth:`set_location` which are available |
| 138 | in the third-party :mod:`bsddb` module from `pybsddb |
Georg Brandl | 5d94134 | 2016-02-26 19:37:12 +0100 | [diff] [blame] | 139 | <https://www.jcea.es/programacion/pybsddb.htm>`_ but not in other database |
Georg Brandl | 1158a33 | 2009-06-04 09:30:30 +0000 | [diff] [blame] | 140 | modules. The *dict* object passed to the constructor must support those |
| 141 | methods. This is generally accomplished by calling one of |
| 142 | :func:`bsddb.hashopen`, :func:`bsddb.btopen` or :func:`bsddb.rnopen`. The |
Georg Brandl | 732324a | 2010-12-04 11:12:43 +0000 | [diff] [blame] | 143 | optional *protocol*, *writeback*, and *keyencoding* parameters have the same |
| 144 | interpretation as for the :class:`Shelf` class. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 145 | |
| 146 | |
Georg Brandl | 1824415 | 2009-09-02 20:34:52 +0000 | [diff] [blame] | 147 | .. class:: DbfilenameShelf(filename, flag='c', protocol=None, writeback=False) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 148 | |
| 149 | A subclass of :class:`Shelf` which accepts a *filename* instead of a dict-like |
Georg Brandl | 0a7ac7d | 2008-05-26 10:29:35 +0000 | [diff] [blame] | 150 | object. The underlying file will be opened using :func:`dbm.open`. By |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 151 | default, the file will be created and opened for both read and write. The |
Georg Brandl | 502d9a5 | 2009-07-26 15:02:41 +0000 | [diff] [blame] | 152 | optional *flag* parameter has the same interpretation as for the :func:`.open` |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 153 | function. The optional *protocol* and *writeback* parameters have the same |
| 154 | interpretation as for the :class:`Shelf` class. |
| 155 | |
| 156 | |
R. David Murray | ff85bca | 2009-05-12 01:40:16 +0000 | [diff] [blame] | 157 | .. _shelve-example: |
| 158 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 159 | Example |
| 160 | ------- |
| 161 | |
| 162 | To summarize the interface (``key`` is a string, ``data`` is an arbitrary |
| 163 | object):: |
| 164 | |
| 165 | import shelve |
| 166 | |
Serhiy Storchaka | dba9039 | 2016-05-10 12:01:23 +0300 | [diff] [blame] | 167 | d = shelve.open(filename) # open -- file may get suffix added by low-level |
| 168 | # library |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 169 | |
Serhiy Storchaka | dba9039 | 2016-05-10 12:01:23 +0300 | [diff] [blame] | 170 | d[key] = data # store data at key (overwrites old data if |
| 171 | # using an existing key) |
| 172 | data = d[key] # retrieve a COPY of data at key (raise KeyError |
| 173 | # if no such key) |
| 174 | del d[key] # delete data stored at key (raises KeyError |
| 175 | # if no such key) |
| 176 | |
| 177 | flag = key in d # true if the key exists |
| 178 | klist = list(d.keys()) # a list of all existing keys (slow!) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 179 | |
| 180 | # as d was opened WITHOUT writeback=True, beware: |
Serhiy Storchaka | dba9039 | 2016-05-10 12:01:23 +0300 | [diff] [blame] | 181 | d['xx'] = [0, 1, 2] # this works as expected, but... |
| 182 | d['xx'].append(3) # *this doesn't!* -- d['xx'] is STILL [0, 1, 2]! |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 183 | |
| 184 | # having opened d without writeback=True, you need to code carefully: |
Serhiy Storchaka | dba9039 | 2016-05-10 12:01:23 +0300 | [diff] [blame] | 185 | temp = d['xx'] # extracts the copy |
| 186 | temp.append(5) # mutates the copy |
| 187 | d['xx'] = temp # stores the copy right back, to persist it |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 188 | |
| 189 | # or, d=shelve.open(filename,writeback=True) would let you just code |
| 190 | # d['xx'].append(5) and have it work as expected, BUT it would also |
| 191 | # consume more memory and make the d.close() operation slower. |
| 192 | |
Serhiy Storchaka | dba9039 | 2016-05-10 12:01:23 +0300 | [diff] [blame] | 193 | d.close() # close it |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 194 | |
| 195 | |
| 196 | .. seealso:: |
| 197 | |
Georg Brandl | 0a7ac7d | 2008-05-26 10:29:35 +0000 | [diff] [blame] | 198 | Module :mod:`dbm` |
| 199 | Generic interface to ``dbm``-style databases. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 200 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 201 | Module :mod:`pickle` |
| 202 | Object serialization used by :mod:`shelve`. |
| 203 | |