Guido van Rossum | cc6764c | 1995-02-09 17:18:10 +0000 | [diff] [blame] | 1 | """Manage shelves of pickled objects. |
| 2 | |
| 3 | A "shelf" is a persistent, dictionary-like object. The difference |
| 4 | with dbm databases is that the values (not the keys!) in a shelf can |
| 5 | be essentially arbitrary Python objects -- anything that the "pickle" |
| 6 | module can handle. This includes most class instances, recursive data |
| 7 | types, and objects containing lots of shared sub-objects. The keys |
| 8 | are ordinary strings. |
| 9 | |
| 10 | To summarize the interface (key is a string, data is an arbitrary |
| 11 | object): |
| 12 | |
Fred Drake | 13a2c27 | 2000-02-10 17:17:14 +0000 | [diff] [blame] | 13 | import shelve |
| 14 | d = shelve.open(filename) # open, with (g)dbm filename -- no suffix |
Guido van Rossum | cc6764c | 1995-02-09 17:18:10 +0000 | [diff] [blame] | 15 | |
Fred Drake | 13a2c27 | 2000-02-10 17:17:14 +0000 | [diff] [blame] | 16 | d[key] = data # store data at key (overwrites old data if |
| 17 | # using an existing key) |
Tim Peters | 0eadaac | 2003-04-24 16:02:54 +0000 | [diff] [blame] | 18 | data = d[key] # retrieve a COPY of the data at key (raise |
Martin v. Löwis | 153c9e4 | 2003-04-19 20:59:03 +0000 | [diff] [blame] | 19 | # KeyError if no such key) -- NOTE that this |
| 20 | # access returns a *copy* of the entry! |
Fred Drake | 13a2c27 | 2000-02-10 17:17:14 +0000 | [diff] [blame] | 21 | del d[key] # delete data stored at key (raises KeyError |
| 22 | # if no such key) |
Guido van Rossum | e2b70bc | 2006-08-18 22:13:04 +0000 | [diff] [blame] | 23 | flag = key in d # true if the key exists |
Fred Drake | 13a2c27 | 2000-02-10 17:17:14 +0000 | [diff] [blame] | 24 | list = d.keys() # a list of all existing keys (slow!) |
Guido van Rossum | cc6764c | 1995-02-09 17:18:10 +0000 | [diff] [blame] | 25 | |
Fred Drake | 13a2c27 | 2000-02-10 17:17:14 +0000 | [diff] [blame] | 26 | d.close() # close it |
Guido van Rossum | cc6764c | 1995-02-09 17:18:10 +0000 | [diff] [blame] | 27 | |
| 28 | Dependent on the implementation, closing a persistent dictionary may |
| 29 | or may not be necessary to flush changes to disk. |
Martin v. Löwis | 153c9e4 | 2003-04-19 20:59:03 +0000 | [diff] [blame] | 30 | |
| 31 | Normally, d[key] returns a COPY of the entry. This needs care when |
| 32 | mutable entries are mutated: for example, if d[key] is a list, |
| 33 | d[key].append(anitem) |
| 34 | does NOT modify the entry d[key] itself, as stored in the persistent |
| 35 | mapping -- it only modifies the copy, which is then immediately |
| 36 | discarded, so that the append has NO effect whatsoever. To append an |
| 37 | item to d[key] in a way that will affect the persistent mapping, use: |
| 38 | data = d[key] |
| 39 | data.append(anitem) |
| 40 | d[key] = data |
| 41 | |
| 42 | To avoid the problem with mutable entries, you may pass the keyword |
| 43 | argument writeback=True in the call to shelve.open. When you use: |
| 44 | d = shelve.open(filename, writeback=True) |
| 45 | then d keeps a cache of all entries you access, and writes them all back |
| 46 | to the persistent mapping when you call d.close(). This ensures that |
| 47 | such usage as d[key].append(anitem) works as intended. |
| 48 | |
| 49 | However, using keyword argument writeback=True may consume vast amount |
| 50 | of memory for the cache, and it may make d.close() very slow, if you |
| 51 | access many of d's entries after opening it in this way: d has no way to |
| 52 | check which of the entries you access are mutable and/or which ones you |
| 53 | actually mutate, so it must cache, and write back at close, all of the |
| 54 | entries that you access. You can call d.sync() to write back all the |
| 55 | entries in the cache, and empty the cache (d.sync() also synchronizes |
| 56 | the persistent dictionary on disk, if feasible). |
Guido van Rossum | cc6764c | 1995-02-09 17:18:10 +0000 | [diff] [blame] | 57 | """ |
Guido van Rossum | a48061a | 1995-01-10 00:31:14 +0000 | [diff] [blame] | 58 | |
Guido van Rossum | 68937b4 | 2007-05-18 00:51:22 +0000 | [diff] [blame] | 59 | from pickle import Pickler, Unpickler |
Brett Cannon | d24fffe | 2007-07-26 03:07:02 +0000 | [diff] [blame] | 60 | from io import BytesIO |
Guido van Rossum | a48061a | 1995-01-10 00:31:14 +0000 | [diff] [blame] | 61 | |
Raymond Hettinger | b9da9bc | 2008-02-04 20:44:31 +0000 | [diff] [blame] | 62 | import collections |
Raymond Hettinger | 7994716 | 2002-11-15 06:46:14 +0000 | [diff] [blame] | 63 | |
Skip Montanaro | 0de6580 | 2001-02-15 22:15:14 +0000 | [diff] [blame] | 64 | __all__ = ["Shelf","BsdDbShelf","DbfilenameShelf","open"] |
Guido van Rossum | cc6764c | 1995-02-09 17:18:10 +0000 | [diff] [blame] | 65 | |
Benjamin Peterson | d631371 | 2008-07-31 16:23:04 +0000 | [diff] [blame] | 66 | class _ClosedDict(collections.MutableMapping): |
| 67 | 'Marker for a closed dict. Access attempts raise a ValueError.' |
| 68 | |
| 69 | def closed(self, *args): |
| 70 | raise ValueError('invalid operation on closed shelf') |
| 71 | __iter__ = __len__ = __getitem__ = __setitem__ = __delitem__ = keys = closed |
| 72 | |
| 73 | def __repr__(self): |
| 74 | return '<Closed Dictionary>' |
| 75 | |
Raymond Hettinger | b9da9bc | 2008-02-04 20:44:31 +0000 | [diff] [blame] | 76 | class Shelf(collections.MutableMapping): |
Tim Peters | 495ad3c | 2001-01-15 01:36:40 +0000 | [diff] [blame] | 77 | """Base class for shelf implementations. |
Guido van Rossum | cc6764c | 1995-02-09 17:18:10 +0000 | [diff] [blame] | 78 | |
Tim Peters | 495ad3c | 2001-01-15 01:36:40 +0000 | [diff] [blame] | 79 | This is initialized with a dictionary-like object. |
| 80 | See the module's __doc__ string for an overview of the interface. |
| 81 | """ |
Guido van Rossum | a48061a | 1995-01-10 00:31:14 +0000 | [diff] [blame] | 82 | |
Martin v. Löwis | 79c3208 | 2007-08-11 06:57:14 +0000 | [diff] [blame] | 83 | def __init__(self, dict, protocol=None, writeback=False, |
| 84 | keyencoding="utf-8"): |
Tim Peters | 495ad3c | 2001-01-15 01:36:40 +0000 | [diff] [blame] | 85 | self.dict = dict |
Martin v. Löwis | 153c9e4 | 2003-04-19 20:59:03 +0000 | [diff] [blame] | 86 | if protocol is None: |
Raymond Hettinger | ba99c58 | 2008-02-10 20:41:56 +0000 | [diff] [blame] | 87 | protocol = 2 |
Martin v. Löwis | 153c9e4 | 2003-04-19 20:59:03 +0000 | [diff] [blame] | 88 | self._protocol = protocol |
| 89 | self.writeback = writeback |
| 90 | self.cache = {} |
Martin v. Löwis | 79c3208 | 2007-08-11 06:57:14 +0000 | [diff] [blame] | 91 | self.keyencoding = "utf-8" |
Guido van Rossum | 2f7df12 | 1999-08-11 01:54:05 +0000 | [diff] [blame] | 92 | |
Raymond Hettinger | b9da9bc | 2008-02-04 20:44:31 +0000 | [diff] [blame] | 93 | def __iter__(self): |
Martin v. Löwis | 79c3208 | 2007-08-11 06:57:14 +0000 | [diff] [blame] | 94 | for k in self.dict.keys(): |
| 95 | yield k.decode(self.keyencoding) |
Guido van Rossum | a48061a | 1995-01-10 00:31:14 +0000 | [diff] [blame] | 96 | |
Tim Peters | 495ad3c | 2001-01-15 01:36:40 +0000 | [diff] [blame] | 97 | def __len__(self): |
| 98 | return len(self.dict) |
Guido van Rossum | a48061a | 1995-01-10 00:31:14 +0000 | [diff] [blame] | 99 | |
Martin v. Löwis | e4913c9 | 2002-10-18 08:58:14 +0000 | [diff] [blame] | 100 | def __contains__(self, key): |
Martin v. Löwis | 79c3208 | 2007-08-11 06:57:14 +0000 | [diff] [blame] | 101 | return key.encode(self.keyencoding) in self.dict |
Martin v. Löwis | e4913c9 | 2002-10-18 08:58:14 +0000 | [diff] [blame] | 102 | |
Tim Peters | 495ad3c | 2001-01-15 01:36:40 +0000 | [diff] [blame] | 103 | def get(self, key, default=None): |
Martin v. Löwis | 79c3208 | 2007-08-11 06:57:14 +0000 | [diff] [blame] | 104 | if key.encode(self.keyencoding) in self.dict: |
Tim Peters | 495ad3c | 2001-01-15 01:36:40 +0000 | [diff] [blame] | 105 | return self[key] |
| 106 | return default |
| 107 | |
| 108 | def __getitem__(self, key): |
Martin v. Löwis | 153c9e4 | 2003-04-19 20:59:03 +0000 | [diff] [blame] | 109 | try: |
| 110 | value = self.cache[key] |
| 111 | except KeyError: |
Martin v. Löwis | 79c3208 | 2007-08-11 06:57:14 +0000 | [diff] [blame] | 112 | f = BytesIO(self.dict[key.encode(self.keyencoding)]) |
Martin v. Löwis | 153c9e4 | 2003-04-19 20:59:03 +0000 | [diff] [blame] | 113 | value = Unpickler(f).load() |
| 114 | if self.writeback: |
| 115 | self.cache[key] = value |
| 116 | return value |
Tim Peters | 495ad3c | 2001-01-15 01:36:40 +0000 | [diff] [blame] | 117 | |
| 118 | def __setitem__(self, key, value): |
Martin v. Löwis | 153c9e4 | 2003-04-19 20:59:03 +0000 | [diff] [blame] | 119 | if self.writeback: |
| 120 | self.cache[key] = value |
Brett Cannon | d24fffe | 2007-07-26 03:07:02 +0000 | [diff] [blame] | 121 | f = BytesIO() |
Martin v. Löwis | 153c9e4 | 2003-04-19 20:59:03 +0000 | [diff] [blame] | 122 | p = Pickler(f, self._protocol) |
Tim Peters | 495ad3c | 2001-01-15 01:36:40 +0000 | [diff] [blame] | 123 | p.dump(value) |
Martin v. Löwis | 79c3208 | 2007-08-11 06:57:14 +0000 | [diff] [blame] | 124 | self.dict[key.encode(self.keyencoding)] = f.getvalue() |
Tim Peters | 495ad3c | 2001-01-15 01:36:40 +0000 | [diff] [blame] | 125 | |
| 126 | def __delitem__(self, key): |
Martin v. Löwis | 79c3208 | 2007-08-11 06:57:14 +0000 | [diff] [blame] | 127 | del self.dict[key.encode(self.keyencoding)] |
Martin v. Löwis | 153c9e4 | 2003-04-19 20:59:03 +0000 | [diff] [blame] | 128 | try: |
| 129 | del self.cache[key] |
| 130 | except KeyError: |
| 131 | pass |
Tim Peters | 495ad3c | 2001-01-15 01:36:40 +0000 | [diff] [blame] | 132 | |
| 133 | def close(self): |
Martin v. Löwis | 153c9e4 | 2003-04-19 20:59:03 +0000 | [diff] [blame] | 134 | self.sync() |
Tim Peters | 495ad3c | 2001-01-15 01:36:40 +0000 | [diff] [blame] | 135 | try: |
| 136 | self.dict.close() |
Raymond Hettinger | 68dcd34 | 2003-05-27 06:30:52 +0000 | [diff] [blame] | 137 | except AttributeError: |
Tim Peters | 495ad3c | 2001-01-15 01:36:40 +0000 | [diff] [blame] | 138 | pass |
Benjamin Peterson | d631371 | 2008-07-31 16:23:04 +0000 | [diff] [blame] | 139 | self.dict = _ClosedDict() |
Tim Peters | 495ad3c | 2001-01-15 01:36:40 +0000 | [diff] [blame] | 140 | |
| 141 | def __del__(self): |
Thomas Wouters | 0e3f591 | 2006-08-11 14:57:12 +0000 | [diff] [blame] | 142 | if not hasattr(self, 'writeback'): |
| 143 | # __init__ didn't succeed, so don't bother closing |
| 144 | return |
Tim Peters | 495ad3c | 2001-01-15 01:36:40 +0000 | [diff] [blame] | 145 | self.close() |
| 146 | |
| 147 | def sync(self): |
Martin v. Löwis | 153c9e4 | 2003-04-19 20:59:03 +0000 | [diff] [blame] | 148 | if self.writeback and self.cache: |
| 149 | self.writeback = False |
Guido van Rossum | cc2b016 | 2007-02-11 06:12:03 +0000 | [diff] [blame] | 150 | for key, entry in self.cache.items(): |
Martin v. Löwis | 153c9e4 | 2003-04-19 20:59:03 +0000 | [diff] [blame] | 151 | self[key] = entry |
| 152 | self.writeback = True |
| 153 | self.cache = {} |
Tim Peters | 495ad3c | 2001-01-15 01:36:40 +0000 | [diff] [blame] | 154 | if hasattr(self.dict, 'sync'): |
| 155 | self.dict.sync() |
| 156 | |
Guido van Rossum | cc6764c | 1995-02-09 17:18:10 +0000 | [diff] [blame] | 157 | |
Guido van Rossum | abad1cc | 1995-08-11 14:19:16 +0000 | [diff] [blame] | 158 | class BsdDbShelf(Shelf): |
Tim Peters | 495ad3c | 2001-01-15 01:36:40 +0000 | [diff] [blame] | 159 | """Shelf implementation using the "BSD" db interface. |
Guido van Rossum | abad1cc | 1995-08-11 14:19:16 +0000 | [diff] [blame] | 160 | |
Tim Peters | 495ad3c | 2001-01-15 01:36:40 +0000 | [diff] [blame] | 161 | This adds methods first(), next(), previous(), last() and |
| 162 | set_location() that have no counterpart in [g]dbm databases. |
Guido van Rossum | abad1cc | 1995-08-11 14:19:16 +0000 | [diff] [blame] | 163 | |
Tim Peters | 495ad3c | 2001-01-15 01:36:40 +0000 | [diff] [blame] | 164 | The actual database must be opened using one of the "bsddb" |
| 165 | modules "open" routines (i.e. bsddb.hashopen, bsddb.btopen or |
| 166 | bsddb.rnopen) and passed to the constructor. |
Guido van Rossum | abad1cc | 1995-08-11 14:19:16 +0000 | [diff] [blame] | 167 | |
Tim Peters | 495ad3c | 2001-01-15 01:36:40 +0000 | [diff] [blame] | 168 | See the module's __doc__ string for an overview of the interface. |
| 169 | """ |
Guido van Rossum | abad1cc | 1995-08-11 14:19:16 +0000 | [diff] [blame] | 170 | |
Martin v. Löwis | 79c3208 | 2007-08-11 06:57:14 +0000 | [diff] [blame] | 171 | def __init__(self, dict, protocol=None, writeback=False, |
| 172 | keyencoding="utf-8"): |
| 173 | Shelf.__init__(self, dict, protocol, writeback, keyencoding) |
Guido van Rossum | abad1cc | 1995-08-11 14:19:16 +0000 | [diff] [blame] | 174 | |
Tim Peters | 495ad3c | 2001-01-15 01:36:40 +0000 | [diff] [blame] | 175 | def set_location(self, key): |
| 176 | (key, value) = self.dict.set_location(key) |
Brett Cannon | d24fffe | 2007-07-26 03:07:02 +0000 | [diff] [blame] | 177 | f = BytesIO(value) |
Martin v. Löwis | 79c3208 | 2007-08-11 06:57:14 +0000 | [diff] [blame] | 178 | return (key.decode(self.keyencoding), Unpickler(f).load()) |
Guido van Rossum | abad1cc | 1995-08-11 14:19:16 +0000 | [diff] [blame] | 179 | |
Tim Peters | 495ad3c | 2001-01-15 01:36:40 +0000 | [diff] [blame] | 180 | def next(self): |
Georg Brandl | a18af4e | 2007-04-21 15:47:16 +0000 | [diff] [blame] | 181 | (key, value) = next(self.dict) |
Brett Cannon | d24fffe | 2007-07-26 03:07:02 +0000 | [diff] [blame] | 182 | f = BytesIO(value) |
Martin v. Löwis | 79c3208 | 2007-08-11 06:57:14 +0000 | [diff] [blame] | 183 | return (key.decode(self.keyencoding), Unpickler(f).load()) |
Guido van Rossum | abad1cc | 1995-08-11 14:19:16 +0000 | [diff] [blame] | 184 | |
Tim Peters | 495ad3c | 2001-01-15 01:36:40 +0000 | [diff] [blame] | 185 | def previous(self): |
| 186 | (key, value) = self.dict.previous() |
Brett Cannon | d24fffe | 2007-07-26 03:07:02 +0000 | [diff] [blame] | 187 | f = BytesIO(value) |
Martin v. Löwis | 79c3208 | 2007-08-11 06:57:14 +0000 | [diff] [blame] | 188 | return (key.decode(self.keyencoding), Unpickler(f).load()) |
Guido van Rossum | abad1cc | 1995-08-11 14:19:16 +0000 | [diff] [blame] | 189 | |
Tim Peters | 495ad3c | 2001-01-15 01:36:40 +0000 | [diff] [blame] | 190 | def first(self): |
| 191 | (key, value) = self.dict.first() |
Brett Cannon | d24fffe | 2007-07-26 03:07:02 +0000 | [diff] [blame] | 192 | f = BytesIO(value) |
Martin v. Löwis | 79c3208 | 2007-08-11 06:57:14 +0000 | [diff] [blame] | 193 | return (key.decode(self.keyencoding), Unpickler(f).load()) |
Guido van Rossum | abad1cc | 1995-08-11 14:19:16 +0000 | [diff] [blame] | 194 | |
Tim Peters | 495ad3c | 2001-01-15 01:36:40 +0000 | [diff] [blame] | 195 | def last(self): |
| 196 | (key, value) = self.dict.last() |
Brett Cannon | d24fffe | 2007-07-26 03:07:02 +0000 | [diff] [blame] | 197 | f = BytesIO(value) |
Martin v. Löwis | 79c3208 | 2007-08-11 06:57:14 +0000 | [diff] [blame] | 198 | return (key.decode(self.keyencoding), Unpickler(f).load()) |
Guido van Rossum | abad1cc | 1995-08-11 14:19:16 +0000 | [diff] [blame] | 199 | |
| 200 | |
| 201 | class DbfilenameShelf(Shelf): |
Georg Brandl | 0a7ac7d | 2008-05-26 10:29:35 +0000 | [diff] [blame] | 202 | """Shelf implementation using the "dbm" generic dbm interface. |
Guido van Rossum | cc6764c | 1995-02-09 17:18:10 +0000 | [diff] [blame] | 203 | |
Tim Peters | 495ad3c | 2001-01-15 01:36:40 +0000 | [diff] [blame] | 204 | This is initialized with the filename for the dbm database. |
| 205 | See the module's __doc__ string for an overview of the interface. |
| 206 | """ |
| 207 | |
Raymond Hettinger | 1bc82f8 | 2004-12-05 03:58:17 +0000 | [diff] [blame] | 208 | def __init__(self, filename, flag='c', protocol=None, writeback=False): |
Georg Brandl | 0a7ac7d | 2008-05-26 10:29:35 +0000 | [diff] [blame] | 209 | import dbm |
| 210 | Shelf.__init__(self, dbm.open(filename, flag), protocol, writeback) |
Guido van Rossum | cc6764c | 1995-02-09 17:18:10 +0000 | [diff] [blame] | 211 | |
| 212 | |
Raymond Hettinger | 1bc82f8 | 2004-12-05 03:58:17 +0000 | [diff] [blame] | 213 | def open(filename, flag='c', protocol=None, writeback=False): |
Tim Peters | 495ad3c | 2001-01-15 01:36:40 +0000 | [diff] [blame] | 214 | """Open a persistent dictionary for reading and writing. |
Guido van Rossum | cc6764c | 1995-02-09 17:18:10 +0000 | [diff] [blame] | 215 | |
Martin v. Löwis | 153c9e4 | 2003-04-19 20:59:03 +0000 | [diff] [blame] | 216 | The filename parameter is the base filename for the underlying |
| 217 | database. As a side-effect, an extension may be added to the |
| 218 | filename and more than one file may be created. The optional flag |
| 219 | parameter has the same interpretation as the flag parameter of |
Georg Brandl | 0a7ac7d | 2008-05-26 10:29:35 +0000 | [diff] [blame] | 220 | dbm.open(). The optional protocol parameter specifies the |
Martin v. Löwis | 153c9e4 | 2003-04-19 20:59:03 +0000 | [diff] [blame] | 221 | version of the pickle protocol (0, 1, or 2). |
| 222 | |
Tim Peters | 495ad3c | 2001-01-15 01:36:40 +0000 | [diff] [blame] | 223 | See the module's __doc__ string for an overview of the interface. |
| 224 | """ |
| 225 | |
Raymond Hettinger | 1bc82f8 | 2004-12-05 03:58:17 +0000 | [diff] [blame] | 226 | return DbfilenameShelf(filename, flag, protocol, writeback) |