Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 1 | :mod:`pickle` --- Python object serialization |
| 2 | ============================================= |
| 3 | |
Terry Jan Reedy | fa089b9 | 2016-06-11 15:02:54 -0400 | [diff] [blame] | 4 | .. module:: pickle |
| 5 | :synopsis: Convert Python objects to streams of bytes and back. |
| 6 | |
| 7 | .. sectionauthor:: Jim Kerr <jbkerr@sr.hp.com>. |
| 8 | .. sectionauthor:: Barry Warsaw <barry@python.org> |
| 9 | |
| 10 | **Source code:** :source:`Lib/pickle.py` |
| 11 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 12 | .. index:: |
| 13 | single: persistence |
| 14 | pair: persistent; objects |
| 15 | pair: serializing; objects |
| 16 | pair: marshalling; objects |
| 17 | pair: flattening; objects |
| 18 | pair: pickling; objects |
| 19 | |
Terry Jan Reedy | fa089b9 | 2016-06-11 15:02:54 -0400 | [diff] [blame] | 20 | -------------- |
Alexandre Vassalotti | 9d7665d | 2009-04-03 06:13:29 +0000 | [diff] [blame] | 21 | |
Antoine Pitrou | d4d6055 | 2013-12-07 00:56:59 +0100 | [diff] [blame] | 22 | The :mod:`pickle` module implements binary protocols for serializing and |
| 23 | de-serializing a Python object structure. *"Pickling"* is the process |
| 24 | whereby a Python object hierarchy is converted into a byte stream, and |
| 25 | *"unpickling"* is the inverse operation, whereby a byte stream |
| 26 | (from a :term:`binary file` or :term:`bytes-like object`) is converted |
| 27 | back into an object hierarchy. Pickling (and unpickling) is alternatively |
| 28 | known as "serialization", "marshalling," [#]_ or "flattening"; however, to |
| 29 | avoid confusion, the terms used here are "pickling" and "unpickling". |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 30 | |
Georg Brandl | 0036bcf | 2010-10-17 10:24:54 +0000 | [diff] [blame] | 31 | .. warning:: |
| 32 | |
Benjamin Peterson | 7dcbf90 | 2015-07-06 11:28:07 -0500 | [diff] [blame] | 33 | The :mod:`pickle` module is not secure against erroneous or maliciously |
Benjamin Peterson | b8fd262 | 2015-07-06 09:40:43 -0500 | [diff] [blame] | 34 | constructed data. Never unpickle data received from an untrusted or |
| 35 | unauthenticated source. |
Georg Brandl | 0036bcf | 2010-10-17 10:24:54 +0000 | [diff] [blame] | 36 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 37 | |
| 38 | Relationship to other Python modules |
| 39 | ------------------------------------ |
| 40 | |
Antoine Pitrou | d4d6055 | 2013-12-07 00:56:59 +0100 | [diff] [blame] | 41 | Comparison with ``marshal`` |
| 42 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 43 | |
| 44 | Python has a more primitive serialization module called :mod:`marshal`, but in |
| 45 | general :mod:`pickle` should always be the preferred way to serialize Python |
| 46 | objects. :mod:`marshal` exists primarily to support Python's :file:`.pyc` |
| 47 | files. |
| 48 | |
Georg Brandl | 5aa580f | 2010-11-30 14:57:54 +0000 | [diff] [blame] | 49 | The :mod:`pickle` module differs from :mod:`marshal` in several significant ways: |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 50 | |
| 51 | * The :mod:`pickle` module keeps track of the objects it has already serialized, |
| 52 | so that later references to the same object won't be serialized again. |
| 53 | :mod:`marshal` doesn't do this. |
| 54 | |
| 55 | This has implications both for recursive objects and object sharing. Recursive |
| 56 | objects are objects that contain references to themselves. These are not |
| 57 | handled by marshal, and in fact, attempting to marshal recursive objects will |
| 58 | crash your Python interpreter. Object sharing happens when there are multiple |
| 59 | references to the same object in different places in the object hierarchy being |
| 60 | serialized. :mod:`pickle` stores such objects only once, and ensures that all |
| 61 | other references point to the master copy. Shared objects remain shared, which |
| 62 | can be very important for mutable objects. |
| 63 | |
| 64 | * :mod:`marshal` cannot be used to serialize user-defined classes and their |
| 65 | instances. :mod:`pickle` can save and restore class instances transparently, |
| 66 | however the class definition must be importable and live in the same module as |
| 67 | when the object was stored. |
| 68 | |
| 69 | * The :mod:`marshal` serialization format is not guaranteed to be portable |
| 70 | across Python versions. Because its primary job in life is to support |
| 71 | :file:`.pyc` files, the Python implementers reserve the right to change the |
| 72 | serialization format in non-backwards compatible ways should the need arise. |
| 73 | The :mod:`pickle` serialization format is guaranteed to be backwards compatible |
| 74 | across Python releases. |
| 75 | |
Antoine Pitrou | d4d6055 | 2013-12-07 00:56:59 +0100 | [diff] [blame] | 76 | Comparison with ``json`` |
| 77 | ^^^^^^^^^^^^^^^^^^^^^^^^ |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 78 | |
Antoine Pitrou | d4d6055 | 2013-12-07 00:56:59 +0100 | [diff] [blame] | 79 | There are fundamental differences between the pickle protocols and |
| 80 | `JSON (JavaScript Object Notation) <http://json.org>`_: |
| 81 | |
| 82 | * JSON is a text serialization format (it outputs unicode text, although |
| 83 | most of the time it is then encoded to ``utf-8``), while pickle is |
| 84 | a binary serialization format; |
| 85 | |
| 86 | * JSON is human-readable, while pickle is not; |
| 87 | |
| 88 | * JSON is interoperable and widely used outside of the Python ecosystem, |
| 89 | while pickle is Python-specific; |
| 90 | |
| 91 | * JSON, by default, can only represent a subset of the Python built-in |
| 92 | types, and no custom classes; pickle can represent an extremely large |
| 93 | number of Python types (many of them automatically, by clever usage |
| 94 | of Python's introspection facilities; complex cases can be tackled by |
| 95 | implementing :ref:`specific object APIs <pickle-inst>`). |
| 96 | |
| 97 | .. seealso:: |
| 98 | The :mod:`json` module: a standard library module allowing JSON |
| 99 | serialization and deserialization. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 100 | |
Antoine Pitrou | 9bcb112 | 2013-12-07 01:05:57 +0100 | [diff] [blame] | 101 | |
| 102 | .. _pickle-protocols: |
| 103 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 104 | Data stream format |
| 105 | ------------------ |
| 106 | |
| 107 | .. index:: |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 108 | single: External Data Representation |
| 109 | |
| 110 | The data format used by :mod:`pickle` is Python-specific. This has the |
| 111 | advantage that there are no restrictions imposed by external standards such as |
Antoine Pitrou | a9494f6 | 2012-05-10 15:38:30 +0200 | [diff] [blame] | 112 | JSON or XDR (which can't represent pointer sharing); however it means that |
| 113 | non-Python programs may not be able to reconstruct pickled Python objects. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 114 | |
Antoine Pitrou | a9494f6 | 2012-05-10 15:38:30 +0200 | [diff] [blame] | 115 | By default, the :mod:`pickle` data format uses a relatively compact binary |
| 116 | representation. If you need optimal size characteristics, you can efficiently |
| 117 | :doc:`compress <archiving>` pickled data. |
| 118 | |
Alexandre Vassalotti | 758bca6 | 2008-10-18 19:25:07 +0000 | [diff] [blame] | 119 | The module :mod:`pickletools` contains tools for analyzing data streams |
Antoine Pitrou | a9494f6 | 2012-05-10 15:38:30 +0200 | [diff] [blame] | 120 | generated by :mod:`pickle`. :mod:`pickletools` source code has extensive |
| 121 | comments about opcodes used by pickle protocols. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 122 | |
Antoine Pitrou | b645724 | 2014-01-21 02:39:54 +0100 | [diff] [blame] | 123 | There are currently 5 different protocols which can be used for pickling. |
| 124 | The higher the protocol used, the more recent the version of Python needed |
| 125 | to read the pickle produced. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 126 | |
Antoine Pitrou | a9494f6 | 2012-05-10 15:38:30 +0200 | [diff] [blame] | 127 | * Protocol version 0 is the original "human-readable" protocol and is |
Alexandre Vassalotti | f7d08c7 | 2009-01-23 04:50:05 +0000 | [diff] [blame] | 128 | backwards compatible with earlier versions of Python. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 129 | |
Antoine Pitrou | a9494f6 | 2012-05-10 15:38:30 +0200 | [diff] [blame] | 130 | * Protocol version 1 is an old binary format which is also compatible with |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 131 | earlier versions of Python. |
| 132 | |
| 133 | * Protocol version 2 was introduced in Python 2.3. It provides much more |
Antoine Pitrou | a9494f6 | 2012-05-10 15:38:30 +0200 | [diff] [blame] | 134 | efficient pickling of :term:`new-style class`\es. Refer to :pep:`307` for |
| 135 | information about improvements brought by protocol 2. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 136 | |
Antoine Pitrou | 9bcb112 | 2013-12-07 01:05:57 +0100 | [diff] [blame] | 137 | * Protocol version 3 was added in Python 3.0. It has explicit support for |
Łukasz Langa | c51d8c9 | 2018-04-03 23:06:53 -0700 | [diff] [blame] | 138 | :class:`bytes` objects and cannot be unpickled by Python 2.x. This was |
| 139 | the default protocol in Python 3.0--3.7. |
Antoine Pitrou | 9bcb112 | 2013-12-07 01:05:57 +0100 | [diff] [blame] | 140 | |
| 141 | * Protocol version 4 was added in Python 3.4. It adds support for very large |
| 142 | objects, pickling more kinds of objects, and some data format |
Łukasz Langa | c51d8c9 | 2018-04-03 23:06:53 -0700 | [diff] [blame] | 143 | optimizations. It is the default protocol starting with Python 3.8. |
| 144 | Refer to :pep:`3154` for information about improvements brought by |
| 145 | protocol 4. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 146 | |
Antoine Pitrou | d4d6055 | 2013-12-07 00:56:59 +0100 | [diff] [blame] | 147 | .. note:: |
| 148 | Serialization is a more primitive notion than persistence; although |
| 149 | :mod:`pickle` reads and writes file objects, it does not handle the issue of |
| 150 | naming persistent objects, nor the (even more complicated) issue of concurrent |
| 151 | access to persistent objects. The :mod:`pickle` module can transform a complex |
| 152 | object into a byte stream and it can transform the byte stream into an object |
| 153 | with the same internal structure. Perhaps the most obvious thing to do with |
| 154 | these byte streams is to write them onto a file, but it is also conceivable to |
| 155 | send them across a network or store them in a database. The :mod:`shelve` |
| 156 | module provides a simple interface to pickle and unpickle objects on |
| 157 | DBM-style database files. |
| 158 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 159 | |
Alexandre Vassalotti | d039286 | 2008-10-24 01:32:40 +0000 | [diff] [blame] | 160 | Module Interface |
| 161 | ---------------- |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 162 | |
Antoine Pitrou | a9494f6 | 2012-05-10 15:38:30 +0200 | [diff] [blame] | 163 | To serialize an object hierarchy, you simply call the :func:`dumps` function. |
| 164 | Similarly, to de-serialize a data stream, you call the :func:`loads` function. |
| 165 | However, if you want more control over serialization and de-serialization, |
| 166 | you can create a :class:`Pickler` or an :class:`Unpickler` object, respectively. |
| 167 | |
| 168 | The :mod:`pickle` module provides the following constants: |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 169 | |
| 170 | |
| 171 | .. data:: HIGHEST_PROTOCOL |
| 172 | |
Antoine Pitrou | 9bcb112 | 2013-12-07 01:05:57 +0100 | [diff] [blame] | 173 | An integer, the highest :ref:`protocol version <pickle-protocols>` |
| 174 | available. This value can be passed as a *protocol* value to functions |
| 175 | :func:`dump` and :func:`dumps` as well as the :class:`Pickler` |
| 176 | constructor. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 177 | |
Alexandre Vassalotti | 758bca6 | 2008-10-18 19:25:07 +0000 | [diff] [blame] | 178 | .. data:: DEFAULT_PROTOCOL |
| 179 | |
Antoine Pitrou | 9bcb112 | 2013-12-07 01:05:57 +0100 | [diff] [blame] | 180 | An integer, the default :ref:`protocol version <pickle-protocols>` used |
| 181 | for pickling. May be less than :data:`HIGHEST_PROTOCOL`. Currently the |
Łukasz Langa | c51d8c9 | 2018-04-03 23:06:53 -0700 | [diff] [blame] | 182 | default protocol is 4, first introduced in Python 3.4 and incompatible |
| 183 | with previous versions. |
Alexandre Vassalotti | 758bca6 | 2008-10-18 19:25:07 +0000 | [diff] [blame] | 184 | |
Łukasz Langa | c51d8c9 | 2018-04-03 23:06:53 -0700 | [diff] [blame] | 185 | .. versionchanged:: 3.0 |
| 186 | |
| 187 | The default protocol is 3. |
| 188 | |
| 189 | .. versionchanged:: 3.8 |
| 190 | |
| 191 | The default protocol is 4. |
Alexandre Vassalotti | 758bca6 | 2008-10-18 19:25:07 +0000 | [diff] [blame] | 192 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 193 | The :mod:`pickle` module provides the following functions to make the pickling |
| 194 | process more convenient: |
| 195 | |
Georg Brandl | 1824415 | 2009-09-02 20:34:52 +0000 | [diff] [blame] | 196 | .. function:: dump(obj, file, protocol=None, \*, fix_imports=True) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 197 | |
Antoine Pitrou | 11cb961 | 2010-09-15 11:11:28 +0000 | [diff] [blame] | 198 | Write a pickled representation of *obj* to the open :term:`file object` *file*. |
| 199 | This is equivalent to ``Pickler(file, protocol).dump(obj)``. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 200 | |
Antoine Pitrou | b645724 | 2014-01-21 02:39:54 +0100 | [diff] [blame] | 201 | The optional *protocol* argument, an integer, tells the pickler to use |
| 202 | the given protocol; supported protocols are 0 to :data:`HIGHEST_PROTOCOL`. |
| 203 | If not specified, the default is :data:`DEFAULT_PROTOCOL`. If a negative |
| 204 | number is specified, :data:`HIGHEST_PROTOCOL` is selected. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 205 | |
Alexandre Vassalotti | 758bca6 | 2008-10-18 19:25:07 +0000 | [diff] [blame] | 206 | The *file* argument must have a write() method that accepts a single bytes |
Serhiy Storchaka | d65c949 | 2015-11-02 14:10:23 +0200 | [diff] [blame] | 207 | argument. It can thus be an on-disk file opened for binary writing, an |
Antoine Pitrou | 11cb961 | 2010-09-15 11:11:28 +0000 | [diff] [blame] | 208 | :class:`io.BytesIO` instance, or any other custom object that meets this |
| 209 | interface. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 210 | |
Serhiy Storchaka | fbc1c26 | 2013-11-29 12:17:13 +0200 | [diff] [blame] | 211 | If *fix_imports* is true and *protocol* is less than 3, pickle will try to |
Alexandre Vassalotti | d05c9ff | 2013-12-07 01:09:27 -0800 | [diff] [blame] | 212 | map the new Python 3 names to the old module names used in Python 2, so |
| 213 | that the pickle data stream is readable with Python 2. |
Antoine Pitrou | d9dfaa9 | 2009-06-04 20:32:06 +0000 | [diff] [blame] | 214 | |
Georg Brandl | 1824415 | 2009-09-02 20:34:52 +0000 | [diff] [blame] | 215 | .. function:: dumps(obj, protocol=None, \*, fix_imports=True) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 216 | |
Alexandre Vassalotti | d05c9ff | 2013-12-07 01:09:27 -0800 | [diff] [blame] | 217 | Return the pickled representation of the object as a :class:`bytes` object, |
| 218 | instead of writing it to a file. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 219 | |
Antoine Pitrou | b645724 | 2014-01-21 02:39:54 +0100 | [diff] [blame] | 220 | Arguments *protocol* and *fix_imports* have the same meaning as in |
| 221 | :func:`dump`. |
Antoine Pitrou | d9dfaa9 | 2009-06-04 20:32:06 +0000 | [diff] [blame] | 222 | |
Georg Brandl | 1824415 | 2009-09-02 20:34:52 +0000 | [diff] [blame] | 223 | .. function:: load(file, \*, fix_imports=True, encoding="ASCII", errors="strict") |
Alexandre Vassalotti | 758bca6 | 2008-10-18 19:25:07 +0000 | [diff] [blame] | 224 | |
Alexandre Vassalotti | d05c9ff | 2013-12-07 01:09:27 -0800 | [diff] [blame] | 225 | Read a pickled object representation from the open :term:`file object` |
| 226 | *file* and return the reconstituted object hierarchy specified therein. |
| 227 | This is equivalent to ``Unpickler(file).load()``. |
Alexandre Vassalotti | 758bca6 | 2008-10-18 19:25:07 +0000 | [diff] [blame] | 228 | |
Alexandre Vassalotti | d05c9ff | 2013-12-07 01:09:27 -0800 | [diff] [blame] | 229 | The protocol version of the pickle is detected automatically, so no |
| 230 | protocol argument is needed. Bytes past the pickled object's |
| 231 | representation are ignored. |
Alexandre Vassalotti | 758bca6 | 2008-10-18 19:25:07 +0000 | [diff] [blame] | 232 | |
| 233 | The argument *file* must have two methods, a read() method that takes an |
| 234 | integer argument, and a readline() method that requires no arguments. Both |
Alexandre Vassalotti | d05c9ff | 2013-12-07 01:09:27 -0800 | [diff] [blame] | 235 | methods should return bytes. Thus *file* can be an on-disk file opened for |
Martin Panter | 7462b649 | 2015-11-02 03:37:02 +0000 | [diff] [blame] | 236 | binary reading, an :class:`io.BytesIO` object, or any other custom object |
Antoine Pitrou | 11cb961 | 2010-09-15 11:11:28 +0000 | [diff] [blame] | 237 | that meets this interface. |
Alexandre Vassalotti | 758bca6 | 2008-10-18 19:25:07 +0000 | [diff] [blame] | 238 | |
Antoine Pitrou | d9dfaa9 | 2009-06-04 20:32:06 +0000 | [diff] [blame] | 239 | Optional keyword arguments are *fix_imports*, *encoding* and *errors*, |
Georg Brandl | 6faee4e | 2010-09-21 14:48:28 +0000 | [diff] [blame] | 240 | which are used to control compatibility support for pickle stream generated |
Alexandre Vassalotti | d05c9ff | 2013-12-07 01:09:27 -0800 | [diff] [blame] | 241 | by Python 2. If *fix_imports* is true, pickle will try to map the old |
| 242 | Python 2 names to the new names used in Python 3. The *encoding* and |
Antoine Pitrou | d9dfaa9 | 2009-06-04 20:32:06 +0000 | [diff] [blame] | 243 | *errors* tell pickle how to decode 8-bit string instances pickled by Python |
Alexandre Vassalotti | d05c9ff | 2013-12-07 01:09:27 -0800 | [diff] [blame] | 244 | 2; these default to 'ASCII' and 'strict', respectively. The *encoding* can |
| 245 | be 'bytes' to read these 8-bit string instances as bytes objects. |
Alexandre Vassalotti | 758bca6 | 2008-10-18 19:25:07 +0000 | [diff] [blame] | 246 | |
Georg Brandl | 1824415 | 2009-09-02 20:34:52 +0000 | [diff] [blame] | 247 | .. function:: loads(bytes_object, \*, fix_imports=True, encoding="ASCII", errors="strict") |
Alexandre Vassalotti | 758bca6 | 2008-10-18 19:25:07 +0000 | [diff] [blame] | 248 | |
| 249 | Read a pickled object hierarchy from a :class:`bytes` object and return the |
Martin Panter | d21e0b5 | 2015-10-10 10:36:22 +0000 | [diff] [blame] | 250 | reconstituted object hierarchy specified therein. |
Alexandre Vassalotti | 758bca6 | 2008-10-18 19:25:07 +0000 | [diff] [blame] | 251 | |
Alexandre Vassalotti | d05c9ff | 2013-12-07 01:09:27 -0800 | [diff] [blame] | 252 | The protocol version of the pickle is detected automatically, so no |
| 253 | protocol argument is needed. Bytes past the pickled object's |
| 254 | representation are ignored. |
Alexandre Vassalotti | 758bca6 | 2008-10-18 19:25:07 +0000 | [diff] [blame] | 255 | |
Antoine Pitrou | d9dfaa9 | 2009-06-04 20:32:06 +0000 | [diff] [blame] | 256 | Optional keyword arguments are *fix_imports*, *encoding* and *errors*, |
Georg Brandl | 6faee4e | 2010-09-21 14:48:28 +0000 | [diff] [blame] | 257 | which are used to control compatibility support for pickle stream generated |
Alexandre Vassalotti | d05c9ff | 2013-12-07 01:09:27 -0800 | [diff] [blame] | 258 | by Python 2. If *fix_imports* is true, pickle will try to map the old |
| 259 | Python 2 names to the new names used in Python 3. The *encoding* and |
Antoine Pitrou | d9dfaa9 | 2009-06-04 20:32:06 +0000 | [diff] [blame] | 260 | *errors* tell pickle how to decode 8-bit string instances pickled by Python |
Alexandre Vassalotti | d05c9ff | 2013-12-07 01:09:27 -0800 | [diff] [blame] | 261 | 2; these default to 'ASCII' and 'strict', respectively. The *encoding* can |
| 262 | be 'bytes' to read these 8-bit string instances as bytes objects. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 263 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 264 | |
Alexandre Vassalotti | 758bca6 | 2008-10-18 19:25:07 +0000 | [diff] [blame] | 265 | The :mod:`pickle` module defines three exceptions: |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 266 | |
| 267 | .. exception:: PickleError |
| 268 | |
Alexandre Vassalotti | 5f3b63a | 2008-10-18 20:47:58 +0000 | [diff] [blame] | 269 | Common base class for the other pickling exceptions. It inherits |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 270 | :exc:`Exception`. |
| 271 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 272 | .. exception:: PicklingError |
| 273 | |
Alexandre Vassalotti | 758bca6 | 2008-10-18 19:25:07 +0000 | [diff] [blame] | 274 | Error raised when an unpicklable object is encountered by :class:`Pickler`. |
| 275 | It inherits :exc:`PickleError`. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 276 | |
Alexandre Vassalotti | 5f3b63a | 2008-10-18 20:47:58 +0000 | [diff] [blame] | 277 | Refer to :ref:`pickle-picklable` to learn what kinds of objects can be |
| 278 | pickled. |
| 279 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 280 | .. exception:: UnpicklingError |
| 281 | |
Ezio Melotti | e62aad3 | 2011-11-18 13:51:10 +0200 | [diff] [blame] | 282 | Error raised when there is a problem unpickling an object, such as a data |
Alexandre Vassalotti | 5f3b63a | 2008-10-18 20:47:58 +0000 | [diff] [blame] | 283 | corruption or a security violation. It inherits :exc:`PickleError`. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 284 | |
Alexandre Vassalotti | 758bca6 | 2008-10-18 19:25:07 +0000 | [diff] [blame] | 285 | Note that other exceptions may also be raised during unpickling, including |
| 286 | (but not necessarily limited to) AttributeError, EOFError, ImportError, and |
| 287 | IndexError. |
| 288 | |
| 289 | |
| 290 | The :mod:`pickle` module exports two classes, :class:`Pickler` and |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 291 | :class:`Unpickler`: |
| 292 | |
Georg Brandl | 1824415 | 2009-09-02 20:34:52 +0000 | [diff] [blame] | 293 | .. class:: Pickler(file, protocol=None, \*, fix_imports=True) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 294 | |
Alexandre Vassalotti | 758bca6 | 2008-10-18 19:25:07 +0000 | [diff] [blame] | 295 | This takes a binary file for writing a pickle data stream. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 296 | |
Antoine Pitrou | b645724 | 2014-01-21 02:39:54 +0100 | [diff] [blame] | 297 | The optional *protocol* argument, an integer, tells the pickler to use |
| 298 | the given protocol; supported protocols are 0 to :data:`HIGHEST_PROTOCOL`. |
| 299 | If not specified, the default is :data:`DEFAULT_PROTOCOL`. If a negative |
| 300 | number is specified, :data:`HIGHEST_PROTOCOL` is selected. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 301 | |
Alexandre Vassalotti | 758bca6 | 2008-10-18 19:25:07 +0000 | [diff] [blame] | 302 | The *file* argument must have a write() method that accepts a single bytes |
Serhiy Storchaka | d65c949 | 2015-11-02 14:10:23 +0200 | [diff] [blame] | 303 | argument. It can thus be an on-disk file opened for binary writing, an |
Alexandre Vassalotti | d05c9ff | 2013-12-07 01:09:27 -0800 | [diff] [blame] | 304 | :class:`io.BytesIO` instance, or any other custom object that meets this |
| 305 | interface. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 306 | |
Serhiy Storchaka | fbc1c26 | 2013-11-29 12:17:13 +0200 | [diff] [blame] | 307 | If *fix_imports* is true and *protocol* is less than 3, pickle will try to |
Alexandre Vassalotti | d05c9ff | 2013-12-07 01:09:27 -0800 | [diff] [blame] | 308 | map the new Python 3 names to the old module names used in Python 2, so |
| 309 | that the pickle data stream is readable with Python 2. |
Antoine Pitrou | d9dfaa9 | 2009-06-04 20:32:06 +0000 | [diff] [blame] | 310 | |
Benjamin Peterson | e41251e | 2008-04-25 01:59:09 +0000 | [diff] [blame] | 311 | .. method:: dump(obj) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 312 | |
Alexandre Vassalotti | 758bca6 | 2008-10-18 19:25:07 +0000 | [diff] [blame] | 313 | Write a pickled representation of *obj* to the open file object given in |
| 314 | the constructor. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 315 | |
Alexandre Vassalotti | 758bca6 | 2008-10-18 19:25:07 +0000 | [diff] [blame] | 316 | .. method:: persistent_id(obj) |
| 317 | |
| 318 | Do nothing by default. This exists so a subclass can override it. |
| 319 | |
| 320 | If :meth:`persistent_id` returns ``None``, *obj* is pickled as usual. Any |
| 321 | other value causes :class:`Pickler` to emit the returned value as a |
| 322 | persistent ID for *obj*. The meaning of this persistent ID should be |
| 323 | defined by :meth:`Unpickler.persistent_load`. Note that the value |
| 324 | returned by :meth:`persistent_id` cannot itself have a persistent ID. |
| 325 | |
| 326 | See :ref:`pickle-persistent` for details and examples of uses. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 327 | |
Antoine Pitrou | 8d3c290 | 2012-03-04 18:31:48 +0100 | [diff] [blame] | 328 | .. attribute:: dispatch_table |
| 329 | |
| 330 | A pickler object's dispatch table is a registry of *reduction |
| 331 | functions* of the kind which can be declared using |
| 332 | :func:`copyreg.pickle`. It is a mapping whose keys are classes |
| 333 | and whose values are reduction functions. A reduction function |
| 334 | takes a single argument of the associated class and should |
Serhiy Storchaka | 5bbbc94 | 2013-10-14 10:43:46 +0300 | [diff] [blame] | 335 | conform to the same interface as a :meth:`__reduce__` |
Antoine Pitrou | 8d3c290 | 2012-03-04 18:31:48 +0100 | [diff] [blame] | 336 | method. |
| 337 | |
| 338 | By default, a pickler object will not have a |
| 339 | :attr:`dispatch_table` attribute, and it will instead use the |
| 340 | global dispatch table managed by the :mod:`copyreg` module. |
| 341 | However, to customize the pickling for a specific pickler object |
| 342 | one can set the :attr:`dispatch_table` attribute to a dict-like |
| 343 | object. Alternatively, if a subclass of :class:`Pickler` has a |
| 344 | :attr:`dispatch_table` attribute then this will be used as the |
| 345 | default dispatch table for instances of that class. |
| 346 | |
| 347 | See :ref:`pickle-dispatch` for usage examples. |
| 348 | |
| 349 | .. versionadded:: 3.3 |
| 350 | |
Alexandre Vassalotti | 758bca6 | 2008-10-18 19:25:07 +0000 | [diff] [blame] | 351 | .. attribute:: fast |
| 352 | |
Alexandre Vassalotti | 73b90a8 | 2008-10-29 23:32:33 +0000 | [diff] [blame] | 353 | Deprecated. Enable fast mode if set to a true value. The fast mode |
| 354 | disables the usage of memo, therefore speeding the pickling process by not |
| 355 | generating superfluous PUT opcodes. It should not be used with |
| 356 | self-referential objects, doing otherwise will cause :class:`Pickler` to |
| 357 | recurse infinitely. |
Alexandre Vassalotti | 758bca6 | 2008-10-18 19:25:07 +0000 | [diff] [blame] | 358 | |
| 359 | Use :func:`pickletools.optimize` if you need more compact pickles. |
| 360 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 361 | |
Georg Brandl | 1824415 | 2009-09-02 20:34:52 +0000 | [diff] [blame] | 362 | .. class:: Unpickler(file, \*, fix_imports=True, encoding="ASCII", errors="strict") |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 363 | |
Alexandre Vassalotti | 758bca6 | 2008-10-18 19:25:07 +0000 | [diff] [blame] | 364 | This takes a binary file for reading a pickle data stream. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 365 | |
Alexandre Vassalotti | 758bca6 | 2008-10-18 19:25:07 +0000 | [diff] [blame] | 366 | The protocol version of the pickle is detected automatically, so no |
| 367 | protocol argument is needed. |
| 368 | |
| 369 | The argument *file* must have two methods, a read() method that takes an |
| 370 | integer argument, and a readline() method that requires no arguments. Both |
Alexandre Vassalotti | d05c9ff | 2013-12-07 01:09:27 -0800 | [diff] [blame] | 371 | methods should return bytes. Thus *file* can be an on-disk file object |
Martin Panter | 7462b649 | 2015-11-02 03:37:02 +0000 | [diff] [blame] | 372 | opened for binary reading, an :class:`io.BytesIO` object, or any other |
Alexandre Vassalotti | d05c9ff | 2013-12-07 01:09:27 -0800 | [diff] [blame] | 373 | custom object that meets this interface. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 374 | |
Antoine Pitrou | d9dfaa9 | 2009-06-04 20:32:06 +0000 | [diff] [blame] | 375 | Optional keyword arguments are *fix_imports*, *encoding* and *errors*, |
Georg Brandl | 6faee4e | 2010-09-21 14:48:28 +0000 | [diff] [blame] | 376 | which are used to control compatibility support for pickle stream generated |
Alexandre Vassalotti | d05c9ff | 2013-12-07 01:09:27 -0800 | [diff] [blame] | 377 | by Python 2. If *fix_imports* is true, pickle will try to map the old |
| 378 | Python 2 names to the new names used in Python 3. The *encoding* and |
Antoine Pitrou | d9dfaa9 | 2009-06-04 20:32:06 +0000 | [diff] [blame] | 379 | *errors* tell pickle how to decode 8-bit string instances pickled by Python |
Alexandre Vassalotti | d05c9ff | 2013-12-07 01:09:27 -0800 | [diff] [blame] | 380 | 2; these default to 'ASCII' and 'strict', respectively. The *encoding* can |
Sebastian Pucilowski | a8d25a1 | 2017-12-21 20:00:49 +1100 | [diff] [blame] | 381 | be 'bytes' to read these 8-bit string instances as bytes objects. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 382 | |
Benjamin Peterson | e41251e | 2008-04-25 01:59:09 +0000 | [diff] [blame] | 383 | .. method:: load() |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 384 | |
Benjamin Peterson | e41251e | 2008-04-25 01:59:09 +0000 | [diff] [blame] | 385 | Read a pickled object representation from the open file object given in |
| 386 | the constructor, and return the reconstituted object hierarchy specified |
Alexandre Vassalotti | 5f3b63a | 2008-10-18 20:47:58 +0000 | [diff] [blame] | 387 | therein. Bytes past the pickled object's representation are ignored. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 388 | |
Alexandre Vassalotti | 758bca6 | 2008-10-18 19:25:07 +0000 | [diff] [blame] | 389 | .. method:: persistent_load(pid) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 390 | |
Ezio Melotti | e62aad3 | 2011-11-18 13:51:10 +0200 | [diff] [blame] | 391 | Raise an :exc:`UnpicklingError` by default. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 392 | |
Alexandre Vassalotti | 758bca6 | 2008-10-18 19:25:07 +0000 | [diff] [blame] | 393 | If defined, :meth:`persistent_load` should return the object specified by |
Alexandre Vassalotti | 5f3b63a | 2008-10-18 20:47:58 +0000 | [diff] [blame] | 394 | the persistent ID *pid*. If an invalid persistent ID is encountered, an |
Ezio Melotti | e62aad3 | 2011-11-18 13:51:10 +0200 | [diff] [blame] | 395 | :exc:`UnpicklingError` should be raised. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 396 | |
Alexandre Vassalotti | 758bca6 | 2008-10-18 19:25:07 +0000 | [diff] [blame] | 397 | See :ref:`pickle-persistent` for details and examples of uses. |
| 398 | |
| 399 | .. method:: find_class(module, name) |
| 400 | |
Alexandre Vassalotti | 5f3b63a | 2008-10-18 20:47:58 +0000 | [diff] [blame] | 401 | Import *module* if necessary and return the object called *name* from it, |
Alexandre Vassalotti | d039286 | 2008-10-24 01:32:40 +0000 | [diff] [blame] | 402 | where the *module* and *name* arguments are :class:`str` objects. Note, |
| 403 | unlike its name suggests, :meth:`find_class` is also used for finding |
| 404 | functions. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 405 | |
Alexandre Vassalotti | 5f3b63a | 2008-10-18 20:47:58 +0000 | [diff] [blame] | 406 | Subclasses may override this to gain control over what type of objects and |
Alexandre Vassalotti | d039286 | 2008-10-24 01:32:40 +0000 | [diff] [blame] | 407 | how they can be loaded, potentially reducing security risks. Refer to |
| 408 | :ref:`pickle-restrict` for details. |
Alexandre Vassalotti | 5f3b63a | 2008-10-18 20:47:58 +0000 | [diff] [blame] | 409 | |
| 410 | |
| 411 | .. _pickle-picklable: |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 412 | |
| 413 | What can be pickled and unpickled? |
| 414 | ---------------------------------- |
| 415 | |
| 416 | The following types can be pickled: |
| 417 | |
| 418 | * ``None``, ``True``, and ``False`` |
| 419 | |
Georg Brandl | ba956ae | 2007-11-29 17:24:34 +0000 | [diff] [blame] | 420 | * integers, floating point numbers, complex numbers |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 421 | |
Georg Brandl | f694518 | 2008-02-01 11:56:49 +0000 | [diff] [blame] | 422 | * strings, bytes, bytearrays |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 423 | |
| 424 | * tuples, lists, sets, and dictionaries containing only picklable objects |
| 425 | |
Ethan Furman | 2498d9e | 2013-10-18 00:45:40 -0700 | [diff] [blame] | 426 | * functions defined at the top level of a module (using :keyword:`def`, not |
| 427 | :keyword:`lambda`) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 428 | |
| 429 | * built-in functions defined at the top level of a module |
| 430 | |
| 431 | * classes that are defined at the top level of a module |
| 432 | |
Serhiy Storchaka | 5bbbc94 | 2013-10-14 10:43:46 +0300 | [diff] [blame] | 433 | * instances of such classes whose :attr:`~object.__dict__` or the result of |
| 434 | calling :meth:`__getstate__` is picklable (see section :ref:`pickle-inst` for |
Eli Bendersky | 78f3ce5 | 2013-01-02 05:53:59 -0800 | [diff] [blame] | 435 | details). |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 436 | |
| 437 | Attempts to pickle unpicklable objects will raise the :exc:`PicklingError` |
| 438 | exception; when this happens, an unspecified number of bytes may have already |
Alexandre Vassalotti | 5f3b63a | 2008-10-18 20:47:58 +0000 | [diff] [blame] | 439 | been written to the underlying file. Trying to pickle a highly recursive data |
Yury Selivanov | f488fb4 | 2015-07-03 01:04:23 -0400 | [diff] [blame] | 440 | structure may exceed the maximum recursion depth, a :exc:`RecursionError` will be |
Alexandre Vassalotti | 5f3b63a | 2008-10-18 20:47:58 +0000 | [diff] [blame] | 441 | raised in this case. You can carefully raise this limit with |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 442 | :func:`sys.setrecursionlimit`. |
| 443 | |
| 444 | Note that functions (built-in and user-defined) are pickled by "fully qualified" |
Ethan Furman | 2498d9e | 2013-10-18 00:45:40 -0700 | [diff] [blame] | 445 | name reference, not by value. [#]_ This means that only the function name is |
Eli Bendersky | 78f3ce5 | 2013-01-02 05:53:59 -0800 | [diff] [blame] | 446 | pickled, along with the name of the module the function is defined in. Neither |
| 447 | the function's code, nor any of its function attributes are pickled. Thus the |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 448 | defining module must be importable in the unpickling environment, and the module |
| 449 | must contain the named object, otherwise an exception will be raised. [#]_ |
| 450 | |
| 451 | Similarly, classes are pickled by named reference, so the same restrictions in |
| 452 | the unpickling environment apply. Note that none of the class's code or data is |
| 453 | pickled, so in the following example the class attribute ``attr`` is not |
| 454 | restored in the unpickling environment:: |
| 455 | |
| 456 | class Foo: |
Alexandre Vassalotti | 5f3b63a | 2008-10-18 20:47:58 +0000 | [diff] [blame] | 457 | attr = 'A class attribute' |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 458 | |
| 459 | picklestring = pickle.dumps(Foo) |
| 460 | |
| 461 | These restrictions are why picklable functions and classes must be defined in |
| 462 | the top level of a module. |
| 463 | |
| 464 | Similarly, when class instances are pickled, their class's code and data are not |
| 465 | pickled along with them. Only the instance data are pickled. This is done on |
| 466 | purpose, so you can fix bugs in a class or add methods to the class and still |
| 467 | load objects that were created with an earlier version of the class. If you |
| 468 | plan to have long-lived objects that will see many versions of a class, it may |
| 469 | be worthwhile to put a version number in the objects so that suitable |
| 470 | conversions can be made by the class's :meth:`__setstate__` method. |
| 471 | |
| 472 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 473 | .. _pickle-inst: |
| 474 | |
Alexandre Vassalotti | 73b90a8 | 2008-10-29 23:32:33 +0000 | [diff] [blame] | 475 | Pickling Class Instances |
| 476 | ------------------------ |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 477 | |
Serhiy Storchaka | 5bbbc94 | 2013-10-14 10:43:46 +0300 | [diff] [blame] | 478 | .. currentmodule:: None |
| 479 | |
Alexandre Vassalotti | 73b90a8 | 2008-10-29 23:32:33 +0000 | [diff] [blame] | 480 | In this section, we describe the general mechanisms available to you to define, |
| 481 | customize, and control how class instances are pickled and unpickled. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 482 | |
Alexandre Vassalotti | 73b90a8 | 2008-10-29 23:32:33 +0000 | [diff] [blame] | 483 | In most cases, no additional code is needed to make instances picklable. By |
| 484 | default, pickle will retrieve the class and the attributes of an instance via |
| 485 | introspection. When a class instance is unpickled, its :meth:`__init__` method |
| 486 | is usually *not* invoked. The default behaviour first creates an uninitialized |
| 487 | instance and then restores the saved attributes. The following code shows an |
| 488 | implementation of this behaviour:: |
Georg Brandl | 85eb8c1 | 2007-08-31 16:33:38 +0000 | [diff] [blame] | 489 | |
Alexandre Vassalotti | 73b90a8 | 2008-10-29 23:32:33 +0000 | [diff] [blame] | 490 | def save(obj): |
| 491 | return (obj.__class__, obj.__dict__) |
| 492 | |
| 493 | def load(cls, attributes): |
| 494 | obj = cls.__new__(cls) |
| 495 | obj.__dict__.update(attributes) |
| 496 | return obj |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 497 | |
Georg Brandl | 6faee4e | 2010-09-21 14:48:28 +0000 | [diff] [blame] | 498 | Classes can alter the default behaviour by providing one or several special |
Georg Brandl | c814826 | 2010-10-17 11:13:37 +0000 | [diff] [blame] | 499 | methods: |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 500 | |
Antoine Pitrou | c9dc4a2 | 2013-11-23 18:59:12 +0100 | [diff] [blame] | 501 | .. method:: object.__getnewargs_ex__() |
| 502 | |
Serhiy Storchaka | b6d8483 | 2015-10-13 21:26:35 +0300 | [diff] [blame] | 503 | In protocols 2 and newer, classes that implements the |
Antoine Pitrou | c9dc4a2 | 2013-11-23 18:59:12 +0100 | [diff] [blame] | 504 | :meth:`__getnewargs_ex__` method can dictate the values passed to the |
| 505 | :meth:`__new__` method upon unpickling. The method must return a pair |
| 506 | ``(args, kwargs)`` where *args* is a tuple of positional arguments |
| 507 | and *kwargs* a dictionary of named arguments for constructing the |
| 508 | object. Those will be passed to the :meth:`__new__` method upon |
| 509 | unpickling. |
| 510 | |
| 511 | You should implement this method if the :meth:`__new__` method of your |
| 512 | class requires keyword-only arguments. Otherwise, it is recommended for |
| 513 | compatibility to implement :meth:`__getnewargs__`. |
| 514 | |
Serhiy Storchaka | b6d8483 | 2015-10-13 21:26:35 +0300 | [diff] [blame] | 515 | .. versionchanged:: 3.6 |
| 516 | :meth:`__getnewargs_ex__` is now used in protocols 2 and 3. |
| 517 | |
Antoine Pitrou | c9dc4a2 | 2013-11-23 18:59:12 +0100 | [diff] [blame] | 518 | |
Georg Brandl | c814826 | 2010-10-17 11:13:37 +0000 | [diff] [blame] | 519 | .. method:: object.__getnewargs__() |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 520 | |
Andrés Delfino | 0e0534c | 2018-06-09 21:41:09 -0300 | [diff] [blame] | 521 | This method serves a similar purpose as :meth:`__getnewargs_ex__`, but |
Serhiy Storchaka | b6d8483 | 2015-10-13 21:26:35 +0300 | [diff] [blame] | 522 | supports only positional arguments. It must return a tuple of arguments |
| 523 | ``args`` which will be passed to the :meth:`__new__` method upon unpickling. |
Antoine Pitrou | c9dc4a2 | 2013-11-23 18:59:12 +0100 | [diff] [blame] | 524 | |
Serhiy Storchaka | b6d8483 | 2015-10-13 21:26:35 +0300 | [diff] [blame] | 525 | :meth:`__getnewargs__` will not be called if :meth:`__getnewargs_ex__` is |
| 526 | defined. |
| 527 | |
| 528 | .. versionchanged:: 3.6 |
| 529 | Before Python 3.6, :meth:`__getnewargs__` was called instead of |
| 530 | :meth:`__getnewargs_ex__` in protocols 2 and 3. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 531 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 532 | |
Georg Brandl | c814826 | 2010-10-17 11:13:37 +0000 | [diff] [blame] | 533 | .. method:: object.__getstate__() |
Alexandre Vassalotti | 73b90a8 | 2008-10-29 23:32:33 +0000 | [diff] [blame] | 534 | |
Georg Brandl | c814826 | 2010-10-17 11:13:37 +0000 | [diff] [blame] | 535 | Classes can further influence how their instances are pickled; if the class |
| 536 | defines the method :meth:`__getstate__`, it is called and the returned object |
| 537 | is pickled as the contents for the instance, instead of the contents of the |
| 538 | instance's dictionary. If the :meth:`__getstate__` method is absent, the |
Serhiy Storchaka | 5bbbc94 | 2013-10-14 10:43:46 +0300 | [diff] [blame] | 539 | instance's :attr:`~object.__dict__` is pickled as usual. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 540 | |
Georg Brandl | c814826 | 2010-10-17 11:13:37 +0000 | [diff] [blame] | 541 | |
| 542 | .. method:: object.__setstate__(state) |
| 543 | |
| 544 | Upon unpickling, if the class defines :meth:`__setstate__`, it is called with |
| 545 | the unpickled state. In that case, there is no requirement for the state |
| 546 | object to be a dictionary. Otherwise, the pickled state must be a dictionary |
| 547 | and its items are assigned to the new instance's dictionary. |
| 548 | |
| 549 | .. note:: |
| 550 | |
| 551 | If :meth:`__getstate__` returns a false value, the :meth:`__setstate__` |
| 552 | method will not be called upon unpickling. |
| 553 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 554 | |
Alexandre Vassalotti | 73b90a8 | 2008-10-29 23:32:33 +0000 | [diff] [blame] | 555 | Refer to the section :ref:`pickle-state` for more information about how to use |
| 556 | the methods :meth:`__getstate__` and :meth:`__setstate__`. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 557 | |
Benjamin Peterson | d23f822 | 2009-04-05 19:13:16 +0000 | [diff] [blame] | 558 | .. note:: |
Georg Brandl | e720c0a | 2009-04-27 16:20:50 +0000 | [diff] [blame] | 559 | |
Benjamin Peterson | d23f822 | 2009-04-05 19:13:16 +0000 | [diff] [blame] | 560 | At unpickling time, some methods like :meth:`__getattr__`, |
| 561 | :meth:`__getattribute__`, or :meth:`__setattr__` may be called upon the |
Antoine Pitrou | c9dc4a2 | 2013-11-23 18:59:12 +0100 | [diff] [blame] | 562 | instance. In case those methods rely on some internal invariant being |
| 563 | true, the type should implement :meth:`__getnewargs__` or |
| 564 | :meth:`__getnewargs_ex__` to establish such an invariant; otherwise, |
| 565 | neither :meth:`__new__` nor :meth:`__init__` will be called. |
Benjamin Peterson | d23f822 | 2009-04-05 19:13:16 +0000 | [diff] [blame] | 566 | |
Georg Brandl | c814826 | 2010-10-17 11:13:37 +0000 | [diff] [blame] | 567 | .. index:: pair: copy; protocol |
Christian Heimes | 05e8be1 | 2008-02-23 18:30:17 +0000 | [diff] [blame] | 568 | |
Alexandre Vassalotti | 73b90a8 | 2008-10-29 23:32:33 +0000 | [diff] [blame] | 569 | As we shall see, pickle does not use directly the methods described above. In |
| 570 | fact, these methods are part of the copy protocol which implements the |
| 571 | :meth:`__reduce__` special method. The copy protocol provides a unified |
| 572 | interface for retrieving the data necessary for pickling and copying |
Georg Brandl | 48310cd | 2009-01-03 21:18:54 +0000 | [diff] [blame] | 573 | objects. [#]_ |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 574 | |
Alexandre Vassalotti | 73b90a8 | 2008-10-29 23:32:33 +0000 | [diff] [blame] | 575 | Although powerful, implementing :meth:`__reduce__` directly in your classes is |
| 576 | error prone. For this reason, class designers should use the high-level |
Antoine Pitrou | c9dc4a2 | 2013-11-23 18:59:12 +0100 | [diff] [blame] | 577 | interface (i.e., :meth:`__getnewargs_ex__`, :meth:`__getstate__` and |
Georg Brandl | c814826 | 2010-10-17 11:13:37 +0000 | [diff] [blame] | 578 | :meth:`__setstate__`) whenever possible. We will show, however, cases where |
| 579 | using :meth:`__reduce__` is the only option or leads to more efficient pickling |
| 580 | or both. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 581 | |
Georg Brandl | c814826 | 2010-10-17 11:13:37 +0000 | [diff] [blame] | 582 | .. method:: object.__reduce__() |
Alexandre Vassalotti | 73b90a8 | 2008-10-29 23:32:33 +0000 | [diff] [blame] | 583 | |
Georg Brandl | c814826 | 2010-10-17 11:13:37 +0000 | [diff] [blame] | 584 | The interface is currently defined as follows. The :meth:`__reduce__` method |
| 585 | takes no argument and shall return either a string or preferably a tuple (the |
| 586 | returned object is often referred to as the "reduce value"). |
Alexandre Vassalotti | 73b90a8 | 2008-10-29 23:32:33 +0000 | [diff] [blame] | 587 | |
Georg Brandl | c814826 | 2010-10-17 11:13:37 +0000 | [diff] [blame] | 588 | If a string is returned, the string should be interpreted as the name of a |
| 589 | global variable. It should be the object's local name relative to its |
| 590 | module; the pickle module searches the module namespace to determine the |
| 591 | object's module. This behaviour is typically useful for singletons. |
Alexandre Vassalotti | 73b90a8 | 2008-10-29 23:32:33 +0000 | [diff] [blame] | 592 | |
Georg Brandl | c814826 | 2010-10-17 11:13:37 +0000 | [diff] [blame] | 593 | When a tuple is returned, it must be between two and five items long. |
| 594 | Optional items can either be omitted, or ``None`` can be provided as their |
| 595 | value. The semantics of each item are in order: |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 596 | |
Georg Brandl | c814826 | 2010-10-17 11:13:37 +0000 | [diff] [blame] | 597 | .. XXX Mention __newobj__ special-case? |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 598 | |
Georg Brandl | c814826 | 2010-10-17 11:13:37 +0000 | [diff] [blame] | 599 | * A callable object that will be called to create the initial version of the |
| 600 | object. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 601 | |
Georg Brandl | c814826 | 2010-10-17 11:13:37 +0000 | [diff] [blame] | 602 | * A tuple of arguments for the callable object. An empty tuple must be given |
| 603 | if the callable does not accept any argument. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 604 | |
Georg Brandl | c814826 | 2010-10-17 11:13:37 +0000 | [diff] [blame] | 605 | * Optionally, the object's state, which will be passed to the object's |
| 606 | :meth:`__setstate__` method as previously described. If the object has no |
| 607 | such method then, the value must be a dictionary and it will be added to |
Serhiy Storchaka | 5bbbc94 | 2013-10-14 10:43:46 +0300 | [diff] [blame] | 608 | the object's :attr:`~object.__dict__` attribute. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 609 | |
Georg Brandl | c814826 | 2010-10-17 11:13:37 +0000 | [diff] [blame] | 610 | * Optionally, an iterator (and not a sequence) yielding successive items. |
| 611 | These items will be appended to the object either using |
| 612 | ``obj.append(item)`` or, in batch, using ``obj.extend(list_of_items)``. |
| 613 | This is primarily used for list subclasses, but may be used by other |
| 614 | classes as long as they have :meth:`append` and :meth:`extend` methods with |
| 615 | the appropriate signature. (Whether :meth:`append` or :meth:`extend` is |
| 616 | used depends on which pickle protocol version is used as well as the number |
| 617 | of items to append, so both must be supported.) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 618 | |
Georg Brandl | c814826 | 2010-10-17 11:13:37 +0000 | [diff] [blame] | 619 | * Optionally, an iterator (not a sequence) yielding successive key-value |
| 620 | pairs. These items will be stored to the object using ``obj[key] = |
| 621 | value``. This is primarily used for dictionary subclasses, but may be used |
| 622 | by other classes as long as they implement :meth:`__setitem__`. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 623 | |
Georg Brandl | c814826 | 2010-10-17 11:13:37 +0000 | [diff] [blame] | 624 | |
| 625 | .. method:: object.__reduce_ex__(protocol) |
| 626 | |
| 627 | Alternatively, a :meth:`__reduce_ex__` method may be defined. The only |
| 628 | difference is this method should take a single integer argument, the protocol |
| 629 | version. When defined, pickle will prefer it over the :meth:`__reduce__` |
| 630 | method. In addition, :meth:`__reduce__` automatically becomes a synonym for |
| 631 | the extended version. The main use for this method is to provide |
| 632 | backwards-compatible reduce values for older Python releases. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 633 | |
Serhiy Storchaka | 5bbbc94 | 2013-10-14 10:43:46 +0300 | [diff] [blame] | 634 | .. currentmodule:: pickle |
| 635 | |
Alexandre Vassalotti | 758bca6 | 2008-10-18 19:25:07 +0000 | [diff] [blame] | 636 | .. _pickle-persistent: |
| 637 | |
Alexandre Vassalotti | 73b90a8 | 2008-10-29 23:32:33 +0000 | [diff] [blame] | 638 | Persistence of External Objects |
| 639 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 640 | |
Christian Heimes | 05e8be1 | 2008-02-23 18:30:17 +0000 | [diff] [blame] | 641 | .. index:: |
| 642 | single: persistent_id (pickle protocol) |
| 643 | single: persistent_load (pickle protocol) |
| 644 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 645 | For the benefit of object persistence, the :mod:`pickle` module supports the |
| 646 | notion of a reference to an object outside the pickled data stream. Such |
Alexandre Vassalotti | 5f3b63a | 2008-10-18 20:47:58 +0000 | [diff] [blame] | 647 | objects are referenced by a persistent ID, which should be either a string of |
| 648 | alphanumeric characters (for protocol 0) [#]_ or just an arbitrary object (for |
| 649 | any newer protocol). |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 650 | |
Alexandre Vassalotti | 5f3b63a | 2008-10-18 20:47:58 +0000 | [diff] [blame] | 651 | The resolution of such persistent IDs is not defined by the :mod:`pickle` |
| 652 | module; it will delegate this resolution to the user defined methods on the |
Serhiy Storchaka | 5bbbc94 | 2013-10-14 10:43:46 +0300 | [diff] [blame] | 653 | pickler and unpickler, :meth:`~Pickler.persistent_id` and |
| 654 | :meth:`~Unpickler.persistent_load` respectively. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 655 | |
| 656 | To pickle objects that have an external persistent id, the pickler must have a |
Serhiy Storchaka | 5bbbc94 | 2013-10-14 10:43:46 +0300 | [diff] [blame] | 657 | custom :meth:`~Pickler.persistent_id` method that takes an object as an |
| 658 | argument and returns either ``None`` or the persistent id for that object. |
| 659 | When ``None`` is returned, the pickler simply pickles the object as normal. |
| 660 | When a persistent ID string is returned, the pickler will pickle that object, |
| 661 | along with a marker so that the unpickler will recognize it as a persistent ID. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 662 | |
| 663 | To unpickle external objects, the unpickler must have a custom |
Serhiy Storchaka | 5bbbc94 | 2013-10-14 10:43:46 +0300 | [diff] [blame] | 664 | :meth:`~Unpickler.persistent_load` method that takes a persistent ID object and |
| 665 | returns the referenced object. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 666 | |
Alexandre Vassalotti | 73b90a8 | 2008-10-29 23:32:33 +0000 | [diff] [blame] | 667 | Here is a comprehensive example presenting how persistent ID can be used to |
| 668 | pickle external objects by reference. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 669 | |
Alexandre Vassalotti | 5f3b63a | 2008-10-18 20:47:58 +0000 | [diff] [blame] | 670 | .. literalinclude:: ../includes/dbpickle.py |
Alexandre Vassalotti | bcd1e3a | 2009-01-23 05:28:16 +0000 | [diff] [blame] | 671 | |
Antoine Pitrou | 8d3c290 | 2012-03-04 18:31:48 +0100 | [diff] [blame] | 672 | .. _pickle-dispatch: |
| 673 | |
| 674 | Dispatch Tables |
| 675 | ^^^^^^^^^^^^^^^ |
| 676 | |
| 677 | If one wants to customize pickling of some classes without disturbing |
| 678 | any other code which depends on pickling, then one can create a |
| 679 | pickler with a private dispatch table. |
| 680 | |
| 681 | The global dispatch table managed by the :mod:`copyreg` module is |
| 682 | available as :data:`copyreg.dispatch_table`. Therefore, one may |
| 683 | choose to use a modified copy of :data:`copyreg.dispatch_table` as a |
| 684 | private dispatch table. |
| 685 | |
| 686 | For example :: |
| 687 | |
| 688 | f = io.BytesIO() |
| 689 | p = pickle.Pickler(f) |
| 690 | p.dispatch_table = copyreg.dispatch_table.copy() |
| 691 | p.dispatch_table[SomeClass] = reduce_SomeClass |
| 692 | |
| 693 | creates an instance of :class:`pickle.Pickler` with a private dispatch |
| 694 | table which handles the ``SomeClass`` class specially. Alternatively, |
| 695 | the code :: |
| 696 | |
| 697 | class MyPickler(pickle.Pickler): |
| 698 | dispatch_table = copyreg.dispatch_table.copy() |
| 699 | dispatch_table[SomeClass] = reduce_SomeClass |
| 700 | f = io.BytesIO() |
| 701 | p = MyPickler(f) |
| 702 | |
| 703 | does the same, but all instances of ``MyPickler`` will by default |
| 704 | share the same dispatch table. The equivalent code using the |
| 705 | :mod:`copyreg` module is :: |
| 706 | |
| 707 | copyreg.pickle(SomeClass, reduce_SomeClass) |
| 708 | f = io.BytesIO() |
| 709 | p = pickle.Pickler(f) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 710 | |
Alexandre Vassalotti | 73b90a8 | 2008-10-29 23:32:33 +0000 | [diff] [blame] | 711 | .. _pickle-state: |
| 712 | |
| 713 | Handling Stateful Objects |
| 714 | ^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 715 | |
| 716 | .. index:: |
| 717 | single: __getstate__() (copy protocol) |
| 718 | single: __setstate__() (copy protocol) |
| 719 | |
| 720 | Here's an example that shows how to modify pickling behavior for a class. |
| 721 | The :class:`TextReader` class opens a text file, and returns the line number and |
Serhiy Storchaka | 5bbbc94 | 2013-10-14 10:43:46 +0300 | [diff] [blame] | 722 | line contents each time its :meth:`!readline` method is called. If a |
Alexandre Vassalotti | 73b90a8 | 2008-10-29 23:32:33 +0000 | [diff] [blame] | 723 | :class:`TextReader` instance is pickled, all attributes *except* the file object |
| 724 | member are saved. When the instance is unpickled, the file is reopened, and |
| 725 | reading resumes from the last location. The :meth:`__setstate__` and |
| 726 | :meth:`__getstate__` methods are used to implement this behavior. :: |
| 727 | |
| 728 | class TextReader: |
| 729 | """Print and number lines in a text file.""" |
| 730 | |
| 731 | def __init__(self, filename): |
| 732 | self.filename = filename |
| 733 | self.file = open(filename) |
| 734 | self.lineno = 0 |
| 735 | |
| 736 | def readline(self): |
| 737 | self.lineno += 1 |
| 738 | line = self.file.readline() |
| 739 | if not line: |
| 740 | return None |
Alexandre Vassalotti | 9d7665d | 2009-04-03 06:13:29 +0000 | [diff] [blame] | 741 | if line.endswith('\n'): |
Alexandre Vassalotti | 73b90a8 | 2008-10-29 23:32:33 +0000 | [diff] [blame] | 742 | line = line[:-1] |
| 743 | return "%i: %s" % (self.lineno, line) |
| 744 | |
| 745 | def __getstate__(self): |
| 746 | # Copy the object's state from self.__dict__ which contains |
| 747 | # all our instance attributes. Always use the dict.copy() |
| 748 | # method to avoid modifying the original state. |
| 749 | state = self.__dict__.copy() |
| 750 | # Remove the unpicklable entries. |
| 751 | del state['file'] |
| 752 | return state |
| 753 | |
| 754 | def __setstate__(self, state): |
| 755 | # Restore instance attributes (i.e., filename and lineno). |
| 756 | self.__dict__.update(state) |
| 757 | # Restore the previously opened file's state. To do so, we need to |
| 758 | # reopen it and read from it until the line count is restored. |
| 759 | file = open(self.filename) |
| 760 | for _ in range(self.lineno): |
| 761 | file.readline() |
| 762 | # Finally, save the file. |
| 763 | self.file = file |
| 764 | |
| 765 | |
| 766 | A sample usage might be something like this:: |
| 767 | |
| 768 | >>> reader = TextReader("hello.txt") |
| 769 | >>> reader.readline() |
| 770 | '1: Hello world!' |
| 771 | >>> reader.readline() |
| 772 | '2: I am line number two.' |
| 773 | >>> new_reader = pickle.loads(pickle.dumps(reader)) |
| 774 | >>> new_reader.readline() |
| 775 | '3: Goodbye!' |
| 776 | |
| 777 | |
Alexandre Vassalotti | d039286 | 2008-10-24 01:32:40 +0000 | [diff] [blame] | 778 | .. _pickle-restrict: |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 779 | |
Alexandre Vassalotti | d039286 | 2008-10-24 01:32:40 +0000 | [diff] [blame] | 780 | Restricting Globals |
Alexandre Vassalotti | 73b90a8 | 2008-10-29 23:32:33 +0000 | [diff] [blame] | 781 | ------------------- |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 782 | |
Christian Heimes | 05e8be1 | 2008-02-23 18:30:17 +0000 | [diff] [blame] | 783 | .. index:: |
Alexandre Vassalotti | d039286 | 2008-10-24 01:32:40 +0000 | [diff] [blame] | 784 | single: find_class() (pickle protocol) |
Christian Heimes | 05e8be1 | 2008-02-23 18:30:17 +0000 | [diff] [blame] | 785 | |
Alexandre Vassalotti | d039286 | 2008-10-24 01:32:40 +0000 | [diff] [blame] | 786 | By default, unpickling will import any class or function that it finds in the |
| 787 | pickle data. For many applications, this behaviour is unacceptable as it |
| 788 | permits the unpickler to import and invoke arbitrary code. Just consider what |
| 789 | this hand-crafted pickle data stream does when loaded:: |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 790 | |
Alexandre Vassalotti | d039286 | 2008-10-24 01:32:40 +0000 | [diff] [blame] | 791 | >>> import pickle |
| 792 | >>> pickle.loads(b"cos\nsystem\n(S'echo hello world'\ntR.") |
| 793 | hello world |
| 794 | 0 |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 795 | |
Alexandre Vassalotti | d039286 | 2008-10-24 01:32:40 +0000 | [diff] [blame] | 796 | In this example, the unpickler imports the :func:`os.system` function and then |
| 797 | apply the string argument "echo hello world". Although this example is |
| 798 | inoffensive, it is not difficult to imagine one that could damage your system. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 799 | |
Alexandre Vassalotti | d039286 | 2008-10-24 01:32:40 +0000 | [diff] [blame] | 800 | For this reason, you may want to control what gets unpickled by customizing |
Serhiy Storchaka | 5bbbc94 | 2013-10-14 10:43:46 +0300 | [diff] [blame] | 801 | :meth:`Unpickler.find_class`. Unlike its name suggests, |
| 802 | :meth:`Unpickler.find_class` is called whenever a global (i.e., a class or |
| 803 | a function) is requested. Thus it is possible to either completely forbid |
| 804 | globals or restrict them to a safe subset. |
Alexandre Vassalotti | d039286 | 2008-10-24 01:32:40 +0000 | [diff] [blame] | 805 | |
| 806 | Here is an example of an unpickler allowing only few safe classes from the |
| 807 | :mod:`builtins` module to be loaded:: |
| 808 | |
| 809 | import builtins |
| 810 | import io |
| 811 | import pickle |
| 812 | |
| 813 | safe_builtins = { |
| 814 | 'range', |
| 815 | 'complex', |
| 816 | 'set', |
| 817 | 'frozenset', |
| 818 | 'slice', |
| 819 | } |
| 820 | |
| 821 | class RestrictedUnpickler(pickle.Unpickler): |
Alexandre Vassalotti | 73b90a8 | 2008-10-29 23:32:33 +0000 | [diff] [blame] | 822 | |
Alexandre Vassalotti | d039286 | 2008-10-24 01:32:40 +0000 | [diff] [blame] | 823 | def find_class(self, module, name): |
| 824 | # Only allow safe classes from builtins. |
| 825 | if module == "builtins" and name in safe_builtins: |
| 826 | return getattr(builtins, name) |
| 827 | # Forbid everything else. |
| 828 | raise pickle.UnpicklingError("global '%s.%s' is forbidden" % |
| 829 | (module, name)) |
| 830 | |
| 831 | def restricted_loads(s): |
| 832 | """Helper function analogous to pickle.loads().""" |
| 833 | return RestrictedUnpickler(io.BytesIO(s)).load() |
| 834 | |
| 835 | A sample usage of our unpickler working has intended:: |
| 836 | |
| 837 | >>> restricted_loads(pickle.dumps([1, 2, range(15)])) |
| 838 | [1, 2, range(0, 15)] |
| 839 | >>> restricted_loads(b"cos\nsystem\n(S'echo hello world'\ntR.") |
| 840 | Traceback (most recent call last): |
| 841 | ... |
| 842 | pickle.UnpicklingError: global 'os.system' is forbidden |
| 843 | >>> restricted_loads(b'cbuiltins\neval\n' |
| 844 | ... b'(S\'getattr(__import__("os"), "system")' |
| 845 | ... b'("echo hello world")\'\ntR.') |
| 846 | Traceback (most recent call last): |
| 847 | ... |
| 848 | pickle.UnpicklingError: global 'builtins.eval' is forbidden |
| 849 | |
Alexandre Vassalotti | 73b90a8 | 2008-10-29 23:32:33 +0000 | [diff] [blame] | 850 | |
| 851 | .. XXX Add note about how extension codes could evade our protection |
Georg Brandl | 48310cd | 2009-01-03 21:18:54 +0000 | [diff] [blame] | 852 | mechanism (e.g. cached classes do not invokes find_class()). |
Alexandre Vassalotti | 73b90a8 | 2008-10-29 23:32:33 +0000 | [diff] [blame] | 853 | |
| 854 | As our examples shows, you have to be careful with what you allow to be |
| 855 | unpickled. Therefore if security is a concern, you may want to consider |
Alexandre Vassalotti | 9d7665d | 2009-04-03 06:13:29 +0000 | [diff] [blame] | 856 | alternatives such as the marshalling API in :mod:`xmlrpc.client` or |
| 857 | third-party solutions. |
Alexandre Vassalotti | 73b90a8 | 2008-10-29 23:32:33 +0000 | [diff] [blame] | 858 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 859 | |
Antoine Pitrou | d4d6055 | 2013-12-07 00:56:59 +0100 | [diff] [blame] | 860 | Performance |
| 861 | ----------- |
| 862 | |
| 863 | Recent versions of the pickle protocol (from protocol 2 and upwards) feature |
| 864 | efficient binary encodings for several common features and built-in types. |
| 865 | Also, the :mod:`pickle` module has a transparent optimizer written in C. |
| 866 | |
| 867 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 868 | .. _pickle-example: |
| 869 | |
Alexandre Vassalotti | 9d7665d | 2009-04-03 06:13:29 +0000 | [diff] [blame] | 870 | Examples |
| 871 | -------- |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 872 | |
Alexandre Vassalotti | 9d7665d | 2009-04-03 06:13:29 +0000 | [diff] [blame] | 873 | For the simplest code, use the :func:`dump` and :func:`load` functions. :: |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 874 | |
| 875 | import pickle |
| 876 | |
Alexandre Vassalotti | bcd1e3a | 2009-01-23 05:28:16 +0000 | [diff] [blame] | 877 | # An arbitrary collection of objects supported by pickle. |
| 878 | data = { |
Alexandre Vassalotti | 9d7665d | 2009-04-03 06:13:29 +0000 | [diff] [blame] | 879 | 'a': [1, 2.0, 3, 4+6j], |
| 880 | 'b': ("character string", b"byte string"), |
Raymond Hettinger | df1b699 | 2014-11-09 15:56:33 -0800 | [diff] [blame] | 881 | 'c': {None, True, False} |
Alexandre Vassalotti | bcd1e3a | 2009-01-23 05:28:16 +0000 | [diff] [blame] | 882 | } |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 883 | |
Alexandre Vassalotti | bcd1e3a | 2009-01-23 05:28:16 +0000 | [diff] [blame] | 884 | with open('data.pickle', 'wb') as f: |
| 885 | # Pickle the 'data' dictionary using the highest protocol available. |
| 886 | pickle.dump(data, f, pickle.HIGHEST_PROTOCOL) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 887 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 888 | |
Alexandre Vassalotti | bcd1e3a | 2009-01-23 05:28:16 +0000 | [diff] [blame] | 889 | The following example reads the resulting pickled data. :: |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 890 | |
Alexandre Vassalotti | bcd1e3a | 2009-01-23 05:28:16 +0000 | [diff] [blame] | 891 | import pickle |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 892 | |
Alexandre Vassalotti | bcd1e3a | 2009-01-23 05:28:16 +0000 | [diff] [blame] | 893 | with open('data.pickle', 'rb') as f: |
| 894 | # The protocol version used is detected automatically, so we do not |
| 895 | # have to specify it. |
| 896 | data = pickle.load(f) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 897 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 898 | |
Alexandre Vassalotti | 9d7665d | 2009-04-03 06:13:29 +0000 | [diff] [blame] | 899 | .. XXX: Add examples showing how to optimize pickles for size (like using |
| 900 | .. pickletools.optimize() or the gzip module). |
| 901 | |
| 902 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 903 | .. seealso:: |
| 904 | |
Alexandre Vassalotti | f7fa63d | 2008-05-11 08:55:36 +0000 | [diff] [blame] | 905 | Module :mod:`copyreg` |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 906 | Pickle interface constructor registration for extension types. |
| 907 | |
Alexandre Vassalotti | 9d7665d | 2009-04-03 06:13:29 +0000 | [diff] [blame] | 908 | Module :mod:`pickletools` |
| 909 | Tools for working with and analyzing pickled data. |
| 910 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 911 | Module :mod:`shelve` |
| 912 | Indexed databases of objects; uses :mod:`pickle`. |
| 913 | |
| 914 | Module :mod:`copy` |
| 915 | Shallow and deep object copying. |
| 916 | |
| 917 | Module :mod:`marshal` |
| 918 | High-performance serialization of built-in types. |
| 919 | |
| 920 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 921 | .. rubric:: Footnotes |
| 922 | |
| 923 | .. [#] Don't confuse this with the :mod:`marshal` module |
| 924 | |
Ethan Furman | 2498d9e | 2013-10-18 00:45:40 -0700 | [diff] [blame] | 925 | .. [#] This is why :keyword:`lambda` functions cannot be pickled: all |
| 926 | :keyword:`lambda` functions share the same name: ``<lambda>``. |
| 927 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 928 | .. [#] The exception raised will likely be an :exc:`ImportError` or an |
| 929 | :exc:`AttributeError` but it could be something else. |
| 930 | |
Alexandre Vassalotti | 73b90a8 | 2008-10-29 23:32:33 +0000 | [diff] [blame] | 931 | .. [#] The :mod:`copy` module uses this protocol for shallow and deep copying |
| 932 | operations. |
Alexandre Vassalotti | 5f3b63a | 2008-10-18 20:47:58 +0000 | [diff] [blame] | 933 | |
Alexandre Vassalotti | d039286 | 2008-10-24 01:32:40 +0000 | [diff] [blame] | 934 | .. [#] The limitation on alphanumeric characters is due to the fact |
| 935 | the persistent IDs, in protocol 0, are delimited by the newline |
| 936 | character. Therefore if any kind of newline characters occurs in |
Alexandre Vassalotti | 5f3b63a | 2008-10-18 20:47:58 +0000 | [diff] [blame] | 937 | persistent IDs, the resulting pickle will become unreadable. |