| \section{\module{pickle} --- |
| Python object serialization} |
| |
| \declaremodule{standard}{pickle} |
| \modulesynopsis{Convert Python objects to streams of bytes and back.} |
| % Substantial improvements by Jim Kerr <jbkerr@sr.hp.com>. |
| |
| \index{persistence} |
| \indexii{persistent}{objects} |
| \indexii{serializing}{objects} |
| \indexii{marshalling}{objects} |
| \indexii{flattening}{objects} |
| \indexii{pickling}{objects} |
| |
| |
| The \module{pickle} module implements a basic but powerful algorithm |
| for ``pickling'' (a.k.a.\ serializing, marshalling or flattening) |
| nearly arbitrary Python objects. This is the act of converting |
| objects to a stream of bytes (and back: ``unpickling''). This is a |
| more primitive notion than persistence --- although \module{pickle} |
| reads and writes file objects, it does not handle the issue of naming |
| persistent objects, nor the (even more complicated) area of concurrent |
| access to persistent objects. The \module{pickle} module can |
| transform a complex object into a byte stream and it can transform the |
| byte stream into an object with the same internal structure. The most |
| obvious thing to do with these byte streams is to write them onto a |
| file, but it is also conceivable to send them across a network or |
| store them in a database. The module |
| \refmodule{shelve}\refstmodindex{shelve} provides a simple interface |
| to pickle and unpickle objects on DBM-style database files. |
| |
| |
| \strong{Note:} The \module{pickle} module is rather slow. A |
| reimplementation of the same algorithm in C, which is up to 1000 times |
| faster, is available as the |
| \refmodule{cPickle}\refbimodindex{cPickle} module. This has the same |
| interface except that \class{Pickler} and \class{Unpickler} are |
| factory functions, not classes (so they cannot be used as base classes |
| for inheritance). |
| |
| Although the \module{pickle} module can use the built-in module |
| \refmodule{marshal}\refbimodindex{marshal} internally, it differs from |
| \refmodule{marshal} in the way it handles certain kinds of data: |
| |
| \begin{itemize} |
| |
| \item Recursive objects (objects containing references to themselves): |
| \module{pickle} keeps track of the objects it has already |
| serialized, so later references to the same object won't be |
| serialized again. (The \refmodule{marshal} module breaks for |
| this.) |
| |
| \item Object sharing (references to the same object in different |
| places): This is similar to self-referencing objects; |
| \module{pickle} stores the object once, and ensures that all |
| other references point to the master copy. Shared objects |
| remain shared, which can be very important for mutable objects. |
| |
| \item User-defined classes and their instances: \refmodule{marshal} |
| does not support these at all, but \module{pickle} can save |
| and restore class instances transparently. The class definition |
| must be importable and live in the same module as when the |
| object was stored. |
| |
| \end{itemize} |
| |
| The data format used by \module{pickle} is Python-specific. This has |
| the advantage that there are no restrictions imposed by external |
| standards such as |
| XDR\index{XDR}\index{External Data Representation} (which can't |
| represent pointer sharing); however it means that non-Python programs |
| may not be able to reconstruct pickled Python objects. |
| |
| By default, the \module{pickle} data format uses a printable \ASCII{} |
| representation. This is slightly more voluminous than a binary |
| representation. The big advantage of using printable \ASCII{} (and of |
| some other characteristics of \module{pickle}'s representation) is that |
| for debugging or recovery purposes it is possible for a human to read |
| the pickled file with a standard text editor. |
| |
| A binary format, which is slightly more efficient, can be chosen by |
| specifying a nonzero (true) value for the \var{bin} argument to the |
| \class{Pickler} constructor or the \function{dump()} and \function{dumps()} |
| functions. The binary format is not the default because of backwards |
| compatibility with the Python 1.4 pickle module. In a future version, |
| the default may change to binary. |
| |
| The \module{pickle} module doesn't handle code objects, which the |
| \refmodule{marshal}\refbimodindex{marshal} module does. I suppose |
| \module{pickle} could, and maybe it should, but there's probably no |
| great need for it right now (as long as \refmodule{marshal} continues |
| to be used for reading and writing code objects), and at least this |
| avoids the possibility of smuggling Trojan horses into a program. |
| |
| For the benefit of persistence modules written using \module{pickle}, it |
| supports the notion of a reference to an object outside the pickled |
| data stream. Such objects are referenced by a name, which is an |
| arbitrary string of printable \ASCII{} characters. The resolution of |
| such names is not defined by the \module{pickle} module --- the |
| persistent object module will have to implement a method |
| \method{persistent_load()}. To write references to persistent objects, |
| the persistent module must define a method \method{persistent_id()} which |
| returns either \code{None} or the persistent ID of the object. |
| |
| There are some restrictions on the pickling of class instances. |
| |
| First of all, the class must be defined at the top level in a module. |
| Furthermore, all its instance variables must be picklable. |
| |
| \setindexsubitem{(pickle protocol)} |
| |
| When a pickled class instance is unpickled, its \method{__init__()} method |
| is normally \emph{not} invoked. \strong{Note:} This is a deviation |
| from previous versions of this module; the change was introduced in |
| Python 1.5b2. The reason for the change is that in many cases it is |
| desirable to have a constructor that requires arguments; it is a |
| (minor) nuisance to have to provide a \method{__getinitargs__()} method. |
| |
| If it is desirable that the \method{__init__()} method be called on |
| unpickling, a class can define a method \method{__getinitargs__()}, |
| which should return a \emph{tuple} containing the arguments to be |
| passed to the class constructor (\method{__init__()}). This method is |
| called at pickle time; the tuple it returns is incorporated in the |
| pickle for the instance. |
| \withsubitem{(copy protocol)}{\ttindex{__getinitargs__()}} |
| \withsubitem{(instance constructor)}{\ttindex{__init__()}} |
| |
| Classes can further influence how their instances are pickled --- if |
| the class |
| \withsubitem{(copy protocol)}{ |
| \ttindex{__getstate__()}\ttindex{__setstate__()}} |
| \withsubitem{(instance attribute)}{ |
| \ttindex{__dict__}} |
| defines the method \method{__getstate__()}, it is called and the return |
| state is pickled as the contents for the instance, and if the class |
| defines the method \method{__setstate__()}, it is called with the |
| unpickled state. (Note that these methods can also be used to |
| implement copying class instances.) If there is no |
| \method{__getstate__()} method, the instance's \member{__dict__} is |
| pickled. If there is no \method{__setstate__()} method, the pickled |
| object must be a dictionary and its items are assigned to the new |
| instance's dictionary. (If a class defines both \method{__getstate__()} |
| and \method{__setstate__()}, the state object needn't be a dictionary |
| --- these methods can do what they want.) This protocol is also used |
| by the shallow and deep copying operations defined in the |
| \refmodule{copy}\refstmodindex{copy} module. |
| |
| Note that when class instances are pickled, their class's code and |
| data are not pickled along with them. Only the instance data are |
| pickled. This is done on purpose, so you can fix bugs in a class or |
| add methods and still load objects that were created with an earlier |
| version of the class. If you plan to have long-lived objects that |
| will see many versions of a class, it may be worthwhile to put a version |
| number in the objects so that suitable conversions can be made by the |
| class's \method{__setstate__()} method. |
| |
| When a class itself is pickled, only its name is pickled --- the class |
| definition is not pickled, but re-imported by the unpickling process. |
| Therefore, the restriction that the class must be defined at the top |
| level in a module applies to pickled classes as well. |
| |
| \setindexsubitem{(in module pickle)} |
| |
| The interface can be summarized as follows. |
| |
| To pickle an object \code{x} onto a file \code{f}, open for writing: |
| |
| \begin{verbatim} |
| p = pickle.Pickler(f) |
| p.dump(x) |
| \end{verbatim} |
| |
| A shorthand for this is: |
| |
| \begin{verbatim} |
| pickle.dump(x, f) |
| \end{verbatim} |
| |
| To unpickle an object \code{x} from a file \code{f}, open for reading: |
| |
| \begin{verbatim} |
| u = pickle.Unpickler(f) |
| x = u.load() |
| \end{verbatim} |
| |
| A shorthand is: |
| |
| \begin{verbatim} |
| x = pickle.load(f) |
| \end{verbatim} |
| |
| The \class{Pickler} class only calls the method \code{f.write()} with a |
| \withsubitem{(class in pickle)}{\ttindex{Unpickler}\ttindex{Pickler}} |
| string argument. The \class{Unpickler} calls the methods \code{f.read()} |
| (with an integer argument) and \code{f.readline()} (without argument), |
| both returning a string. It is explicitly allowed to pass non-file |
| objects here, as long as they have the right methods. |
| |
| The constructor for the \class{Pickler} class has an optional second |
| argument, \var{bin}. If this is present and true, the binary |
| pickle format is used; if it is absent or false, the (less efficient, |
| but backwards compatible) text pickle format is used. The |
| \class{Unpickler} class does not have an argument to distinguish |
| between binary and text pickle formats; it accepts either format. |
| |
| The following types can be pickled: |
| |
| \begin{itemize} |
| |
| \item \code{None} |
| |
| \item integers, long integers, floating point numbers |
| |
| \item normal and Unicode strings |
| |
| \item tuples, lists and dictionaries containing only picklable objects |
| |
| \item functions defined at the top level of a module (by name |
| reference, not storage of the implementation) |
| |
| \item built-in functions |
| |
| \item classes that are defined at the top level in a module |
| |
| \item instances of such classes whose \member{__dict__} or |
| \method{__setstate__()} is picklable |
| |
| \end{itemize} |
| |
| Attempts to pickle unpicklable objects will raise the |
| \exception{PicklingError} exception; when this happens, an unspecified |
| number of bytes may have been written to the file. |
| |
| It is possible to make multiple calls to the \method{dump()} method of |
| the same \class{Pickler} instance. These must then be matched to the |
| same number of calls to the \method{load()} method of the |
| corresponding \class{Unpickler} instance. If the same object is |
| pickled by multiple \method{dump()} calls, the \method{load()} will all |
| yield references to the same object. \emph{Warning}: this is intended |
| for pickling multiple objects without intervening modifications to the |
| objects or their parts. If you modify an object and then pickle it |
| again using the same \class{Pickler} instance, the object is not |
| pickled again --- a reference to it is pickled and the |
| \class{Unpickler} will return the old value, not the modified one. |
| (There are two problems here: (a) detecting changes, and (b) |
| marshalling a minimal set of changes. I have no answers. Garbage |
| Collection may also become a problem here.) |
| |
| Apart from the \class{Pickler} and \class{Unpickler} classes, the |
| module defines the following functions, and an exception: |
| |
| \begin{funcdesc}{dump}{object, file\optional{, bin}} |
| Write a pickled representation of \var{object} to the open file object |
| \var{file}. This is equivalent to |
| \samp{Pickler(\var{file}, \var{bin}).dump(\var{object})}. |
| If the optional \var{bin} argument is present and nonzero, the binary |
| pickle format is used; if it is zero or absent, the (less efficient) |
| text pickle format is used. |
| \end{funcdesc} |
| |
| \begin{funcdesc}{load}{file} |
| Read a pickled object from the open file object \var{file}. This is |
| equivalent to \samp{Unpickler(\var{file}).load()}. |
| \end{funcdesc} |
| |
| \begin{funcdesc}{dumps}{object\optional{, bin}} |
| Return the pickled representation of the object as a string, instead |
| of writing it to a file. If the optional \var{bin} argument is |
| present and nonzero, the binary pickle format is used; if it is zero |
| or absent, the (less efficient) text pickle format is used. |
| \end{funcdesc} |
| |
| \begin{funcdesc}{loads}{string} |
| Read a pickled object from a string instead of a file. Characters in |
| the string past the pickled object's representation are ignored. |
| \end{funcdesc} |
| |
| \begin{excdesc}{PicklingError} |
| This exception is raised when an unpicklable object is passed to |
| \method{Pickler.dump()}. |
| \end{excdesc} |
| |
| |
| \begin{seealso} |
| \seemodule[copyreg]{copy_reg}{pickle interface constructor |
| registration} |
| |
| \seemodule{shelve}{indexed databases of objects; uses \module{pickle}} |
| |
| \seemodule{copy}{shallow and deep object copying} |
| |
| \seemodule{marshal}{high-performance serialization of built-in types} |
| \end{seealso} |
| |
| |
| \subsection{Example \label{pickle-example}} |
| |
| Here's a simple example of how to modify pickling behavior for a |
| class. The \class{TextReader} class opens a text file, and returns |
| the line number and line contents each time its \method{readline()} |
| method is called. If a \class{TextReader} instance is pickled, all |
| attributes \emph{except} the file object member are saved. When the |
| instance is unpickled, the file is reopened, and reading resumes from |
| the last location. The \method{__setstate__()} and |
| \method{__getstate__()} methods are used to implement this behavior. |
| |
| \begin{verbatim} |
| # illustrate __setstate__ and __getstate__ methods |
| # used in pickling. |
| |
| class TextReader: |
| "Print and number lines in a text file." |
| def __init__(self,file): |
| self.file = file |
| self.fh = open(file,'r') |
| self.lineno = 0 |
| |
| def readline(self): |
| self.lineno = self.lineno + 1 |
| line = self.fh.readline() |
| if not line: |
| return None |
| return "%d: %s" % (self.lineno,line[:-1]) |
| |
| # return data representation for pickled object |
| def __getstate__(self): |
| odict = self.__dict__ # get attribute dictionary |
| del odict['fh'] # remove filehandle entry |
| return odict |
| |
| # restore object state from data representation generated |
| # by __getstate__ |
| def __setstate__(self,dict): |
| fh = open(dict['file']) # reopen file |
| count = dict['lineno'] # read from file... |
| while count: # until line count is restored |
| fh.readline() |
| count = count - 1 |
| dict['fh'] = fh # create filehandle entry |
| self.__dict__ = dict # make dict our attribute dictionary |
| \end{verbatim} |
| |
| A sample usage might be something like this: |
| |
| \begin{verbatim} |
| >>> import TextReader |
| >>> obj = TextReader.TextReader("TextReader.py") |
| >>> obj.readline() |
| '1: #!/usr/local/bin/python' |
| >>> # (more invocations of obj.readline() here) |
| ... obj.readline() |
| '7: class TextReader:' |
| >>> import pickle |
| >>> pickle.dump(obj,open('save.p','w')) |
| |
| (start another Python session) |
| |
| >>> import pickle |
| >>> reader = pickle.load(open('save.p')) |
| >>> reader.readline() |
| '8: "Print and number lines in a text file."' |
| \end{verbatim} |
| |
| |
| \section{\module{cPickle} --- |
| Alternate implementation of \module{pickle}} |
| |
| \declaremodule{builtin}{cPickle} |
| \modulesynopsis{Faster version of \refmodule{pickle}, but not subclassable.} |
| \moduleauthor{Jim Fulton}{jfulton@digicool.com} |
| \sectionauthor{Fred L. Drake, Jr.}{fdrake@acm.org} |
| |
| |
| The \module{cPickle} module provides a similar interface and identical |
| functionality as the \refmodule{pickle}\refstmodindex{pickle} module, |
| but can be up to 1000 times faster since it is implemented in C. The |
| only other important difference to note is that \function{Pickler()} |
| and \function{Unpickler()} are functions and not classes, and so |
| cannot be subclassed. This should not be an issue in most cases. |
| |
| The format of the pickle data is identical to that produced using the |
| \refmodule{pickle} module, so it is possible to use \refmodule{pickle} and |
| \module{cPickle} interchangeably with existing pickles. |
| |
| (Since the pickle data format is actually a tiny stack-oriented |
| programming language, and there are some freedoms in the encodings of |
| certain objects, it's possible that the two modules produce different |
| pickled data for the same input objects; however they will always be |
| able to read each other's pickles back in.) |