Doc/lib/libpickle.tex

\section{\module{pickle} ---
         Python object serialization}

\declaremodule{standard}{pickle}
\modulesynopsis{Convert Python objects to streams of bytes and back.}
% Substantial improvements by Jim Kerr <jbkerr@sr.hp.com>.

\index{persistence}
\indexii{persistent}{objects}
\indexii{serializing}{objects}
\indexii{marshalling}{objects}
\indexii{flattening}{objects}
\indexii{pickling}{objects}


The \module{pickle} module implements a basic but powerful algorithm
for ``pickling'' (a.k.a.\ serializing, marshalling or flattening)
nearly arbitrary Python objects.  This is the act of converting
objects to a stream of bytes (and back: ``unpickling'').  This is a
more primitive notion than persistence --- although \module{pickle}
reads and writes file objects, it does not handle the issue of naming
persistent objects, nor the (even more complicated) area of concurrent
access to persistent objects.  The \module{pickle} module can
transform a complex object into a byte stream and it can transform the
byte stream into an object with the same internal structure.  The most
obvious thing to do with these byte streams is to write them onto a
file, but it is also conceivable to send them across a network or
store them in a database.  The module
\refmodule{shelve}\refstmodindex{shelve} provides a simple interface
to pickle and unpickle objects on DBM-style database files.


\strong{Note:} The \module{pickle} module is rather slow.  A
reimplementation of the same algorithm in C, which is up to 1000 times
faster, is available as the
\refmodule{cPickle}\refbimodindex{cPickle} module.  This has the same
interface except that \class{Pickler} and \class{Unpickler} are
factory functions, not classes (so they cannot be used as base classes
for inheritance).

Although the \module{pickle} module can use the built-in module
\refmodule{marshal}\refbimodindex{marshal} internally, it differs from 
\refmodule{marshal} in the way it handles certain kinds of data:

\begin{itemize}

\item Recursive objects (objects containing references to themselves): 
      \module{pickle} keeps track of the objects it has already
      serialized, so later references to the same object won't be
      serialized again.  (The \refmodule{marshal} module breaks for
      this.)

\item Object sharing (references to the same object in different
      places):  This is similar to self-referencing objects;
      \module{pickle} stores the object once, and ensures that all
      other references point to the master copy.  Shared objects
      remain shared, which can be very important for mutable objects.

\item User-defined classes and their instances:  \refmodule{marshal}
      does not support these at all, but \module{pickle} can save
      and restore class instances transparently.  The class definition 
      must be importable and live in the same module as when the
      object was stored.

\end{itemize}

The data format used by \module{pickle} is Python-specific.  This has
the advantage that there are no restrictions imposed by external
standards such as
XDR\index{XDR}\index{External Data Representation} (which can't
represent pointer sharing); however it means that non-Python programs
may not be able to reconstruct pickled Python objects.

By default, the \module{pickle} data format uses a printable \ASCII{}
representation.  This is slightly more voluminous than a binary
representation.  The big advantage of using printable \ASCII{} (and of
some other characteristics of \module{pickle}'s representation) is that
for debugging or recovery purposes it is possible for a human to read
the pickled file with a standard text editor.

A binary format, which is slightly more efficient, can be chosen by
specifying a nonzero (true) value for the \var{bin} argument to the
\class{Pickler} constructor or the \function{dump()} and \function{dumps()}
functions.  The binary format is not the default because of backwards
compatibility with the Python 1.4 pickle module.  In a future version,
the default may change to binary.

The \module{pickle} module doesn't handle code objects, which the
\refmodule{marshal}\refbimodindex{marshal} module does.  I suppose
\module{pickle} could, and maybe it should, but there's probably no
great need for it right now (as long as \refmodule{marshal} continues
to be used for reading and writing code objects), and at least this
avoids the possibility of smuggling Trojan horses into a program.

For the benefit of persistence modules written using \module{pickle}, it
supports the notion of a reference to an object outside the pickled
data stream.  Such objects are referenced by a name, which is an
arbitrary string of printable \ASCII{} characters.  The resolution of
such names is not defined by the \module{pickle} module --- the
persistent object module will have to implement a method
\method{persistent_load()}.  To write references to persistent objects,
the persistent module must define a method \method{persistent_id()} which
returns either \code{None} or the persistent ID of the object.

There are some restrictions on the pickling of class instances.

First of all, the class must be defined at the top level in a module.
Furthermore, all its instance variables must be picklable.

\setindexsubitem{(pickle protocol)}

When a pickled class instance is unpickled, its \method{__init__()} method
is normally \emph{not} invoked.  \strong{Note:} This is a deviation
from previous versions of this module; the change was introduced in
Python 1.5b2.  The reason for the change is that in many cases it is
desirable to have a constructor that requires arguments; it is a
(minor) nuisance to have to provide a \method{__getinitargs__()} method.

If it is desirable that the \method{__init__()} method be called on
unpickling, a class can define a method \method{__getinitargs__()},
which should return a \emph{tuple} containing the arguments to be
passed to the class constructor (\method{__init__()}).  This method is
called at pickle time; the tuple it returns is incorporated in the
pickle for the instance.
\withsubitem{(copy protocol)}{\ttindex{__getinitargs__()}}
\withsubitem{(instance constructor)}{\ttindex{__init__()}}

Classes can further influence how their instances are pickled --- if
the class
\withsubitem{(copy protocol)}{
  \ttindex{__getstate__()}\ttindex{__setstate__()}}
\withsubitem{(instance attribute)}{
  \ttindex{__dict__}}
defines the method \method{__getstate__()}, it is called and the return
state is pickled as the contents for the instance, and if the class
defines the method \method{__setstate__()}, it is called with the
unpickled state.  (Note that these methods can also be used to
implement copying class instances.)  If there is no
\method{__getstate__()} method, the instance's \member{__dict__} is
pickled.  If there is no \method{__setstate__()} method, the pickled
object must be a dictionary and its items are assigned to the new
instance's dictionary.  (If a class defines both \method{__getstate__()}
and \method{__setstate__()}, the state object needn't be a dictionary
--- these methods can do what they want.)  This protocol is also used
by the shallow and deep copying operations defined in the
\refmodule{copy}\refstmodindex{copy} module.

Note that when class instances are pickled, their class's code and
data are not pickled along with them.  Only the instance data are
pickled.  This is done on purpose, so you can fix bugs in a class or
add methods and still load objects that were created with an earlier
version of the class.  If you plan to have long-lived objects that
will see many versions of a class, it may be worthwhile to put a version
number in the objects so that suitable conversions can be made by the
class's \method{__setstate__()} method.

When a class itself is pickled, only its name is pickled --- the class
definition is not pickled, but re-imported by the unpickling process.
Therefore, the restriction that the class must be defined at the top
level in a module applies to pickled classes as well.

\setindexsubitem{(in module pickle)}

The interface can be summarized as follows.

To pickle an object \code{x} onto a file \code{f}, open for writing:

\begin{verbatim}
p = pickle.Pickler(f)
p.dump(x)
\end{verbatim}

A shorthand for this is:

\begin{verbatim}
pickle.dump(x, f)
\end{verbatim}

To unpickle an object \code{x} from a file \code{f}, open for reading:

\begin{verbatim}
u = pickle.Unpickler(f)
x = u.load()
\end{verbatim}

A shorthand is:

\begin{verbatim}
x = pickle.load(f)
\end{verbatim}

The \class{Pickler} class only calls the method \code{f.write()} with a
\withsubitem{(class in pickle)}{\ttindex{Unpickler}\ttindex{Pickler}}
string argument.  The \class{Unpickler} calls the methods \code{f.read()}
(with an integer argument) and \code{f.readline()} (without argument),
both returning a string.  It is explicitly allowed to pass non-file
objects here, as long as they have the right methods.

The constructor for the \class{Pickler} class has an optional second
argument, \var{bin}.  If this is present and true, the binary
pickle format is used; if it is absent or false, the (less efficient,
but backwards compatible) text pickle format is used.  The
\class{Unpickler} class does not have an argument to distinguish
between binary and text pickle formats; it accepts either format.

The following types can be pickled:

\begin{itemize}

\item \code{None}

\item integers, long integers, floating point numbers

\item normal and Unicode strings

\item tuples, lists and dictionaries containing only picklable objects

\item functions defined at the top level of a module (by name
      reference, not storage of the implementation)

\item built-in functions

\item classes that are defined at the top level in a module

\item instances of such classes whose \member{__dict__} or
\method{__setstate__()} is picklable

\end{itemize}

Attempts to pickle unpicklable objects will raise the
\exception{PicklingError} exception; when this happens, an unspecified
number of bytes may have been written to the file.

It is possible to make multiple calls to the \method{dump()} method of
the same \class{Pickler} instance.  These must then be matched to the
same number of calls to the \method{load()} method of the
corresponding \class{Unpickler} instance.  If the same object is
pickled by multiple \method{dump()} calls, the \method{load()} will all
yield references to the same object.  \emph{Warning}: this is intended
for pickling multiple objects without intervening modifications to the
objects or their parts.  If you modify an object and then pickle it
again using the same \class{Pickler} instance, the object is not
pickled again --- a reference to it is pickled and the
\class{Unpickler} will return the old value, not the modified one.
(There are two problems here: (a) detecting changes, and (b)
marshalling a minimal set of changes.  I have no answers.  Garbage
Collection may also become a problem here.)

Apart from the \class{Pickler} and \class{Unpickler} classes, the
module defines the following functions, and an exception:

\begin{funcdesc}{dump}{object, file\optional{, bin}}
Write a pickled representation of \var{object} to the open file object
\var{file}.  This is equivalent to
\samp{Pickler(\var{file}, \var{bin}).dump(\var{object})}.
If the optional \var{bin} argument is present and nonzero, the binary
pickle format is used; if it is zero or absent, the (less efficient)
text pickle format is used.
\end{funcdesc}

\begin{funcdesc}{load}{file}
Read a pickled object from the open file object \var{file}.  This is
equivalent to \samp{Unpickler(\var{file}).load()}.
\end{funcdesc}

\begin{funcdesc}{dumps}{object\optional{, bin}}
Return the pickled representation of the object as a string, instead
of writing it to a file.  If the optional \var{bin} argument is
present and nonzero, the binary pickle format is used; if it is zero
or absent, the (less efficient) text pickle format is used.
\end{funcdesc}

\begin{funcdesc}{loads}{string}
Read a pickled object from a string instead of a file.  Characters in
the string past the pickled object's representation are ignored.
\end{funcdesc}

\begin{excdesc}{PicklingError}
This exception is raised when an unpicklable object is passed to
\method{Pickler.dump()}.
\end{excdesc}


\begin{seealso}
  \seemodule[copyreg]{copy_reg}{Pickle interface constructor
                                registration for extension types.}

  \seemodule{shelve}{Indexed databases of objects; uses \module{pickle}.}

  \seemodule{copy}{Shallow and deep object copying.}

  \seemodule{marshal}{High-performance serialization of built-in types.}
\end{seealso}


\subsection{Example \label{pickle-example}}

Here's a simple example of how to modify pickling behavior for a
class.  The \class{TextReader} class opens a text file, and returns
the line number and line contents each time its \method{readline()}
method is called. If a \class{TextReader} instance is pickled, all
attributes \emph{except} the file object member are saved. When the
instance is unpickled, the file is reopened, and reading resumes from
the last location. The \method{__setstate__()} and
\method{__getstate__()} methods are used to implement this behavior.

\begin{verbatim}
# illustrate __setstate__ and __getstate__  methods
# used in pickling.

class TextReader:
    "Print and number lines in a text file."
    def __init__(self,file):
        self.file = file
        self.fh = open(file,'r')
        self.lineno = 0

    def readline(self):
        self.lineno = self.lineno + 1
        line = self.fh.readline()
        if not line:
            return None
        return "%d: %s" % (self.lineno,line[:-1])

    # return data representation for pickled object
    def __getstate__(self):
        odict = self.__dict__    # get attribute dictionary
        del odict['fh']          # remove filehandle entry
        return odict

    # restore object state from data representation generated 
    # by __getstate__
    def __setstate__(self,dict):
        fh = open(dict['file'])  # reopen file
        count = dict['lineno']   # read from file...
        while count:             # until line count is restored
            fh.readline()
            count = count - 1
        dict['fh'] = fh          # create filehandle entry
        self.__dict__ = dict     # make dict our attribute dictionary
\end{verbatim}

A sample usage might be something like this:

\begin{verbatim}
>>> import TextReader
>>> obj = TextReader.TextReader("TextReader.py")
>>> obj.readline()
'1: #!/usr/local/bin/python'
>>> # (more invocations of obj.readline() here)
... obj.readline()
'7: class TextReader:'
>>> import pickle
>>> pickle.dump(obj,open('save.p','w'))

  (start another Python session)

>>> import pickle
>>> reader = pickle.load(open('save.p'))
>>> reader.readline()
'8:     "Print and number lines in a text file."'
\end{verbatim}


\section{\module{cPickle} ---
         Alternate implementation of \module{pickle}}

\declaremodule{builtin}{cPickle}
\modulesynopsis{Faster version of \refmodule{pickle}, but not subclassable.}
\moduleauthor{Jim Fulton}{jfulton@digicool.com}
\sectionauthor{Fred L. Drake, Jr.}{fdrake@acm.org}


The \module{cPickle} module provides a similar interface and identical
functionality as the \refmodule{pickle}\refstmodindex{pickle} module,
but can be up to 1000 times faster since it is implemented in C.  The
only other important difference to note is that \function{Pickler()}
and \function{Unpickler()} are functions and not classes, and so
cannot be subclassed.  This should not be an issue in most cases.

The format of the pickle data is identical to that produced using the
\refmodule{pickle} module, so it is possible to use \refmodule{pickle} and
\module{cPickle} interchangeably with existing pickles.

(Since the pickle data format is actually a tiny stack-oriented
programming language, and there are some freedoms in the encodings of
certain objects, it's possible that the two modules produce different
pickled data for the same input objects; however they will always be
able to read each other's pickles back in.)