Doc/lib/libweakref.tex

\section{\module{weakref} ---
         Weak references}

\declaremodule{extension}{weakref}
\modulesynopsis{Support for weak references and weak dictionaries.}
\moduleauthor{Fred L. Drake, Jr.}{fdrake@acm.org}
\moduleauthor{Neil Schemenauer}{nas@arctrix.com}
\moduleauthor{Martin von L\"owis}{martin@loewis.home.cs.tu-berlin.de}
\sectionauthor{Fred L. Drake, Jr.}{fdrake@acm.org}

\versionadded{2.1}

% When making changes to the examples in this file, be sure to update
% Lib/test/test_weakref.py::libreftest too!

The \module{weakref} module allows the Python programmer to create
\dfn{weak references} to objects.

In the following, the term \dfn{referent} means the
object which is referred to by a weak reference.

A weak reference to an object is not enough to keep the object alive:
when the only remaining references to a referent are weak references,
garbage collection is free to destroy the referent and reuse its memory
for something else.  A primary use for weak references is to implement
caches or mappings holding large objects, where it's desired that a
large object not be kept alive solely because it appears in a cache or
mapping.  For example, if you have a number of large binary image objects,
you may wish to associate a name with each.  If you used a Python
dictionary to map names to images, or images to names, the image objects
would remain alive just because they appeared as values or keys in the
dictionaries.  The \class{WeakKeyDictionary} and
\class{WeakValueDictionary} classes supplied by the \module{weakref}
module are an alternative, using weak references to construct mappings
that don't keep objects alive solely because they appear in the mapping
objects.  If, for example, an image object is a value in a
\class{WeakValueDictionary}, then when the last remaining
references to that image object are the weak references held by weak
mappings, garbage collection can reclaim the object, and its corresponding
entries in weak mappings are simply deleted.

\class{WeakKeyDictionary} and \class{WeakValueDictionary} use weak
references in their implementation, setting up callback functions on
the weak references that notify the weak dictionaries when a key or value
has been reclaimed by garbage collection.  Most programs should find that
using one of these weak dictionary types is all they need -- it's
not usually necessary to create your own weak references directly.  The
low-level machinery used by the weak dictionary implementations is exposed
by the \module{weakref} module for the benefit of advanced uses.

Not all objects can be weakly referenced; those objects which can
include class instances, functions written in Python (but not in C),
methods (both bound and unbound), sets, frozensets, file objects,
generators, type objects, DBcursor objects from the \module{bsddb} module,
sockets, arrays, deques, and regular expression pattern objects.
\versionchanged[Added support for files, sockets, arrays, and patterns]{2.4}

Several builtin types such as \class{list} and \class{dict} do not
directly support weak references but can add support through subclassing:

\begin{verbatim}
class Dict(dict):
    pass

obj = Dict(red=1, green=2, blue=3)   # this object is weak referencable
\end{verbatim}

Extension types can easily be made to support weak references; see section
\ref{weakref-extension}, ``Weak References in Extension Types,'' for more
information.


\begin{classdesc}{ref}{object\optional{, callback}}
  Return a weak reference to \var{object}.  The original object can be
  retrieved by calling the reference object if the referent is still
  alive; if the referent is no longer alive, calling the reference
  object will cause \constant{None} to be returned.  If \var{callback} is
  provided and not \constant{None}, and the returned weakref object is
  still alive, the callback will be called when the object is about to be
  finalized; the weak reference object will be passed as the only
  parameter to the callback; the referent will no longer be available.

  It is allowable for many weak references to be constructed for the
  same object.  Callbacks registered for each weak reference will be
  called from the most recently registered callback to the oldest
  registered callback.

  Exceptions raised by the callback will be noted on the standard
  error output, but cannot be propagated; they are handled in exactly
  the same way as exceptions raised from an object's
  \method{__del__()} method.

  Weak references are hashable if the \var{object} is hashable.  They
  will maintain their hash value even after the \var{object} was
  deleted.  If \function{hash()} is called the first time only after
  the \var{object} was deleted, the call will raise
  \exception{TypeError}.

  Weak references support tests for equality, but not ordering.  If
  the referents are still alive, two references have the same
  equality relationship as their referents (regardless of the
  \var{callback}).  If either referent has been deleted, the
  references are equal only if the reference objects are the same
  object.

  \versionchanged[This is now a subclassable type rather than a
                  factory function; it derives from \class{object}]
                  {2.4}
\end{classdesc}

\begin{funcdesc}{proxy}{object\optional{, callback}}
  Return a proxy to \var{object} which uses a weak reference.  This
  supports use of the proxy in most contexts instead of requiring the
  explicit dereferencing used with weak reference objects.  The
  returned object will have a type of either \code{ProxyType} or
  \code{CallableProxyType}, depending on whether \var{object} is
  callable.  Proxy objects are not hashable regardless of the
  referent; this avoids a number of problems related to their
  fundamentally mutable nature, and prevent their use as dictionary
  keys.  \var{callback} is the same as the parameter of the same name
  to the \function{ref()} function.
\end{funcdesc}

\begin{funcdesc}{getweakrefcount}{object}
  Return the number of weak references and proxies which refer to
  \var{object}.
\end{funcdesc}

\begin{funcdesc}{getweakrefs}{object}
  Return a list of all weak reference and proxy objects which refer to
  \var{object}.
\end{funcdesc}

\begin{classdesc}{WeakKeyDictionary}{\optional{dict}}
  Mapping class that references keys weakly.  Entries in the
  dictionary will be discarded when there is no longer a strong
  reference to the key.  This can be used to associate additional data
  with an object owned by other parts of an application without adding
  attributes to those objects.  This can be especially useful with
  objects that override attribute accesses.

  \note{Caution:  Because a \class{WeakKeyDictionary} is built on top
        of a Python dictionary, it must not change size when iterating
        over it.  This can be difficult to ensure for a
        \class{WeakKeyDictionary} because actions performed by the
        program during iteration may cause items in the dictionary
        to vanish "by magic" (as a side effect of garbage collection).}
\end{classdesc}

\class{WeakKeyDictionary} objects have the following additional
methods.  These expose the internal references directly.  The
references are not guaranteed to be ``live'' at the time they are
used, so the result of calling the references needs to be checked
before being used.  This can be used to avoid creating references that
will cause the garbage collector to keep the keys around longer than
needed.

\begin{methoddesc}{iterkeyrefs}{}
  Return an iterator that yields the weak references to the keys.
  \versionadded{2.5}
\end{methoddesc}

\begin{methoddesc}{keyrefs}{}
  Return a list of weak references to the keys.
  \versionadded{2.5}
\end{methoddesc}

\begin{classdesc}{WeakValueDictionary}{\optional{dict}}
  Mapping class that references values weakly.  Entries in the
  dictionary will be discarded when no strong reference to the value
  exists any more.

  \note{Caution:  Because a \class{WeakValueDictionary} is built on top
        of a Python dictionary, it must not change size when iterating
        over it.  This can be difficult to ensure for a
        \class{WeakValueDictionary} because actions performed by the
        program during iteration may cause items in the dictionary
        to vanish "by magic" (as a side effect of garbage collection).}
\end{classdesc}

\class{WeakValueDictionary} objects have the following additional
methods.  These method have the same issues as the
\method{iterkeyrefs()} and \method{keyrefs()} methods of
\class{WeakKeyDictionary} objects.

\begin{methoddesc}{itervaluerefs}{}
  Return an iterator that yields the weak references to the values.
  \versionadded{2.5}
\end{methoddesc}

\begin{methoddesc}{valuerefs}{}
  Return a list of weak references to the values.
  \versionadded{2.5}
\end{methoddesc}

\begin{datadesc}{ReferenceType}
  The type object for weak references objects.
\end{datadesc}

\begin{datadesc}{ProxyType}
  The type object for proxies of objects which are not callable.
\end{datadesc}

\begin{datadesc}{CallableProxyType}
  The type object for proxies of callable objects.
\end{datadesc}

\begin{datadesc}{ProxyTypes}
  Sequence containing all the type objects for proxies.  This can make
  it simpler to test if an object is a proxy without being dependent
  on naming both proxy types.
\end{datadesc}

\begin{excdesc}{ReferenceError}
  Exception raised when a proxy object is used but the underlying
  object has been collected.  This is the same as the standard
  \exception{ReferenceError} exception.
\end{excdesc}


\begin{seealso}
  \seepep{0205}{Weak References}{The proposal and rationale for this
                feature, including links to earlier implementations
                and information about similar features in other
                languages.}
\end{seealso}


\subsection{Weak Reference Objects
            \label{weakref-objects}}

Weak reference objects have no attributes or methods, but do allow the
referent to be obtained, if it still exists, by calling it:

\begin{verbatim}
>>> import weakref
>>> class Object:
...     pass
...
>>> o = Object()
>>> r = weakref.ref(o)
>>> o2 = r()
>>> o is o2
True
\end{verbatim}

If the referent no longer exists, calling the reference object returns
\constant{None}:

\begin{verbatim}
>>> del o, o2
>>> print r()
None
\end{verbatim}

Testing that a weak reference object is still live should be done
using the expression \code{\var{ref}() is not None}.  Normally,
application code that needs to use a reference object should follow
this pattern:

\begin{verbatim}
# r is a weak reference object
o = r()
if o is None:
    # referent has been garbage collected
    print "Object has been deallocated; can't frobnicate."
else:
    print "Object is still live!"
    o.do_something_useful()
\end{verbatim}

Using a separate test for ``liveness'' creates race conditions in
threaded applications; another thread can cause a weak reference to
become invalidated before the weak reference is called; the
idiom shown above is safe in threaded applications as well as
single-threaded applications.

Specialized versions of \class{ref} objects can be created through
subclassing.  This is used in the implementation of the
\class{WeakValueDictionary} to reduce the memory overhead for each
entry in the mapping.  This may be most useful to associate additional
information with a reference, but could also be used to insert
additional processing on calls to retrieve the referent.

This example shows how a subclass of \class{ref} can be used to store
additional information about an object and affect the value that's
returned when the referent is accessed:

\begin{verbatim}
import weakref

class ExtendedRef(weakref.ref):
    def __init__(self, ob, callback=None, **annotations):
        super(ExtendedRef, self).__init__(ob, callback)
        self.__counter = 0
        for k, v in annotations.iteritems():
            setattr(self, k, v)

    def __call__(self):
        """Return a pair containing the referent and the number of
        times the reference has been called.
        """
        ob = super(ExtendedRef, self).__call__()
        if ob is not None:
            self.__counter += 1
            ob = (ob, self.__counter)
        return ob
\end{verbatim}


\subsection{Example \label{weakref-example}}

This simple example shows how an application can use objects IDs to
retrieve objects that it has seen before.  The IDs of the objects can
then be used in other data structures without forcing the objects to
remain alive, but the objects can still be retrieved by ID if they
do.

% Example contributed by Tim Peters.
\begin{verbatim}
import weakref

_id2obj_dict = weakref.WeakValueDictionary()

def remember(obj):
    oid = id(obj)
    _id2obj_dict[oid] = obj
    return oid

def id2obj(oid):
    return _id2obj_dict[oid]
\end{verbatim}


\subsection{Weak References in Extension Types
            \label{weakref-extension}}

One of the goals of the implementation is to allow any type to
participate in the weak reference mechanism without incurring the
overhead on those objects which do not benefit by weak referencing
(such as numbers).

For an object to be weakly referencable, the extension must include a
\ctype{PyObject*} field in the instance structure for the use of the
weak reference mechanism; it must be initialized to \NULL{} by the
object's constructor.  It must also set the \member{tp_weaklistoffset}
field of the corresponding type object to the offset of the field.
Also, it needs to add \constant{Py_TPFLAGS_HAVE_WEAKREFS} to the
tp_flags slot.  For example, the instance type is defined with the
following structure:

\begin{verbatim}
typedef struct {
    PyObject_HEAD
    PyClassObject *in_class;       /* The class object */
    PyObject      *in_dict;        /* A dictionary */
    PyObject      *in_weakreflist; /* List of weak references */
} PyInstanceObject;
\end{verbatim}

The statically-declared type object for instances is defined this way:

\begin{verbatim}
PyTypeObject PyInstance_Type = {
    PyObject_HEAD_INIT(&PyType_Type)
    0,
    "module.instance",

    /* Lots of stuff omitted for brevity... */

    Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_WEAKREFS   /* tp_flags */
    0,                                          /* tp_doc */
    0,                                          /* tp_traverse */
    0,                                          /* tp_clear */
    0,                                          /* tp_richcompare */
    offsetof(PyInstanceObject, in_weakreflist), /* tp_weaklistoffset */
};
\end{verbatim}

The type constructor is responsible for initializing the weak reference
list to \NULL:

\begin{verbatim}
static PyObject *
instance_new() {
    /* Other initialization stuff omitted for brevity */

    self->in_weakreflist = NULL;

    return (PyObject *) self;
}
\end{verbatim}

The only further addition is that the destructor needs to call the
weak reference manager to clear any weak references.  This should be
done before any other parts of the destruction have occurred, but is
only required if the weak reference list is non-\NULL:

\begin{verbatim}
static void
instance_dealloc(PyInstanceObject *inst)
{
    /* Allocate temporaries if needed, but do not begin
       destruction just yet.
     */

    if (inst->in_weakreflist != NULL)
        PyObject_ClearWeakRefs((PyObject *) inst);

    /* Proceed with object destruction normally. */
}
\end{verbatim}