Doc/lib/libcodecs.tex

\section{\module{codecs} ---
         Codec registry and base classes}

\declaremodule{standard}{codecs}
\modulesynopsis{Encode and decode data and streams.}
\moduleauthor{Marc-Andre Lemburg}{mal@lemburg.com}
\sectionauthor{Marc-Andre Lemburg}{mal@lemburg.com}


\index{Unicode}
\index{Codecs}
\indexii{Codecs}{encode}
\indexii{Codecs}{decode}
\index{streams}
\indexii{stackable}{streams}


This module defines base classes for standard Python codecs (encoders
and decoders) and provides access to the internal Python codec
registry which manages the codec and error handling lookup process.

It defines the following functions:

\begin{funcdesc}{register}{search_function}
Register a codec search function. Search functions are expected to
take one argument, the encoding name in all lower case letters, and
return a tuple of functions \code{(\var{encoder}, \var{decoder}, \var{stream_reader},
\var{stream_writer})} taking the following arguments:

  \var{encoder} and \var{decoder}: These must be functions or methods
  which have the same interface as the
  \method{encode()}/\method{decode()} methods of Codec instances (see
  Codec Interface). The functions/methods are expected to work in a
  stateless mode.

  \var{stream_reader} and \var{stream_writer}: These have to be
  factory functions providing the following interface:

        \code{factory(\var{stream}, \var{errors}='strict')}

  The factory functions must return objects providing the interfaces
  defined by the base classes \class{StreamWriter} and
  \class{StreamReader}, respectively. Stream codecs can maintain
  state.

  Possible values for errors are \code{'strict'} (raise an exception
  in case of an encoding error), \code{'replace'} (replace malformed
  data with a suitable replacement marker, such as \character{?}) and
  \code{'ignore'} (ignore malformed data and continue without further
  notice).

In case a search function cannot find a given encoding, it should
return \code{None}.
\end{funcdesc}

\begin{funcdesc}{lookup}{encoding}
Looks up a codec tuple in the Python codec registry and returns the
function tuple as defined above.

Encodings are first looked up in the registry's cache. If not found,
the list of registered search functions is scanned. If no codecs tuple
is found, a \exception{LookupError} is raised. Otherwise, the codecs
tuple is stored in the cache and returned to the caller.
\end{funcdesc}

To simplify access to the various codecs, the module provides these
additional functions which use \function{lookup()} for the codec
lookup:

\begin{funcdesc}{getencoder}{encoding}
Lookup up the codec for the given encoding and return its encoder
function.

Raises a \exception{LookupError} in case the encoding cannot be found.
\end{funcdesc}

\begin{funcdesc}{getdecoder}{encoding}
Lookup up the codec for the given encoding and return its decoder
function.

Raises a \exception{LookupError} in case the encoding cannot be found.
\end{funcdesc}

\begin{funcdesc}{getreader}{encoding}
Lookup up the codec for the given encoding and return its StreamReader
class or factory function.

Raises a \exception{LookupError} in case the encoding cannot be found.
\end{funcdesc}

\begin{funcdesc}{getwriter}{encoding}
Lookup up the codec for the given encoding and return its StreamWriter
class or factory function.

Raises a \exception{LookupError} in case the encoding cannot be found.
\end{funcdesc}

\begin{funcdesc}{register_error}{name, error_handler}
Register the error handling function \var{error_handler} under the
name \var{name}. \var{error_handler} will be called during encoding
and decoding in case of an error, when \var{name} is specified as the
errors parameter. \var{error_handler} will be called with an
\exception{UnicodeEncodeError}, \exception{UnicodeDecodeError} or
\exception{UnicodeTranslateError} instance and must return a tuple
with a replacement for the unencodable/undecodable part of the input
and a position where encoding/decoding should continue.
\end{funcdesc}

\begin{funcdesc}{lookup_error}{name}
Return the error handler previously register under the name \var{name}.

Raises a \exception{LookupError} in case the handler cannot be found.
\end{funcdesc}

\begin{funcdesc}{strict_errors}{exception}
Implements the \code{strict} error handling.
\end{funcdesc}

\begin{funcdesc}{replace_errors}{exception}
Implements the \code{replace} error handling.
\end{funcdesc}

\begin{funcdesc}{ignore_errors}{exception}
Implements the \code{ignore} error handling.
\end{funcdesc}

\begin{funcdesc}{xmlcharrefreplace_errors_errors}{exception}
Implements the \code{xmlcharrefreplace} error handling.
\end{funcdesc}

\begin{funcdesc}{backslashreplace_errors_errors}{exception}
Implements the \code{backslashreplace} error handling.
\end{funcdesc}

To simplify working with encoded files or stream, the module
also defines these utility functions:

\begin{funcdesc}{open}{filename, mode\optional{, encoding\optional{,
                       errors\optional{, buffering}}}}
Open an encoded file using the given \var{mode} and return
a wrapped version providing transparent encoding/decoding.

\note{The wrapped version will only accept the object format
defined by the codecs, i.e.\ Unicode objects for most built-in
codecs.  Output is also codec-dependent and will usually be Unicode as
well.}

\var{encoding} specifies the encoding which is to be used for the
the file.

\var{errors} may be given to define the error handling. It defaults
to \code{'strict'} which causes a \exception{ValueError} to be raised
in case an encoding error occurs.

\var{buffering} has the same meaning as for the built-in
\function{open()} function.  It defaults to line buffered.
\end{funcdesc}

\begin{funcdesc}{EncodedFile}{file, input\optional{,
                              output\optional{, errors}}}
Return a wrapped version of file which provides transparent
encoding translation.

Strings written to the wrapped file are interpreted according to the
given \var{input} encoding and then written to the original file as
strings using the \var{output} encoding. The intermediate encoding will
usually be Unicode but depends on the specified codecs.

If \var{output} is not given, it defaults to \var{input}.

\var{errors} may be given to define the error handling. It defaults to
\code{'strict'}, which causes \exception{ValueError} to be raised in case
an encoding error occurs.
\end{funcdesc}

The module also provides the following constants which are useful
for reading and writing to platform dependent files:

\begin{datadesc}{BOM}
\dataline{BOM_BE}
\dataline{BOM_LE}
\dataline{BOM_UTF8}
\dataline{BOM_UTF16}
\dataline{BOM_UTF16_BE}
\dataline{BOM_UTF16_LE}
\dataline{BOM_UTF32}
\dataline{BOM_UTF32_BE}
\dataline{BOM_UTF32_LE}
These constants define various encodings of the Unicode byte order mark
(BOM) used in UTF-16 and UTF-32 data streams to indicate the byte order
used in the stream or file and in UTF-8 as a Unicode signature.
\constant{BOM_UTF16} is either \constant{BOM_UTF16_BE} or
\constant{BOM_UTF16_LE} depending on the platform's native byte order,
\constant{BOM} is an alias for \constant{BOM_UTF16}, \constant{BOM_LE}
for \constant{BOM_UTF16_LE} and \constant{BOM_BE} for \constant{BOM_UTF16_BE}.
The others represent the BOM in UTF-8 and UTF-32 encodings.
\end{datadesc}


\begin{seealso}
  \seeurl{http://sourceforge.net/projects/python-codecs/}{A
          SourceForge project working on additional support for Asian
          codecs for use with Python.  They are in the early stages of
          development at the time of this writing --- look in their
          FTP area for downloadable files.}
\end{seealso}


\subsection{Codec Base Classes}

The \module{codecs} defines a set of base classes which define the
interface and can also be used to easily write you own codecs for use
in Python.

Each codec has to define four interfaces to make it usable as codec in
Python: stateless encoder, stateless decoder, stream reader and stream
writer. The stream reader and writers typically reuse the stateless
encoder/decoder to implement the file protocols.

The \class{Codec} class defines the interface for stateless
encoders/decoders.

To simplify and standardize error handling, the \method{encode()} and
\method{decode()} methods may implement different error handling
schemes by providing the \var{errors} string argument.  The following
string values are defined and implemented by all standard Python
codecs:

\begin{tableii}{l|l}{code}{Value}{Meaning}
  \lineii{'strict'}{Raise \exception{UnicodeError} (or a subclass);
                    this is the default.}
  \lineii{'ignore'}{Ignore the character and continue with the next.}
  \lineii{'replace'}{Replace with a suitable replacement character;
                     Python will use the official U+FFFD REPLACEMENT
                     CHARACTER for the built-in Unicode codecs on
                     decoding and '?' on encoding.}
  \lineii{'xmlcharrefreplace'}{Replace with the appropriate XML
                     character reference (only for encoding).}
  \lineii{'backslashreplace'}{Replace with backslashed escape sequences
                     (only for encoding).}
\end{tableii}

The set of allowed values can be extended via \method{register_error}.


\subsubsection{Codec Objects \label{codec-objects}}

The \class{Codec} class defines these methods which also define the
function interfaces of the stateless encoder and decoder:

\begin{methoddesc}{encode}{input\optional{, errors}}
  Encodes the object \var{input} and returns a tuple (output object,
  length consumed).  While codecs are not restricted to use with Unicode, in
  a Unicode context, encoding converts a Unicode object to a plain string
  using a particular character set encoding (e.g., \code{cp1252} or
  \code{iso-8859-1}).

  \var{errors} defines the error handling to apply. It defaults to
  \code{'strict'} handling.

  The method may not store state in the \class{Codec} instance. Use
  \class{StreamCodec} for codecs which have to keep state in order to
  make encoding/decoding efficient.

  The encoder must be able to handle zero length input and return an
  empty object of the output object type in this situation.
\end{methoddesc}

\begin{methoddesc}{decode}{input\optional{, errors}}
  Decodes the object \var{input} and returns a tuple (output object,
  length consumed).  In a Unicode context, decoding converts a plain string
  encoded using a particular character set encoding to a Unicode object.

  \var{input} must be an object which provides the \code{bf_getreadbuf}
  buffer slot.  Python strings, buffer objects and memory mapped files
  are examples of objects providing this slot.

  \var{errors} defines the error handling to apply. It defaults to
  \code{'strict'} handling.

  The method may not store state in the \class{Codec} instance. Use
  \class{StreamCodec} for codecs which have to keep state in order to
  make encoding/decoding efficient.

  The decoder must be able to handle zero length input and return an
  empty object of the output object type in this situation.
\end{methoddesc}

The \class{StreamWriter} and \class{StreamReader} classes provide
generic working interfaces which can be used to implement new
encodings submodules very easily. See \module{encodings.utf_8} for an
example on how this is done.


\subsubsection{StreamWriter Objects \label{stream-writer-objects}}

The \class{StreamWriter} class is a subclass of \class{Codec} and
defines the following methods which every stream writer must define in
order to be compatible to the Python codec registry.

\begin{classdesc}{StreamWriter}{stream\optional{, errors}}
  Constructor for a \class{StreamWriter} instance. 

  All stream writers must provide this constructor interface. They are
  free to add additional keyword arguments, but only the ones defined
  here are used by the Python codec registry.

  \var{stream} must be a file-like object open for writing (binary)
  data.

  The \class{StreamWriter} may implement different error handling
  schemes by providing the \var{errors} keyword argument. These
  parameters are predefined:

  \begin{itemize}
    \item \code{'strict'} Raise \exception{ValueError} (or a subclass);
                          this is the default.
    \item \code{'ignore'} Ignore the character and continue with the next.
    \item \code{'replace'} Replace with a suitable replacement character
    \item \code{'xmlcharrefreplace'} Replace with the appropriate XML
                     character reference
    \item \code{'backslashreplace'} Replace with backslashed escape sequences.
  \end{itemize}

  The \var{errors} argument will be assigned to an attribute of the
  same name. Assigning to this attribute makes it possible to switch
  between different error handling strategies during the lifetime
  of the \class{StreamWriter} object.

  The set of allowed values for the \var{errors} argument can
  be extended with \function{register_error()}.
\end{classdesc}

\begin{methoddesc}{write}{object}
  Writes the object's contents encoded to the stream.
\end{methoddesc}

\begin{methoddesc}{writelines}{list}
  Writes the concatenated list of strings to the stream (possibly by
  reusing the \method{write()} method).
\end{methoddesc}

\begin{methoddesc}{reset}{}
  Flushes and resets the codec buffers used for keeping state.

  Calling this method should ensure that the data on the output is put
  into a clean state, that allows appending of new fresh data without
  having to rescan the whole stream to recover state.
\end{methoddesc}

In addition to the above methods, the \class{StreamWriter} must also
inherit all other methods and attribute from the underlying stream.


\subsubsection{StreamReader Objects \label{stream-reader-objects}}

The \class{StreamReader} class is a subclass of \class{Codec} and
defines the following methods which every stream reader must define in
order to be compatible to the Python codec registry.

\begin{classdesc}{StreamReader}{stream\optional{, errors}}
  Constructor for a \class{StreamReader} instance. 

  All stream readers must provide this constructor interface. They are
  free to add additional keyword arguments, but only the ones defined
  here are used by the Python codec registry.

  \var{stream} must be a file-like object open for reading (binary)
  data.

  The \class{StreamReader} may implement different error handling
  schemes by providing the \var{errors} keyword argument. These
  parameters are defined:

  \begin{itemize}
    \item \code{'strict'} Raise \exception{ValueError} (or a subclass);
                          this is the default.
    \item \code{'ignore'} Ignore the character and continue with the next.
    \item \code{'replace'} Replace with a suitable replacement character.
  \end{itemize}

  The \var{errors} argument will be assigned to an attribute of the
  same name. Assigning to this attribute makes it possible to switch
  between different error handling strategies during the lifetime
  of the \class{StreamReader} object.

  The set of allowed values for the \var{errors} argument can
  be extended with \function{register_error()}.
\end{classdesc}

\begin{methoddesc}{read}{\optional{size}}
  Decodes data from the stream and returns the resulting object.

  \var{size} indicates the approximate maximum number of bytes to read
  from the stream for decoding purposes. The decoder can modify this
  setting as appropriate. The default value -1 indicates to read and
  decode as much as possible.  \var{size} is intended to prevent having
  to decode huge files in one step.

  The method should use a greedy read strategy meaning that it should
  read as much data as is allowed within the definition of the encoding
  and the given size, e.g.  if optional encoding endings or state
  markers are available on the stream, these should be read too.
\end{methoddesc}

\begin{methoddesc}{readline}{[size]}
  Read one line from the input stream and return the
  decoded data.

  Unlike the \method{readlines()} method, this method inherits
  the line breaking knowledge from the underlying stream's
  \method{readline()} method -- there is currently no support for line
  breaking using the codec decoder due to lack of line buffering.
  Sublcasses should however, if possible, try to implement this method
  using their own knowledge of line breaking.

  \var{size}, if given, is passed as size argument to the stream's
  \method{readline()} method.
\end{methoddesc}

\begin{methoddesc}{readlines}{[sizehint]}
  Read all lines available on the input stream and return them as list
  of lines.

  Line breaks are implemented using the codec's decoder method and are
  included in the list entries.

  \var{sizehint}, if given, is passed as \var{size} argument to the
  stream's \method{read()} method.
\end{methoddesc}

\begin{methoddesc}{reset}{}
  Resets the codec buffers used for keeping state.

  Note that no stream repositioning should take place.  This method is
  primarily intended to be able to recover from decoding errors.
\end{methoddesc}

In addition to the above methods, the \class{StreamReader} must also
inherit all other methods and attribute from the underlying stream.

The next two base classes are included for convenience. They are not
needed by the codec registry, but may provide useful in practice.


\subsubsection{StreamReaderWriter Objects \label{stream-reader-writer}}

The \class{StreamReaderWriter} allows wrapping streams which work in
both read and write modes.

The design is such that one can use the factory functions returned by
the \function{lookup()} function to construct the instance.

\begin{classdesc}{StreamReaderWriter}{stream, Reader, Writer, errors}
  Creates a \class{StreamReaderWriter} instance.
  \var{stream} must be a file-like object.
  \var{Reader} and \var{Writer} must be factory functions or classes
  providing the \class{StreamReader} and \class{StreamWriter} interface
  resp.
  Error handling is done in the same way as defined for the
  stream readers and writers.
\end{classdesc}

\class{StreamReaderWriter} instances define the combined interfaces of
\class{StreamReader} and \class{StreamWriter} classes. They inherit
all other methods and attribute from the underlying stream.


\subsubsection{StreamRecoder Objects \label{stream-recoder-objects}}

The \class{StreamRecoder} provide a frontend - backend view of
encoding data which is sometimes useful when dealing with different
encoding environments.

The design is such that one can use the factory functions returned by
the \function{lookup()} function to construct the instance.

\begin{classdesc}{StreamRecoder}{stream, encode, decode,
                                 Reader, Writer, errors}
  Creates a \class{StreamRecoder} instance which implements a two-way
  conversion: \var{encode} and \var{decode} work on the frontend (the
  input to \method{read()} and output of \method{write()}) while
  \var{Reader} and \var{Writer} work on the backend (reading and
  writing to the stream).

  You can use these objects to do transparent direct recodings from
  e.g.\ Latin-1 to UTF-8 and back.

  \var{stream} must be a file-like object.

  \var{encode}, \var{decode} must adhere to the \class{Codec}
  interface, \var{Reader}, \var{Writer} must be factory functions or
  classes providing objects of the the \class{StreamReader} and
  \class{StreamWriter} interface respectively.

  \var{encode} and \var{decode} are needed for the frontend
  translation, \var{Reader} and \var{Writer} for the backend
  translation.  The intermediate format used is determined by the two
  sets of codecs, e.g. the Unicode codecs will use Unicode as
  intermediate encoding.

  Error handling is done in the same way as defined for the
  stream readers and writers.
\end{classdesc}

\class{StreamRecoder} instances define the combined interfaces of
\class{StreamReader} and \class{StreamWriter} classes. They inherit
all other methods and attribute from the underlying stream.