Doc/library/email.charset.rst

:mod:`email.charset`: Representing character sets
-------------------------------------------------

.. module:: email.charset
   :synopsis: Character Sets


This module provides a class :class:`Charset` for representing character sets
and character set conversions in email messages, as well as a character set
registry and several convenience methods for manipulating this registry.
Instances of :class:`Charset` are used in several other modules within the
:mod:`email` package.

Import this class from the :mod:`email.charset` module.


.. class:: Charset(input_charset=DEFAULT_CHARSET)

   Map character sets to their email properties.

   This class provides information about the requirements imposed on email for a
   specific character set.  It also provides convenience routines for converting
   between character sets, given the availability of the applicable codecs.  Given
   a character set, it will do its best to provide information on how to use that
   character set in an email message in an RFC-compliant way.

   Certain character sets must be encoded with quoted-printable or base64 when used
   in email headers or bodies.  Certain character sets must be converted outright,
   and are not allowed in email.

   Optional *input_charset* is as described below; it is always coerced to lower
   case.  After being alias normalized it is also used as a lookup into the
   registry of character sets to find out the header encoding, body encoding, and
   output conversion codec to be used for the character set.  For example, if
   *input_charset* is ``iso-8859-1``, then headers and bodies will be encoded using
   quoted-printable and no output conversion codec is necessary.  If
   *input_charset* is ``euc-jp``, then headers will be encoded with base64, bodies
   will not be encoded, but output text will be converted from the ``euc-jp``
   character set to the ``iso-2022-jp`` character set.

   :class:`Charset` instances have the following data attributes:

   .. attribute:: input_charset

      The initial character set specified.  Common aliases are converted to
      their *official* email names (e.g. ``latin_1`` is converted to
      ``iso-8859-1``).  Defaults to 7-bit ``us-ascii``.


   .. attribute:: header_encoding

      If the character set must be encoded before it can be used in an email
      header, this attribute will be set to ``Charset.QP`` (for
      quoted-printable), ``Charset.BASE64`` (for base64 encoding), or
      ``Charset.SHORTEST`` for the shortest of QP or BASE64 encoding. Otherwise,
      it will be ``None``.


   .. attribute:: body_encoding

      Same as *header_encoding*, but describes the encoding for the mail
      message's body, which indeed may be different than the header encoding.
      ``Charset.SHORTEST`` is not allowed for *body_encoding*.


   .. attribute:: output_charset

      Some character sets must be converted before they can be used in email
      headers or bodies.  If the *input_charset* is one of them, this attribute
      will contain the name of the character set output will be converted to.
      Otherwise, it will be ``None``.


   .. attribute:: input_codec

      The name of the Python codec used to convert the *input_charset* to
      Unicode.  If no conversion codec is necessary, this attribute will be
      ``None``.


   .. attribute:: output_codec

      The name of the Python codec used to convert Unicode to the
      *output_charset*.  If no conversion codec is necessary, this attribute
      will have the same value as the *input_codec*.


   :class:`Charset` instances also have the following methods:

   .. method:: get_body_encoding()

      Return the content transfer encoding used for body encoding.

      This is either the string ``quoted-printable`` or ``base64`` depending on
      the encoding used, or it is a function, in which case you should call the
      function with a single argument, the Message object being encoded.  The
      function should then set the :mailheader:`Content-Transfer-Encoding`
      header itself to whatever is appropriate.

      Returns the string ``quoted-printable`` if *body_encoding* is ``QP``,
      returns the string ``base64`` if *body_encoding* is ``BASE64``, and
      returns the string ``7bit`` otherwise.

   .. XXX to_splittable and from_splittable are not there anymore!

   .. method to_splittable(s)

      Convert a possibly multibyte string to a safely splittable format. *s* is
      the string to split.

      Uses the *input_codec* to try and convert the string to Unicode, so it can
      be safely split on character boundaries (even for multibyte characters).

      Returns the string as-is if it isn't known how to convert *s* to Unicode
      with the *input_charset*.

      Characters that could not be converted to Unicode will be replaced with
      the Unicode replacement character ``'U+FFFD'``.


   .. method from_splittable(ustr[, to_output])

      Convert a splittable string back into an encoded string.  *ustr* is a
      Unicode string to "unsplit".

      This method uses the proper codec to try and convert the string from
      Unicode back into an encoded format.  Return the string as-is if it is not
      Unicode, or if it could not be converted from Unicode.

      Characters that could not be converted from Unicode will be replaced with
      an appropriate character (usually ``'?'``).

      If *to_output* is ``True`` (the default), uses *output_codec* to convert
      to an encoded format.  If *to_output* is ``False``, it uses *input_codec*.


   .. method:: get_output_charset()

      Return the output character set.

      This is the *output_charset* attribute if that is not ``None``, otherwise
      it is *input_charset*.


   .. method:: header_encode(string)

      Header-encode the string *string*.

      The type of encoding (base64 or quoted-printable) will be based on the
      *header_encoding* attribute.


   .. method:: header_encode_lines(string, maxlengths)

      Header-encode a *string* by converting it first to bytes.

      This is similar to :meth:`header_encode` except that the string is fit
      into maximum line lengths as given by the argument *maxlengths*, which
      must be an iterator: each element returned from this iterator will provide
      the next maximum line length.


   .. method:: body_encode(string)

      Body-encode the string *string*.

      The type of encoding (base64 or quoted-printable) will be based on the
      *body_encoding* attribute.

   The :class:`Charset` class also provides a number of methods to support
   standard operations and built-in functions.


   .. method:: __str__()

      Returns *input_charset* as a string coerced to lower
      case. :meth:`__repr__` is an alias for :meth:`__str__`.


   .. method:: __eq__(other)

      This method allows you to compare two :class:`Charset` instances for
      equality.


   .. method:: __ne__(other)

      This method allows you to compare two :class:`Charset` instances for
      inequality.

The :mod:`email.charset` module also provides the following functions for adding
new entries to the global character set, alias, and codec registries:


.. function:: add_charset(charset, header_enc=None, body_enc=None, output_charset=None)

   Add character properties to the global registry.

   *charset* is the input character set, and must be the canonical name of a
   character set.

   Optional *header_enc* and *body_enc* is either ``Charset.QP`` for
   quoted-printable, ``Charset.BASE64`` for base64 encoding,
   ``Charset.SHORTEST`` for the shortest of quoted-printable or base64 encoding,
   or ``None`` for no encoding.  ``SHORTEST`` is only valid for
   *header_enc*. The default is ``None`` for no encoding.

   Optional *output_charset* is the character set that the output should be in.
   Conversions will proceed from input charset, to Unicode, to the output charset
   when the method :meth:`Charset.convert` is called.  The default is to output in
   the same character set as the input.

   Both *input_charset* and *output_charset* must have Unicode codec entries in the
   module's character set-to-codec mapping; use :func:`add_codec` to add codecs the
   module does not know about.  See the :mod:`codecs` module's documentation for
   more information.

   The global character set registry is kept in the module global dictionary
   ``CHARSETS``.


.. function:: add_alias(alias, canonical)

   Add a character set alias.  *alias* is the alias name, e.g. ``latin-1``.
   *canonical* is the character set's canonical name, e.g. ``iso-8859-1``.

   The global charset alias registry is kept in the module global dictionary
   ``ALIASES``.


.. function:: add_codec(charset, codecname)

   Add a codec that map characters in the given character set to and from Unicode.

   *charset* is the canonical name of a character set. *codecname* is the name of a
   Python codec, as appropriate for the second argument to the :class:`str`'s
   :meth:`~str.encode` method