Doc/whatsnew/3.2.rst

****************************
  What's New In Python 3.2
****************************

:Author: Raymond Hettinger
:Release: |release|
:Date: |today|

.. $Id$
   Rules for maintenance:

   * Anyone can add text to this document.  Do not spend very much time
   on the wording of your changes, because your text will probably
   get rewritten to some degree.

   * The maintainer will go through Misc/NEWS periodically and add
   changes; it's therefore more important to add your changes to
   Misc/NEWS than to this file.

   * This is not a complete list of every single change; completeness
   is the purpose of Misc/NEWS.  Some changes I consider too small
   or esoteric to include.  If such a change is added to the text,
   I'll just remove it.  (This is another reason you shouldn't spend
   too much time on writing your addition.)

   * If you want to draw your new text to the attention of the
   maintainer, add 'XXX' to the beginning of the paragraph or
   section.

   * It's OK to just add a fragmentary note about a change.  For
   example: "XXX Describe the transmogrify() function added to the
   socket module."  The maintainer will research the change and
   write the necessary text.

   * You can comment out your additions if you like, but it's not
   necessary (especially when a final release is some months away).

   * Credit the author of a patch or bugfix.   Just the name is
   sufficient; the e-mail address isn't necessary.  It's helpful to
   add the issue number:

     XXX Describe the transmogrify() function added to the socket
     module.

     (Contributed by P.Y. Developer; :issue:`12345`.)

   This saves the maintainer the effort of going through the SVN log
   when researching a change.

This article explains the new features in Python 3.2, compared to 3.1.


PEP 391:  Dictionary Based Configuration for Logging
====================================================

The :mod:`logging` module provided two kinds of configuration, one style with
function calls for each option or another style driven by an external file saved
in a :mod:`ConfigParser` format.  Those options did not provide the flexibility
to create configurations from JSON or YAML files, nor did they support
incremental configuration, which is needed for specifying logger options from a
command line.

To support a more flexible style, the module now offers
:func:`logging.config.dictConfig` for specifying logging configuration with
plain Python dictionaries.  The configuration options include formatters,
handlers, filters, and loggers.  Here's a working example of a configuration
dictionary::

   {"version": 1,
    "formatters": {"brief": {"format": "%(levelname)-8s: %(name)-15s: %(message)s"},
                   "full": {"format": "%(asctime)s %(name)-15s %(levelname)-8s %(message)s"},
                   },
    "handlers": {"console": {
                      "class": "logging.StreamHandler",
                      "formatter": "brief",
                      "level": "INFO",
                      "stream": "ext://sys.stdout"},
                 "console_priority": {
                      "class": "logging.StreamHandler",
                      "formatter": "full",
                      "level": "ERROR",
                      "stream": "ext://sys.stderr"},
                 },
    "root": {"level": "DEBUG", "handlers": ["console", "console_priority"]}}


If that dictionary is stored in a file called "conf.json", it can loaded
and called with code like this::

   >>> import logging.config
   >>> logging.config.dictConfig(json.load(open('conf.json', 'rb')))
   >>> logging.info("Transaction completed normally")
   >>> logging.critical("Abnormal termination")

.. seealso::

   :pep:`391` - Dictionary Based Configuration for Logging
      PEP written by Vinay Sajip.


PEP 3147:  PYC Repository Directories
=====================================

Python's scheme for caching bytecode in *.pyc* files did not work well in
environments with multiple python interpreters.  If one interpreter encountered
a cached file created by another interpreter, it would recompile the source and
overwrite the cached file, thus losing the benefits of caching.

The issue of "pyc fights" has become more pronounced as it has become
commonplace for Linux distributions to ship with multiple versions of Python.
These conflicts also arise with CPython alternatives such as Unladen Swallow.

To solve this problem, Python's import machinery has been extended to use
distinct filenames for each interpreter.  Instead of Python 3.2 and Python 3.3 and
Unladen Swallow each competing for a file called "mymodule.pyc", they will now
look for "mymodule.cpython-32.pyc", "mymodule.cpython-33.pyc", and
"mymodule.unladen10.pyc".  And to prevent all of these new files from
cluttering source directories, the *pyc* files are now collected in a
"__pycache__" directory stored under the package directory.

Aside from the filenames and target directories, the new scheme has a few
aspects that are visible to the programmer:

* Imported modules now have a :attr:`__cached__` attribute which stores the name
  of the actual file that was imported:

   >>> import collections
   >>> collections.__cached__
   'c:/py32/lib/__pycache__/collections.cpython-32.pyc'

* The tag that is unique to each interpreter is accessible from the :mod:`imp`
  module:

   >>> import imp
   >>> imp.get_tag()
   'cpython-32'

* Scripts that try to deduce source filename from the imported file now need to
  be smarter.  It is no longer sufficient to simply strip the "c" from a ".pyc"
  filename.  Instead, use the new functions in the :mod:`imp` module:

  >>> imp.source_from_cache('c:/py32/lib/__pycache__/collections.cpython-32.pyc')
  'c:/py32/lib/collections.py'
  >>> imp.cache_from_source('c:/py32/lib/collections.py')
  'c:/py32/lib/__pycache__/collections.cpython-32.pyc'

* The :mod:`py_compile` and :mod:`compileall` modules have been updated to
  reflect the new naming convention and target directory.

.. seealso::

   :pep:`3147` - PYC Repository Directories
      PEP written by Barry Warsaw.


PEP 3149 ABI Version Tagged .so Files
=====================================

The PYC repository directory allows multiple bytecode cache files to be
co-located.  This PEP implements a similar mechanism for shared object files by
giving them a common directory and distinct names for each version.

The common directory is "pyshared" and the file names are made distinct by
identifying the Python implementation (such as CPython, PyPy, Jython, etc.), the
major and minor version numbers, and optional build flags (such as "d" for
debug, "m" for pymalloc, "u" for wide-unicode).  For an arbitrary package "foo",
you may see these files when the distribution package is installed::

   /usr/share/pyshared/foo.cpython-32m.so
   /usr/share/pyshared/foo.cpython-33md.so

In Python itself, the tags are accessible from functions in the :mod:`sysconfig`
module::

   >>> import sysconfig
   >>> sysconfig.get_config_var('SOABI')    # find the version tag
   'cpython-32mu'
   >>> sysconfig.get_config_var('SO')       # find the full filename extension
   'cpython-32mu.so'

.. seealso::

   :pep:`3149` - ABI Version Tagged .so Files
      PEP written by Barry Warsaw.


Email 5.1
=========

The email package is extended to be able to parse and generate email messages
in bytes format.

* New functions :func:`~email.message_from_bytes` and
  :func:`~email.message_from_binary_file`, and new classes
  :class:`~email.parser.BytesFeedParser` and :class:`~email.parser.BytesParser`
  allow binary message data to be parsed into model objects.

* Given bytes input to the model, :meth:`~email.message.Message.get_payload`
  will by default decode a message body that has a
  :mailheader:`Content-Transfer-Encoding` of ``8bit`` using the charset
  specified in the MIME headers and return the resulting string.

* Given bytes input to the model, :class:`~email.generator.Generator` will
  convert message bodies that have a :mailheader:`Content-Transfer-Encoding` of
  8bit to instead have a 7bit Content-Transfer-Encoding.

* New class :class:`~email.generator.BytesGenerator` produces bytes
  as output, preserving any unchanged non-ASCII data that was
  present in the input used to build the model, including message bodies
  with a :mailheader:`Content-Transfer-Encoding` of 8bit.

  (Proposed and implemented by R. David Murray, :issue:`4661`.)


Other Language Changes
======================

Some smaller changes made to the core Python language are:

* The :func:`hasattr` function used to catch and suppress any Exception.  Now,
  it only catches :exc:`AttributeError`.  Under the hood, :func:`hasattr` works
  by calling :func:`getattr` and throwing away the results.  This is necessary
  because dynamic attribute creation is possible using :meth:`__getattribute__`
  or :meth:`__getattr__`.  If :func:`hasattr` were to just scan instance and class
  dictionaries it would miss the dynamic methods and make it difficult to
  implement proxy objects.

  (Discovered by Yury Selivanov and fixed by Benjamin Peterson; :issue:`9666`.)

* The :func:`str` of a float or complex number is now the same as its
  :func:`repr`. Previously, the :func:`str` form was shorter but that just
  caused confusion and is no longer needed now that the shortest possible
  :func:`repr` is displayed by default:

   >>> repr(math.pi)
   '3.141592653589793'
   >>> str(math.pi)
   '3.141592653589793'

  (Proposed and implemented by Mark Dickinson; :issue:`9337`.)

* :class:`memoryview` objects now have a :meth:`release()` method and support
  the context manager protocol.  This allows timely release of any resources
  that were acquired when requesting a buffer from the original object.

  (Added by Antoine Pitrou; :issue:`9757`.)

* A warning message will now get printed at interpreter shutdown if the
  :data:`gc.garbage` list isn't empty.  This is meant to make the programmer
  aware that their code contains object finalization issues.

  (Added by Antoine Pitrou; :issue:`477863`.)

* Mark Dickinson crafted an elegant and efficient scheme for assuring that
  different numeric datatypes will have the same hash value whenever their
  actual values are equal::

   >>> assert hash(Fraction(3, 2)) == hash(1.5) == \
              hash(Decimal("1.5")) == hash(complex(1.5, 0))

  (See :issue:`8188`.)

* Previously it was illegal to delete a name from the local namespace if it
  occurs as a free variable in a nested block::

   >>> def outer(x):
   ...     def inner():
   ...        return x
   ...     inner()
   ...     del x

  This is now allowed.  Remember that the target of an :keyword:`except` clause
  is cleared, so this code which used to work with Python 2.6, raised a
  :exc:`SyntaxError` with Python 3.1 and now works again::

   >>> def f():
   ...     def print_error():
   ...        print(e)
   ...     try:
   ...        something
   ...     except Exception as e:
   ...        print_error()
   ...        # implicit "del e" here

  (See :issue:`4617`.)


New, Improved, and Deprecated Modules
=====================================

* XXX mention :mod:`argparse`.

* The :mod:`functools` module includes a new decorator for caching function
  calls.  :func:`functools.lru_cache` can save repeated queries to an external
  resource whenever the results are expected to be the same.

  For example, adding a caching decorator to a database query function can save
  database accesses for popular searches::

     @functools.lru_cache(maxsize=300)
     def get_phone_number(name):
         c = conn.cursor()
         c.execute('SELECT phonenumber FROM phonelist WHERE name=?', (name,))
         return c.fetchone()[0]

  To help with choosing an effective cache size, the wrapped function is
  instrumented with two attributes *cache_hits* and *cache_misses*:

  >>> for name in user_requests:
  ...     get_phone_number(name)
  >>> print(get_phone_number.cache_hits, get_phone_number.cache_misses)
  4805 980

  If the phonelist table gets updated, the outdated contents of the cache can be
  cleared with:

  >>> get_phone_number.cache_clear()

  (Contributed by Raymond Hettinger.)

* The :func:`functools.wraps` decorator now adds a :attr:`__wrapped__` attribute
  pointing to the original callable function.  This allows wrapped functions to
  be introspected.  It also copies :attr:`__annotations__` if defined.  And now
  it also gracefully skips over missing attributes such as :attr:`__doc__` which
  might not be defined for the wrapped callable.

  (By Nick Coghlan and Terrence Cole; :issue:`9567`, :issue:`3445`, and
  :issue:`8814`.)

* The :mod:`nntplib` module gets a revamped implementation with better
  bytes / unicode semantics as well as more practical APIs.  These improvements
  break compatibility with the nntplib version in Python 3.1, which was
  partly dysfunctional in itself.

  (Contributed by Antoine Pitrou in :issue:`9360`)

* The :mod:`abc` module now supports :func:`~abc.abstractclassmethod` and
  :func:`~abc.abstractstaticmethod`.

  (Patch submitted by Daniel Urban; :issue:`5867`.)

* The previously deprecated :func:`contextlib.nested` function has been removed
  in favor of a plain :keyword:`with` statement which can accept multiple
  context managers.  The latter technique is faster (because it is built-in),
  and it does a better job finalizing multiple context managers when one of them
  raises an exception.

  (Contributed by Georg Brandl and Mattias Brändström;
  `appspot issue 53094 <http://codereview.appspot.com/53094>`_.)

* The :class:`ftplib.FTP` class now supports the context manager protocol to
  unconditionally consume :exc:`socket.error` exceptions and to close the FTP
  connection when done::

   >>> from ftplib import FTP
   >>> with FTP("ftp1.at.proftpd.org") as ftp:
   ...     ftp.login()
   ...     ftp.dir()
   ...
   '230 Anonymous login ok, restrictions apply.'
   dr-xr-xr-x   9 ftp      ftp           154 May  6 10:43 .
   dr-xr-xr-x   9 ftp      ftp           154 May  6 10:43 ..
   dr-xr-xr-x   5 ftp      ftp          4096 May  6 10:43 CentOS
   dr-xr-xr-x   3 ftp      ftp            18 Jul 10  2008 Fedora

  Other file-like objects such as :class:`mmap.mmap` and :func:`fileinput.input`
  also grew auto-closing context managers::

      with fileinput.input(files=('log1.txt', 'log2.txt')) as f:
          for line in f:
              process(line)

  (Contributed by Tarek Ziadé and Giampaolo Rodolà in :issue:`4972`, and
  by Georg Brandl in :issue:`8046` and :issue:`1286`.)

* :class:`gzip.GzipFile` now implements the :class:`io.BufferedIOBase` ABC
  (except for ``truncate()``), has a :meth:`~gzip.GzipFile.peek` method,
  and supports unseekable as well as zero-padded file objects.

  (Contributed by Antoine Pitrou, Nir Aides and Brian Curtin in :issue:`9962`,
  :issue:`1675951`, :issue:`7471` and :issue:`2846`.)

  The :mod:`gzip` module also gains the :func:`~gzip.compress` and
  :func:`~gzip.decompress` functions for easier in-memory compression and
  decompression.

  (Contributed by Anand B. Pillai in :issue:`3488`.)

* The :mod:`os` module now has the :const:`ST_RDONLY` and :const:`ST_NOSUID`
  constants, for use with the :func:`~os.statvfs` function.

  (Patch by Adam Jackson; :issue:`7647`.)

* :func:`os.getppid` is now supported on Windows.  Note that it will continue to
  return the same pid even after the parent process has exited.

  (Patch by Jon Anglin; :issue:`6394`.)

* The :func:`shutil.copytree` function has two new options:

  * *ignore_dangling_symlinks*: when ``symlinks=False`` so that the function
    copies the file pointed to by the symlink, not the symlink itself. This
    option will silence the error raised if the file doesn't exist.

  * *copy_function*: is a callable that will be used to copy files.
    :func:`shutil.copy2` is used by default.

  (Contributed by Tarek Ziadé.)

* Socket objects now have a :meth:`~socket.socket.detach()` method which puts
  the socket into closed state without actually closing the underlying file
  descriptor.  The latter can then be reused for other purposes.

  (Added by Antoine Pitrou; :issue:`8524`.)

* The :mod:`sqlite3` module has two new capabilities.

  The :attr:`Connection.in_transit` attribute is true if there is an active
  transaction for uncommitted changes.

  The :meth:`Connection.enable_load_extension` and
  :meth:`Connection.load_extension` methods allows you to load SQLite extensions
  from ".so" files.  One well-known extension is the fulltext-search extension
  distributed with SQLite.

  (Contributed by R. David Murray and Shashwat Anand; :issue:`8845`.)

* The :mod:`ssl` module has a new class, :class:`~ssl.SSLContext` which serves
  as a container for various persistent SSL data, such as protocol settings,
  certificates, private keys, and various other options.  The
  :meth:`~ssl.SSLContext.wrap_socket` method allows to create an SSL socket from
  such an SSL context.  (Added by Antoine Pitrou; :issue:`8550`.)

  A new function, :func:`ssl.match_hostname`, helps implement server identity
  verification for higher-level protocols by implementing the rules of
  HTTPS (from :rfc:`2818`), which are also suitable for other protocols.
  (Added by Antoine Pitrou, :issue:`1589`).

  The :func:`ssl.wrap_socket` constructor function now takes a *ciphers*
  argument that's a string listing the encryption algorithms to be allowed; the
  format of the string is described `in the OpenSSL documentation
  <http://www.openssl.org/docs/apps/ciphers.html#CIPHER_LIST_FORMAT>`__.  (Added
  by Antoine Pitrou; :issue:`8322`.)

  Various options have been added to the :mod:`ssl` module, such as
  :data:`~ssl.OP_NO_SSLv2` which allows to force disabling of the insecure and
  obsolete SSLv2 protocol.  (Added by Antoine Pitrou; :issue:`4870`.)

  Another change makes the extension load all of OpenSSL's ciphers and digest
  algorithms so that they're all available.  Some SSL certificates couldn't be
  verified, reporting an "unknown algorithm" error.  (Reported by Beda Kosata,
  and fixed by Antoine Pitrou; :issue:`8484`.)

  The version of OpenSSL being used is now available as the module attributes
  :data:`ssl.OPENSSL_VERSION` (a string), :data:`ssl.OPENSSL_VERSION_INFO` (a
  5-tuple), and :data:`ssl.OPENSSL_VERSION_NUMBER` (an integer).  (Added by
  Antoine Pitrou; :issue:`8321`.)

* Instances of :class:`unittest.TestCase` have two new methods
  :meth:`~unittest.TestCase.assertWarns` and :meth:`~unittest.TestCase.assertWarnsRegexp`
  to check that a given warning type was triggered by the code under test::

      with self.assertWarns(DeprecationWarning):
          legacy_function('XYZ')


* The previously deprecated :func:`string.maketrans` function has been removed
  in favor of the static methods, :meth:`bytes.maketrans` and
  :meth:`bytearray.maketrans`.  This change solves the confusion around which
  types were supported by the :mod:`string` module.  Now, :class:`str`,
  :class:`bytes`, and :class:`bytearray` each have their own **maketrans** and
  **translate** methods with intermediate translation tables of the appropriate
  type.

  (Contributed by Georg Brandl; :issue:`5675`.)

* :class:`~poplib.POP3_SSL` class now accepts a *context* parameter, which is a
  :class:`ssl.SSLContext` object allowing bundling SSL configuration options,
  certificates and private keys into a single (potentially long-lived)
  structure.

  (Contributed by Giampaolo Rodolà; :issue:`8807`.)

* :func:`socket.create_connection` now supports the context manager protocol
  to unconditionally consume :exc:`socket.error` exceptions and to close the
  socket when done.

  (Contributed by Giampaolo Rodolà; :issue:`9794`.)

* :class:`asyncore.dispatcher` now provides a
  :meth:`~asyncore.dispatcher.handle_accepted()` method
  returning a `(sock, addr)` pair which is called when a connection has actually
  been established with a new remote endpoint. This is supposed to be used as a
  replacement for old :meth:`~asyncore.dispatcher.handle_accept()` and avoids
  the user  to call :meth:`~asyncore.dispatcher.accept()` directly.

  (Contributed by Giampaolo Rodolà; :issue:`6706`.)

* The :mod:`tempfile` module has a new context manager,
  :class:`~tempfile.TemporaryDirectory` which provides easy deterministic
  cleanup of temporary directories.

  (Contributed by Neil Schemenauer and Nick Coghlan; :issue:`5178`.)


Multi-threading
===============

* The mechanism for serializing execution of concurrently running Python threads
  (generally known as the GIL or Global Interpreter Lock) has been rewritten.
  Among the objectives were more predictable switching intervals and reduced
  overhead due to lock contention and the number of ensuing system calls.  The
  notion of a "check interval" to allow thread switches has been abandoned and
  replaced by an absolute duration expressed in seconds.  This parameter is
  tunable through :func:`sys.setswitchinterval()`.  It currently defaults to 5
  milliseconds.

  Additional details about the implementation can be read from a `python-dev
  mailing-list message
  <http://mail.python.org/pipermail/python-dev/2009-October/093321.html>`_
  (however, "priority requests" as exposed in this message have not been kept
  for inclusion).

  (Contributed by Antoine Pitrou.)

* Recursive locks (created with the :func:`threading.RLock` API) now benefit
  from a C implementation which makes them as fast as regular locks, and between
  10x and 15x faster than their previous pure Python implementation.

  (Contributed by Antoine Pitrou; :issue:`3001`.)

* Regular and recursive locks now accept an optional *timeout* argument to their
  :meth:`acquire` method.  (Contributed by Antoine Pitrou; :issue:`7316`.)

  Similarly, :meth:`threading.Semaphore.acquire` also gains a *timeout*
  argument.  (Contributed by Torsten Landschoff; :issue:`850728`.)


Optimizations
=============

A number of small performance enhancements have been added:

* JSON decoding performance is improved and memory consumption is reduced
  whenever the same string is repeated for multiple keys.

  (Contributed by Antoine Pitrou; :issue:`7451`.)

* Python's peephole optimizer now recognizes patterns such ``x in {1, 2, 3}`` as
  being a test for membership in a set of constants.  The optimizer recasts the
  :class:`set` as a :class:`frozenset` and stores the pre-built constant.

  Now that the speed penalty is gone, it is practical to start writing
  membership tests using set-notation.  This style is both semantically clear
  and operationally fast::

      extension = name.rpartition('.')[2]
      if extension in {'xml', 'html', 'xhtml', 'css'}:
          handle(name)

  (Patch and additional tests by Dave Malcolm; :issue:`6690`).

* The fast-search algorithm in stringlib is now used by the :meth:`split`,
  :meth:`rsplit`, :meth:`splitlines` and :meth:`replace` methods on
  :class:`bytes`, :class:`bytearray` and :class:`str` objects. Likewise, the
  algorithm is also used by :meth:`rfind`, :meth:`rindex`, :meth:`rsplit` and
  :meth:`rpartition`.

  (Patch by Florent Xicluna in :issue:`7622` and :issue:`7462`.)

* Serializing and unserializing data using the :mod:`pickle` module is now
  several times faster. (Contributed by Alexandre Vassalotti, Antoine Pitrou
  and the Unladen Swallow team in :issue:`9410` and :issue:`3873`.)


Filenames and Unicode
=====================

The :mod:`os` module has two new functions: :func:`~os.fsencode` and
:func:`~os.fsdecode`.

.. XXX mention Victor's improvements for support of undecodable filenames.


.. IDLE
   ====

   * Stub


Build and C API Changes
=======================

Changes to Python's build process and to the C API include:

* The C functions that access the Unicode Database now accept and return
  characters from the full Unicode range, even on narrow unicode builds
  (Py_UNICODE_TOLOWER, Py_UNICODE_ISDECIMAL, and others).  A visible difference
  in Python is that :func:`unicodedata.numeric` now returns the correct value
  for large code points, and :func:`repr` may consider more characters as
  printable.

  (Reported by Bupjoe Lee and fixed by Amaury Forgeot D'Arc; :issue:`5127`.)

* Computed gotos are now enabled by default on supported compilers (which are
  detected by the configure script).  They can still be disabled selectively by
  specifying ``--without-computed-gotos``.

  (Contributed by Antoine Pitrou; :issue:`9203`.)

* The option ``--with-wctype-functions`` was removed.  The built-in unicode
  database is now used for all functions.

  (Contributed by Amaury Forgeot D'Arc; :issue:`9210`.)

* Hash values are now values of a new type, Py_hash_t, which is defined to
  be the same size as a pointer.  Previously they were of type long, which
  on some 64-bit operating systems is still only 32 bits long.

  (Contributed by Benjamin Peterson; :issue:`9778`.)


Porting to Python 3.2
=====================

This section lists previously described changes and other bugfixes that may
require changes to your code:

* The :mod:`nntplib` module was reworked extensively, meaning that its APIs
  are often incompatible with the 3.1 APIs.

* :class:`bytearray` objects cannot be used anymore as filenames: convert them
  to :class:`bytes`.

* PyArg_Parse*() functions:

  * "t#" format has been removed: use "s#" or "s*" instead
  * "w" and "w#" formats has been removed: use "w*" instead

* The :c:type:`PyCObject` type, deprecated in 3.1, has been removed.  To wrap
  opaque C pointers in Python objects, the :c:type:`PyCapsule` API should be used
  instead; the new type has a well-defined interface for passing typing safety
  information and a less complicated signature for calling a destructor.

* mbcs encoding doesn't ignore the error handler argument anymore. By default
  (strict mode), it raises an UnicodeDecodeError on undecodable byte sequence
  and UnicodeEncodeError on unencodable character. To get the mbcs encoding of
  Python 3.1, use ``'ignore'`` error handler to decode and ``'replace'`` error
  handler to encode. mbcs now supports ``'strict'`` and ``'ignore'`` error
  handlers for decoding, and ``'strict'`` and ``'replace'`` for encoding.