Doc/whatsnew/3.2.rst

****************************
  What's New In Python 3.2
****************************

:Author: Raymond Hettinger
:Release: |release|
:Date: |today|

.. $Id$
   Rules for maintenance:

   * Anyone can add text to this document.  Do not spend very much time
   on the wording of your changes, because your text will probably
   get rewritten to some degree.

   * The maintainer will go through Misc/NEWS periodically and add
   changes; it's therefore more important to add your changes to
   Misc/NEWS than to this file.

   * This is not a complete list of every single change; completeness
   is the purpose of Misc/NEWS.  Some changes I consider too small
   or esoteric to include.  If such a change is added to the text,
   I'll just remove it.  (This is another reason you shouldn't spend
   too much time on writing your addition.)

   * If you want to draw your new text to the attention of the
   maintainer, add 'XXX' to the beginning of the paragraph or
   section.

   * It's OK to just add a fragmentary note about a change.  For
   example: "XXX Describe the transmogrify() function added to the
   socket module."  The maintainer will research the change and
   write the necessary text.

   * You can comment out your additions if you like, but it's not
   necessary (especially when a final release is some months away).

   * Credit the author of a patch or bugfix.   Just the name is
   sufficient; the e-mail address isn't necessary.

   * It's helpful to add the bug/patch number as a comment:

   % Patch 12345
   XXX Describe the transmogrify() function added to the socket
   module.
   (Contributed by P.Y. Developer.)

   This saves the maintainer the effort of going through the SVN log
   when researching a change.

This article explains the new features in Python 3.2, compared to 3.1.


PEP 3147:  PYC Repository Directories
=====================================

Python's scheme for caching bytecode in *.pyc* files did not work well in
environments with multiple python interpreters.  If one interpreter encountered
a cached file created by another interpreter, it would recompile the source and
overwrite the cached file, thus losing the benefits of caching.

The issue of "pyc fights" has become more pronounced as it has become
common-place for Linux distributions to ship with multiple versions of Python.
These conflicts also arise with CPython alternatives such as Unladen Swallow.

To solve this problem, Python's import machinery has been extended to use
distinct filenames for each interpreter.  Instead of Python3.2 and Python3.3 and
UnladenSwallow each competing for a file called "mymodule.pyc", they will now
look for "mymodule.cpython-32.pyc", "mymodule.cpython-33.pyc", and
"mymodule.unladen10.pyc".  And to keep prevent all of these new files from
cluttering source directories, the *pyc* files are now collected in a
"__pycache__" directory stored under the package directory.

Aside from the filenames and target directories, the new scheme has a few
aspects that are visible to the programmer:

* Imported modules now have a :attr:`__cached__` attribute which stores the
  name of the actual file that was imported::

   >>> import collections
   >>> collections.__cached__
   'c:/py32/lib/__pycache__/collections.cpython-32.pyc'

* The tag that is unique to each interpreter is accessible from the :mod:`imp`
  module::

   >>> import imp
   >>> imp.get_tag()
   'cpython-32'

* Scripts that try to deduce source filename from the imported file now need to
  be smarter.  It is no longer sufficient to simply strip the "c" from a ".pyc"
  filename.  Instead, use the new functions in the :mod:`imp` module:

   >>> imp.source_from_cache('c:/py32/lib/__pycache__/collections.cpython-32.pyc')
   'c:/py32/lib/collections.py'
   >>> imp.cache_from_source('c:/py32/lib/collections.py')
   'c:/py32/lib/__pycache__/collections.cpython-32.pyc'

* The :mod:`py_compile` and :mod:`compileall` modules have been updated to
  reflect the new naming convention and target directory.

.. seealso::

   :pep:`3147` - PYC Repository Directories
      PEP written by Barry Warsaw.

PEP 3149 ABI Version Tagged .so Files
=====================================

The PYC repository directory allows multiple bytecode cache files to be
co-located.  This PEP implements a similar mechanism for shared object files by
giving them a common directory and distinct names for each version.

The common directory is "pyshared" and the file names are made distinct by
identifying the Python implementation (such as CPython, PyPy, Jython, etc.), the
major and minor version numbers, and optional build flags (such as "d" for
debug, "m" for pymalloc, "u" for wide-unicode).  For an arbtrary package, "foo",
you may see these files when the distribution package is installed::

   /usr/share/pyshared/foo.cpython-32m.so
   /usr/share/pyshared/foo.cpython-33md.so

In Python itself, the tags are accessible from functions in the :mod:`sysconfig`
module::

   >>> import sysconfig
   >>> sysconfig.get_config_var('SOABI')    # find the version tag
   'cpython-32mu'
   >>> sysconfig.get_config_var('SO')       # find the full filename extension
   'cpython-32mu.so'

.. seealso::

   :pep:`3149` - ABI Version Tagged .so Files
      PEP written by Barry Warsaw.


Other Language Changes
======================

Some smaller changes made to the core Python language are:

* Stub


New, Improved, and Deprecated Modules
=====================================

* The :mod:`functools` module now includes a new decorator for caching
  function calls. :func:`functools.lru_cache` can save repeated queries to an
  external resource whenever the results are expected to be the same.

  For example, adding a caching decorator to a database query function can save
  database accesses for popular searches::

      @functools.lru_cache(maxsize=300)
      def get_phone_number(name):
          c = conn.cursor()
          c.execute('SELECT phonenumber FROM phonelist WHERE name=?', (name,))
          return c.fetchone()[0]

  To help with choosing an effective cache size, the wrapped function is
  instrumented with two attributes *cache_hits* and *cache_misses*::

        >>> for name in user_requests:
        ...     get_phone_number(name)
        >>> print(get_phone_number.cache_hits, get_phone_number.cache_misses)
        4805 980

  If the phonelist table gets updated, the outdated contents of the cache can be
  cleared with::

        >>> get_phone_number.cache_clear()

  (Contributed by Raymond Hettinger)

* The previously deprecated :func:`contextlib.nested` function has been
  removed in favor of a plain :keyword:`with` statement which can
  accept multiple context managers.  The latter technique is faster
  (because it is built-in), and it does a better job finalizing multiple
  context managers when one of them raises an exception.

  (Contributed by Georg Brandl and Mattias Brändström;
  `appspot issue 53094 <http://codereview.appspot.com/53094>`_.)

* The :class:`ftplib.FTP` class now supports the context manager protocol
  (Contributed by Tarek Ziadé and Giampaolo Rodolà; :issue:`4972`.)

* A warning message will now get printed at interpreter shutdown if
  the :data:`gc.garbage` list isn't empty.  This is meant to make the
  programmer aware that his code contains object finalization issues.
  (Added by Antoine Pitrou; :issue:`477863`.)

* The :mod:`os` module now has the :const:`ST_RDONLY` and :const:`ST_NOSUID`
  constants, for use with the :func:`~os.statvfs` function.
  (Patch by Adam Jackson; :issue:`7647`.)

* The :func:`shutil.copytree` function has two new options:

  * *ignore_dangling_symlinks*: when ``symlinks=False`` (meaning that the
    function copies the file pointed to by the symlink, not the symlink
    itself) this option will silence the error raised if the file doesn't
    exist.

  * *copy_function*: a callable that will be used to copy files.
    :func:`shutil.copy2` is used by default.

  (Contributed by Tarek Ziadé.)

* Socket objects now have a :meth:`~socket.socket.detach()` method which
  puts the socket into closed state without actually closing the underlying
  file descriptor.  The latter can then be reused for other purposes.

  (Added by Antoine Pitrou; :issue:`8524`.)

* The :mod:`sqlite3` module has some new features:

  * XXX *enable_load_extension*

  * XXX *load_extension*

  * New :class:`~sqlite3.Connection` attribute
    :attr:`~sqlite3.Connection.in_transaction` is :const:`True` when there
    are uncommitted changes, and :const:`False` otherwise.  (Contributed
    by R. David Murray and Shashwat Anand, :issue:`8845`.)

* The :mod:`ssl` module has a new class, :class:`~ssl.SSLContext` which
  serves as a container for various persistent SSL data, such as protocol
  settings, certificates, private keys, and various other options.
  The :meth:`~ssl.SSLContext.wrap_socket` method allows to create an
  SSL socket from such an SSL context.
  (Added by Antoine Pitrou; :issue:`8550`.)

  The :func:`ssl.wrap_socket` constructor function now takes a
  *ciphers* argument that's a string listing the encryption algorithms
  to be allowed; the format of the string is described
  `in the OpenSSL documentation
  <http://www.openssl.org/docs/apps/ciphers.html#CIPHER_LIST_FORMAT>`__.
  (Added by Antoine Pitrou; :issue:`8322`.)

  Various options have been added to the :mod:`ssl` module, such as
  :data:`~ssl.OP_NO_SSLv2` which allows to force disabling of the insecure
  and obsolete SSLv2 protocol.
  (Added by Antoine Pitrou; :issue:`4870`.)

  Another change makes the extension load all of OpenSSL's ciphers and
  digest algorithms so that they're all available.  Some SSL
  certificates couldn't be verified, reporting an "unknown algorithm"
  error.  (Reported by Beda Kosata, and fixed by Antoine Pitrou;
  :issue:`8484`.)

  The version of OpenSSL being used is now available as the module
  attributes :data:`ssl.OPENSSL_VERSION` (a string),
  :data:`ssl.OPENSSL_VERSION_INFO` (a 5-tuple), and
  :data:`ssl.OPENSSL_VERSION_NUMBER` (an integer).  (Added by Antoine
  Pitrou; :issue:`8321`.)

* The previously deprecated :func:`string.maketrans` function has been
  removed in favor of the static methods, :meth:`bytes.maketrans` and
  :meth:`bytearray.maketrans`.  This change solves the confusion around which
  types were supported by the :mod:`string` module. Now, :class:`str`,
  :class:`bytes`, and :class:`bytearray` each have their own **maketrans** and
  **translate** methods with intermediate translation tables of the
  appropriate type.

  (Contributed by Georg Brandl; :issue:`5675`.)

* Parameters passed to :func:`socket.getaddrinfo()` function can now be
  specified as single keyword arguments.

  (Contributed by Giampaolo Rodolà; :issue:`8866`.)

* :class:`~poplib.POP3_SSL` class now accepts a *context* parameter, which is a
  :class:`ssl.SSLContext` object allowing bundling SSL configuration options,
  certificates and private keys into a single (potentially long-lived)
  structure.

  (Contributed by Giampaolo Rodolà; :issue:`8807`.)

Multi-threading
===============

* The mechanism for serializing execution of concurrently running Python
  threads (generally known as the GIL or Global Interpreter Lock) has been
  rewritten.  Among the objectives were more predictable switching intervals
  and reduced overhead due to lock contention and the number of ensuing
  system calls.  The notion of a "check interval" to allow thread switches
  has been abandoned and replaced by an absolute duration expressed in
  seconds.  This parameter is tunable through :func:`sys.setswitchinterval()`.
  It currently defaults to 5 milliseconds.

  Additional details about the implementation can be read from a `python-dev
  mailing-list message
  <http://mail.python.org/pipermail/python-dev/2009-October/093321.html>`_
  (however, "priority requests" as exposed in this message have not been
  kept for inclusion).

  (Contributed by Antoine Pitrou.)

* Recursive locks (created with the :func:`threading.RLock` API) now benefit
  from a C implementation which makes them as fast as regular locks, and
  between 10x and 15x faster than their previous pure Python implementation.

  (Contributed by Antoine Pitrou; :issue:`3001`.)

* Regular and recursive locks now accept an optional *timeout* argument
  to their ``acquire`` method. (Contributed by Antoine Pitrou; :issue:`7316`)
  Similarly, :meth:`threading.Semaphore.acquire` also gains a *timeout*
  argument. (Contributed by Torsten Landschoff; :issue:`850728`.)


Optimizations
=============

Major performance enhancements have been added:

* Stub


Filenames and unicode
=====================

The filesystem encoding can be specified by setting the
:envvar:`PYTHONFSENCODING` environment variable before running the interpreter.
The value is an encoding name, e.g. ``iso-8859-1``. This variable is not
available (ignored) on Windows and Mac OS X: the filesystem encoding is pinned
to ``'mbcs'`` on Windows and ``'utf-8'`` on Mac OS X.

The :mod:`os` module has two new functions: :func:`os.fsencode` and
:func:`os.fsdecode`.


IDLE
====

* Stub


Build and C API Changes
=======================

Changes to Python's build process and to the C API include:

* Stub


Porting to Python 3.2
=====================

This section lists previously described changes and other bugfixes
that may require changes to your code:

* bytearray objects cannot be used anymore as filenames: convert them to bytes

* PyArg_Parse*() functions:

  * "t#" format has been removed: use "s#" or "s*" instead
  * "w" and "w#" formats has been removed: use "w*" instead

* The :ctype:`PyCObject` type, deprecated in 3.1, has been removed.  To wrap
  opaque C pointers in Python objects, the :ctype:`PyCapsule` API should be
  used instead; the new type has a well defined interface for passing typing
  safety information and a less complicated signature for calling a destructor.