Doc/whatsnew/3.0.rst

****************************
  What's New In Python 3.0  
****************************

.. XXX add trademark info for Apple, Microsoft, SourceForge.

.. XXX turn all PEP references into :pep:`NNN` markup.

:Author: Guido van Rossum
:Release: |release|
:Date: |today|

.. $Id$
   Rules for maintenance:
   
   * Anyone can add text to this document.  Do not spend very much time
   on the wording of your changes, because your text will probably
   get rewritten to some degree.
   
   * The maintainer will go through Misc/NEWS periodically and add
   changes; it's therefore more important to add your changes to
   Misc/NEWS than to this file.
   
   * This is not a complete list of every single change; completeness
   is the purpose of Misc/NEWS.  Some changes I consider too small
   or esoteric to include.  If such a change is added to the text,
   I'll just remove it.  (This is another reason you shouldn't spend
   too much time on writing your addition.)
   
   * If you want to draw your new text to the attention of the
   maintainer, add 'XXX' to the beginning of the paragraph or
   section.
   
   * It's OK to just add a fragmentary note about a change.  For
   example: "XXX Describe the transmogrify() function added to the
   socket module."  The maintainer will research the change and
   write the necessary text.
   
   * You can comment out your additions if you like, but it's not
   necessary (especially when a final release is some months away).
   
   * Credit the author of a patch or bugfix.   Just the name is
   sufficient; the e-mail address isn't necessary.
   
   * It's helpful to add the bug/patch number as a comment:
   
   % Patch 12345
   XXX Describe the transmogrify() function added to the socket
   module.
   (Contributed by P.Y. Developer.)
   
   This saves the maintainer the effort of going through the SVN log
   when researching a change.

This article explains the new features in Python 3.0, compared to 2.6.
Python 3.0 is the first ever *intentionally incompatible* release.
There are more changes than in a typical release, and more that are
important for all Python users.  Nevertheless, after digesting the
changes, you'll find that Python really hasn't changed all that much
-- by and large, we're merely fixing well-known annoyances and warts.

This article doesn't attempt to provide a complete specification of
the new features, but instead provides a convenient overview.  For
full details, you should refer to the documentation for Python 3.0. If
you want to understand the complete implementation and design
rationale, refer to the PEP for a particular new feature.

.. Compare with previous release in 2 - 3 sentences here.
.. add hyperlink when the documentation becomes available online.

.. ======================================================================
.. Large, PEP-level features and changes should be described here.
.. Should there be a new section here for 3k migration?
.. Or perhaps a more general section describing module changes/deprecation?
.. sets module deprecated
.. ======================================================================


Common Stumbling Blocks
=======================

This section briefly lists a few changes that are more likely to trip
people up, without necessarily raising obvious errors.  Most issues
are explained in more detail in later sections.

Print Is A Function
-------------------

The ``print`` statement has been replaced with a :func:`print` function,
with keyword arguments to replace most of the special syntax of the
old ``print`` statement (PEP 3105).  Examples::

  Old: print "The answer is", 2*2
  New: print("The answer is", 2*2)

  Old: print x,           # Trailing comma suppresses newline
  New: print(x, end=" ")  # Appends a space instead of a newline

  Old: print              # Prints a newline
  New: print()            # You must call the function!

  Old: print >>sys.stderr, "fatal error"
  New: print("fatal error", file=sys.stderr)

  Old: print (x, y)       # prints repr((x, y))
  New: print((x, y))      # Not the same as print(x, y)!

You can also customize the separator between items, e.g.::

  print("There are <", 2**32, "> possibilities!", sep="")

which produces::

  There are <4294967296> possibilities!

Note:

* The :func:`print` function doesn't support the "softspace" feature of
  the old ``print`` statement.  For example, in Python 2.x,
  ``print "A\n", "B"`` would write ``"A\nB\n"``; but in Python 3.0,
  ``print("A\n", "B")`` writes ``"A\n B\n"``.

* Initially, you'll be finding yourself typing the old ``print x``
  a lot in interactive mode.  Time to retrain your fingers to type
  ``print(x)`` instead!

* When using the ``2to3`` source-to-source conversion tool, all
  ``print`` statements are automatically converted to :func:`print`
  function calls, so this is mostly a non-issue for larger projects.

Text Strings Vs. Bytes
----------------------

Everything you thought you knew about binary data and Unicode has
changed:

* Python 3.0 uses *strings* and *bytes* instead of *Unicode strings*
  and *8-bit strings*.  The difference is that any attempt to mix
  strings and bytes in Python 3.0 raises a TypeError exception,
  whereas if you were to mix Unicode and 8-bit strings in Python 2.x,
  you would only get an exception if the 8-bit string contained
  non-ASCII values.  As a consequence, pretty much all code that
  uses Unicode, encodings or binary data most likely has to change.
  The change is for the better, as in the 2.x world there were
  numerous bugs having to do with mixing encoded and unencoded text.

* Files opened as text files (still the default mode for :func:`open`)
  always use an encoding to map between strings (in memory) and bytes
  (on disk).  Binary files (opened with a ``b`` in the mode argument)
  always use bytes in memory.  This means that if a file is opened
  using an incorrect mode or encoding, I/O will likely fail.  There is
  a platform-dependent default encoding, which on Unixy platforms can
  be set with the ``LANG`` environment variable (and sometimes also
  with some other platform-specific locale-related environment
  variables).  In many cases, but not all, the system default is
  UTF-8; you should never could on this default.  Any application
  reading or writing more than pure ASCII text should probably have a
  way to override the encoding.

* XXX More below?

* See also the *Unicode HOWTO*.  (XXX How to make this a link?)
  (XXX Move to longer section below?)

Views And Interators Instead Of Lists
-------------------------------------

Some well-known APIs no longer return lists:

* :class:`dict` methods :meth:`dict.keys`, :meth:`dict.items` and
  :meth:`dict.values` return "views" instead of lists.  For example,
  this no longer works: ``k = d.keys(); k.sort()``.  Use ``k =
  sorted(d)`` instead.

* Also, the :meth:`dict.iterkeys`, :meth:`dict.iteritems` and
  :meth:`dict.itervalues` methods are no longer supported.

* :func:`map` and :func:`filter` return iterators.  A quick fix is e.g.
  ``list(map(...))``, but a better fix is often to use a list
  comprehension (especially when the original code uses :keyword:`lambda`).
  Particularly tricky is :func:`map` invoked for the side effects of the
  function; the correct transformation is to use a for-loop.

* :func:`range` now behaves like :func:`xrange` used to behave.
  The latter no longer exists.

* :func:`zip` now returns an iterator.

* XXX More below?

Ordering Comparisons
--------------------

Python 3.0 has simplified the rules for ordering comparisons:

* The ordering comparison operators (``<``, ``<=``, ``>=``, ``>``)
  raise a TypeError exception when the operands don't have a
  meaningful natural ordering.  Thus, expressions like ``1 < ''``, ``0
  > None`` or ``len <= len`` are no longer valid.  A corollary is that
  sorting a heterogeneous list no longer makes sense -- all the
  elements must be comparable to each other.  Note that this does not
  apply to the ``==`` and ``!=`` operators: objects of different
  uncomparable types always compare unequal to each other, and an
  object always compares equal to itself (i.e., ``x is y`` implies ``x
  = y``; this is true even for ``NaN``).

* :meth:`builtin.sorted` and :meth:`list.sort` no longer accept the *cmp*
  argument providing a comparison function.  Use the *key* argument
  instead. N.B. the *key* and *reverse* arguments are now "keyword-only".

* The :func:`cmp` function is gone, and the :meth:`__cmp__` special
  method is no longer supported.  Use :meth:`__lt__` for sorting,
  :meth:`__eq__` with :meth:`__hash__`, and other rich comparisons as
  needed.  if you really need the :func:`cmp` functionality, the
  expression ``(a > b) - (a < b)`` is equivalent to ``cmp(a, b)``.

* XXX More below?

Integers
--------

* We unified the :class:`int` and :class:`long` types.  All integers
  are now of type :class:`int`.

* ``1/2`` returns a float.  Use ``1//2`` to get the truncating behavior.
  (The latter syntax has existed for years, at least since Python 2.2.)

* The :func:`repr` of a long integer doesn't include the trailing ``L``
  anymore, so code that unconditionally strips that character will
  chop off the last digit instead.

* The :data:`sys.maxint` constant was removed, since there is no
  longer a limit to the value of ints.  However, :data:`sys.maxsize`
  can be used as an integer larger than any practical list or string
  index.  It conforms to the implementation's "natural" integer size
  and is typically the same as :data:`sys.maxint` in previous releases
  on the same platform (assuming the same build options).

* XXX More below?


Overview Of Syntactic Changes
=============================

This section gives a brief overview of every *syntactic* change.
Several of these are discussed at greater length later.

XXX Did I get everything?

Additions
---------

* Function argument and return value annotations (see below).  XXX

* A lone ``*`` in a formal parameter list implies that any following
  arguments *must* be specified in keyword form.  (XXX Didn't this make
  it into 2.6 as well?)

* Keyword arguments are allowed after the list of base classes in a
  class definition.  This is used by the new convention for specifying
  a metaclass, but can be used for other purposes as well, as long as
  the metaclass supports it.

* Tuple-unpacking assignment now has a *wildcard* syntax, e.g.::

    (a, b, *rest) = range(5)

  This sets *a* to 0, *b* to 1, and *rest to ``[2, 3, 4]``.

* Dictionary comprehensions: ``{k: v for k, v in stuff}`` means the
  same thing as ``dict(stuff)`` but is more flexible.

* Set literals, e.g. ``{1, 2}``.  Note that ``{}`` is an empty
  dictionary; use ``set()`` for an empty set.  Set comprehensions
  are also supported; ``{x for x in stuff}`` means the same thing
  as ``set(stuff)`` but is more flexible.

* New octal literals, e.g. ``0o720`` (already in 2.6).  The old octal
  literals (``0720`` are gone.

* New binary literals, e.g. ``0b1010`` (already in 2.6).

* Bytes literals are introduced with a leading ``b`` or ``B``.

Changes
-------

* New :keyword:`raise` statement syntax: ``raise [expr [from expr]]``.

* New keywords: :keyword:`as`, :keyword:`with` (already in 2.6),
  :keyword:`None` (partially enforced in 2.6), :keyword:`True`,
  :keyword:`False` (these were built-ins previously), and
  :keyword:`nonlocal` (for the new ``nonlocal`` statement).

* Change from ``except exc, var:`` to ``except exc as var:``.  XXX

* *Very* subtle changes in the syntax for list comprehensions,
  generator expressions, :keyword:`lambda expression and :keyword:`if`
  expressions.  For example, this is valid in Python 2.6::

     [ x for x in lambda: True, lambda: False if x() ]

  In Python 3.0 you'll have to add parentheses, like this::

     [ x for x in (lambda: True, lambda: False) if x() ]

* The *ellipsis* (``...``) can be used as an atomic expression anywhere.
  (Previously it was only allowed in slices.)

Removals
--------

* Tuple parameter unpacking removed.  XXX

* Removal of backticks.  XXX

* Removal of ``<>``.  Use ``!=`` instead.  XXX

* Removed keyword: :func:`exec` is no longer a keyword; it remains as
  a function.  (Fortunately the function syntax was also accepted in
  2.x.)

* Integer literals no longer support a trailing ``l`` or ``L``.

* String literals no longer support a leading ``u`` or ``U``.

* The *ellipsis* must now be spelled as ``...``; previously it could
  (by a mere accident of the grammar) also be spelled as ``. . .``.


Changes Already Present In Python 2.6
=====================================

This section reminds the reader of new features that were originally
designed for Python 3.0 but that were already introduced in Python
2.6.  The descriptions in "What's New in Python 2.6" are hereby
included by reference.

* XXX List each of those briefly.

Strings And Bytes
=================

This section discusses the many changes in string 

* There is only one string type; its name is :class:`str` but its behavior and
  implementation are like :class:`unicode` in 2.x.

* The :class:`basestring` superclass has been removed. The ``2to3`` tool
  replaces every occurrence of :class:`basestring` with :class:`str`.

* PEP 3137: There is a new type, :class:`bytes`, to represent binary data (and
  encoded text, which is treated as binary data until you decide to decode it).
  The :class:`str` and :class:`bytes` types cannot be mixed; you must always
  explicitly convert between them, using the :meth:`str.encode` (str -> bytes)
  or :meth:`bytes.decode` (bytes -> str) methods.

* All backslashes in raw strings are interpreted literally.  This means that
  ``'\U'`` and ``'\u'`` escapes in raw strings are not treated specially.

.. XXX add bytearray

* PEP 3112: Bytes literals, e.g. ``b"abc"``, create :class:`bytes` instances.

* PEP 3120: UTF-8 default source encoding.

* PEP 3131: Non-ASCII identifiers.  (However, the standard library remains
  ASCII-only with the exception of contributor names in comments.)

* PEP 3116: New I/O Implementation.  The API is nearly 100% backwards
  compatible, but completely reimplemented (currently mostly in Python).  Also,
  binary files use bytes instead of strings.

* The :mod:`StringIO` and :mod:`cStringIO` modules are gone.  Instead, import
  :class:`io.StringIO` or :class:`io.BytesIO`.



PEP 3101: A New Approach To String Formatting
=============================================

* A new system for  built-in string formatting operations replaces the
  ``%`` string  formatting operator.   (However, the ``%``  operator is
  still supported;  it will  be deprecated in  Python 3.1  and removed
  from the language at some later time.)

.. XXX expand this


PEP 3106: Revamping dict :meth:`dict.keys`, :meth:`dict.items` and :meth:`dict.values`
======================================================================================

.. XXX expand this (but note that the "pitfalls" section currently has
.. XXX more detail :-)

* The :meth:`dict.iterkeys`, :meth:`dict.itervalues` and :meth:`dict.iteritems`
  methods have been removed.

* :meth:`dict.keys`, :meth:`dict.values` and :meth:`dict.items` return objects
  with set behavior that reference the underlying dict; these are often
  referred to as *dictionary views*.


PEP 3107: Function Annotations
==============================

.. XXX expand this

* A standardized way of annotating a function's parameters and return values.


Exception Stuff
===============

* PEP 352: All exceptions must be derived (directly or indirectly)
  from :exc:`BaseException`.  This is the root of the exception
  hierarchy.  Most exceptions should actually be derived from
  :exc:`Exception`.  This is not a new recommendation, but the
  *requirement* to inherit from :exc:`BaseException` is new.  (Python
  2.6 still allowed classic classes to be raised, and placed no
  restriction on what you can catch.)

* :exc:`StandardError` was removed (in 2.6, actually).

* Dropping sequence behavior (slicing!) and :attr:`message` attribute of
  exception instances.

* PEP 3109: Raising exceptions.  You must now use ``raise Exception(args)``
  instead of ``raise Exception, args``.

* PEP 3110: Catching exceptions.  You must now use ``except SomeException as
  identifier:`` instead of ``except Exception, identifier:``

* PEP 3134: Exception chaining.

* A few exception messages are improved when Windows fails to load an extension
  module.  For example, ``error code 193`` is now ``%1 is not a valid Win32
  application``.  Strings now deal with non-English locales.


New Class And Metaclass Stuff
=============================

* Classic classes are gone.

* PEP 3115: New Metaclass Syntax.

* PEP 3119: Abstract Base Classes (ABCs); ``@abstractmethod`` and
  ``@abstractproperty`` decorators; collection ABCs.

* PEP 3129: Class decorators.

* PEP 3141: Numeric ABCs.


Other Language Changes
======================

Here are most of the changes that Python 3.0 makes to the core Python
language and built-in functions.

* Removed backticks (use :func:`repr` instead).

* Removed ``<>`` (use ``!=`` instead).

* ``!=`` now returns the opposite of ``==``, unless ``==`` returns
  ``NotImplemented``.

* :keyword:`as` and :keyword:`with` are keywords.

* ``True``, ``False``, and ``None`` are keywords.

* PEP 237: :class:`long` renamed to :class:`int`.  That is, there is only one
  built-in integral type, named :class:`int`; but it behaves like the old
  :class:`long` type, with the exception that the literal suffix ``L`` is
  neither supported by the parser nor produced by :func:`repr` anymore.
  :data:`sys.maxint` was also removed since the int type has no maximum value
  anymore.  Use :data:`sys.maxsize` instead.
  XXX Is this a dupe from the intro section on integers?

* PEP 238: int division returns a float.

* The ordering operators behave differently: for example, ``x < y`` where ``x``
  and ``y`` have incompatible types raises :exc:`TypeError` instead of returning
  a pseudo-random boolean.

* :meth:`__getslice__` and friends killed.  The syntax ``a[i:j]`` now translates
  to ``a.__getitem__(slice(i, j))`` (or :meth:`__setitem__` or
  :meth:`__delitem__`, depending on context).

* PEP 3102: Keyword-only arguments.  Named parameters occurring after ``*args``
  in the parameter list *must* be specified using keyword syntax in the call.
  You can also use a bare ``*`` in the parameter list to indicate that you don't
  accept a variable-length argument list, but you do have keyword-only
  arguments.

* PEP 3104: :keyword:`nonlocal` statement.  Using ``nonlocal x`` you can now
  assign directly to a variable in an outer (but non-global) scope.

* PEP 3111: :func:`raw_input` renamed to :func:`input`.  That is, the new
  :func:`input` function reads a line from :data:`sys.stdin` and returns it with
  the trailing newline stripped.  It raises :exc:`EOFError` if the input is
  terminated prematurely.  To get the old behavior of :func:`input`, use
  ``eval(input())``.

* :func:`xrange` renamed to :func:`range`, so :func:`range` will no longer
  produce a list but an iterable yielding integers when iterated over.

* PEP 3113: Tuple parameter unpacking removed.  You can no longer write ``def
  foo(a, (b, c)): ...``.  Use ``def foo(a, b_c): b, c = b_c`` instead.

* PEP 3114: ``.next()`` renamed to :meth:`__next__`, new builtin :func:`next` to
  call the :meth:`__next__` method on an object.

* PEP 3127: New octal literals; binary literals and :func:`bin`.  Instead of
  ``0666``, you write ``0o666``.  The :func:`oct` function is modified
  accordingly.  Also, ``0b1010`` equals 10, and ``bin(10)`` returns
  ``"0b1010"``.  ``0666`` is now a :exc:`SyntaxError`.

* PEP 3132: Extended Iterable Unpacking.  You can now write things like ``a, b,
  *rest = some_sequence``.  And even ``*rest, a = stuff``.  The ``rest`` object
  is always a list; the right-hand side may be any iterable.

* PEP 3135: New :func:`super`.  You can now invoke :func:`super` without
  arguments and the right class and instance will automatically be chosen.  With
  arguments, its behavior is unchanged.

* :func:`zip`, :func:`map` and :func:`filter` return iterators.

* :data:`string.letters` and its friends (:data:`string.lowercase` and
  :data:`string.uppercase`) are gone.  Use :data:`string.ascii_letters`
  etc. instead.

* Removed: :func:`apply`, :func:`callable`, :func:`coerce`, :func:`execfile`,
  :func:`file`, :func:`reduce`, :func:`reload`.

* Removed: :meth:`dict.has_key` -- use the ``in`` operator instead.

* :func:`exec` is now a function.

* The :meth:`__oct__` and :meth:`__hex__` special methods are removed --
  :func:`oct` and :func:`hex` use :meth:`__index__` now to convert the argument
  to an integer.

* Support is removed for :attr:`__members__` and :attr:`__methods__`.

* Renamed the boolean conversion C-level slot and method: ``nb_nonzero`` is now
  ``nb_bool`` and :meth:`__nonzero__` is now :meth:`__bool__`.


.. ======================================================================


Optimizations
-------------

* Detailed changes are listed here.

The net result of the 3.0 generalizations is that Python 3.0 runs the
pystone benchmark around a third slower than Python 2.5.  There's room
for improvement, but it will happen after 3.0 is released!

.. ======================================================================


New, Improved, And Deprecated Modules
=====================================

As usual, Python's standard library received a number of enhancements and bug
fixes.  Here's a partial list of the most notable changes, sorted alphabetically
by module name. Consult the :file:`Misc/NEWS` file in the source tree for a more
complete list of changes, or look through the Subversion logs for all the
details.

* The :mod:`cPickle` module is gone.  Use :mod:`pickle` instead.  Eventually
  we'll have a transparent accelerator module.

* The :mod:`imageop` module is gone.

* The :mod:`audiodev`, :mod:`Bastion`, :mod:`bsddb185`, :mod:`exceptions`,
  :mod:`linuxaudiodev`, :mod:`md5`, :mod:`MimeWriter`, :mod:`mimify`,
  :mod:`popen2`, :mod:`rexec`, :mod:`sets`, :mod:`sha`, :mod:`stringold`,
  :mod:`strop`, :mod:`sunaudiodev`, :mod:`timing`, and :mod:`xmllib` modules are
  gone.

* The :mod:`bsddb` module is gone.  It is being maintained externally
  with its own release schedule better mirroring that of BerkeleyDB.
  See http://www.jcea.es/programacion/pybsddb.htm.

* The :mod:`new` module is gone.

* The functions :func:`os.tmpnam`, :func:`os.tempnam` and :func:`os.tmpfile`
  have been removed in favor of the :mod:`tempfile` module.

* The :mod:`tokenize` module has been changed to work with bytes.  The main
  entry point is now :func:`tokenize.tokenize`, instead of generate_tokens.

.. ======================================================================
.. whole new modules get described in subsections here

.. ======================================================================


Build And C API Changes
=======================

Changes to Python's build process and to the C API include:

* PEP 3118: New Buffer API.

* PEP 3121: Extension Module Initialization & Finalization.

* PEP 3123: Making :cmacro:`PyObject_HEAD` conform to standard C.

* No more C API support for restricted execution.

* :cfunc:`PyNumber_Coerce`, :cfunc:`PyNumber_CoerceEx`, :cfunc:`PyMember_Get`,
  and :cfunc:`PyMember_Set` C APIs are removed.

* New C API :cfunc:`PyImport_ImportModuleNoBlock`, works like
  :cfunc:`PyImport_ImportModule` but won't block on the import lock (returning
  an error instead).

.. ======================================================================


Port-Specific Changes
---------------------

Platform-specific changes go here.


.. ======================================================================


.. _30section-other:

Other Changes And Fixes
=======================

As usual, there were a bunch of other improvements and bugfixes
scattered throughout the source tree.  A search through the change
logs finds there were XXX patches applied and YYY bugs fixed between
Python 2.6 and 3.0.  Both figures are likely to be underestimates.

Some of the more notable changes are:

* Details go here.

.. ======================================================================


Porting To Python 3.0
=====================

This section lists previously described changes that may require
changes to your code:

* Everything is all in the details!

* Developers can include :file:`intobject.h` after :file:`Python.h` for
  some ``PyInt_`` aliases.

* XXX Mention 2to3.

* XXX Reference external doc about porting extensions?

.. ======================================================================