| **************************** |
| What's New In Python 3.4 |
| **************************** |
| |
| .. :Author: Someone <email> |
| (uncomment if there is a principal author) |
| |
| .. Rules for maintenance: |
| |
| * Anyone can add text to this document, but the maintainer reserves the |
| right to rewrite any additions. In particular, for obscure or esoteric |
| features, the maintainer may reduce any addition to a simple reference to |
| the new documentation rather than explaining the feature inline. |
| |
| * While the maintainer will periodically go through Misc/NEWS |
| and add changes, it's best not to rely on this. We know from experience |
| that any changes that aren't in the What's New documentation around the |
| time of the original release will remain largely unknown to the community |
| for years, even if they're added later. We also know from experience that |
| other priorities can arise, and the maintainer will run out of time to do |
| updates - in such cases, end users will be much better served by partial |
| notifications that at least give a hint about new features to |
| investigate. |
| |
| * This is not a complete list of every single change; completeness |
| is the purpose of Misc/NEWS. The What's New should focus on changes that |
| are visible to Python *users* and that *require* a feature release (i.e. |
| most bug fixes should only be recorded in Misc/NEWS) |
| |
| * PEPs should not be marked Final until they have an entry in What's New. |
| A placeholder entry that is just a section header and a link to the PEP |
| (e.g ":pep:`397` has been implemented") is acceptable. If a PEP has been |
| implemented and noted in What's New, don't forget to mark it as Final! |
| |
| * If you want to draw your new text to the attention of the |
| maintainer, add 'XXX' to the beginning of the paragraph or |
| section. |
| |
| * It's OK to add just a very brief note about a change. For |
| example: "The :ref:`~socket.transmogrify()` function was added to the |
| :mod:`socket` module." The maintainer will research the change and |
| write the necessary text (if appropriate). The advantage of doing this |
| is that even if no more descriptive text is ever added, readers will at |
| least have a notification that the new feature exists and a link to the |
| relevant documentation. |
| |
| * You can comment out your additions if you like, but it's not |
| necessary (especially when a final release is some months away). |
| |
| * Credit the author of a patch or bugfix. Just the name is |
| sufficient; the e-mail address isn't necessary. |
| |
| * It's helpful to add the bug/patch number as a comment: |
| |
| The :ref:`~socket.transmogrify()` function was added to the |
| :mod:`socket` module. (Contributed by P.Y. Developer in :issue:`12345`.) |
| |
| This saves the maintainer the effort of going through the Mercurial log |
| when researching a change. |
| |
| * Cross referencing tip: :ref:`mod.attr` will display as ``mod.attr``, |
| while :ref:`~mod.attr` will display as ``attr``. |
| |
| This article explains the new features in Python 3.4, compared to 3.3. |
| |
| .. Python 3.4 was released on TBD. |
| |
| For full details, see the |
| `changelog <http://docs.python.org/3.4/whatsnew/changelog.html>`_. |
| |
| .. note:: Prerelease users should be aware that this document is currently in |
| draft form. It will be updated substantially as Python 3.4 moves towards |
| release, so it's worth checking back even after reading earlier versions. |
| |
| |
| .. seealso:: |
| |
| :pep:`429` - Python 3.4 Release Schedule |
| |
| |
| Summary -- Release highlights |
| ============================= |
| |
| .. This section singles out the most important changes in Python 3.4. |
| Brevity is key. |
| |
| New syntax features: |
| |
| * No new syntax features were added in Python 3.4. |
| |
| New expected features for Python implementations: |
| |
| * :ref:`Make newly created file descriptors non-inheritable <whatsnew-pep-446>` |
| (:pep:`446`). |
| * command line option for :ref:`isolated mode <using-on-misc-options>`, |
| (:issue:`16499`). |
| * :ref:`improvements <codec-handling-improvements>` in the handling of |
| codecs that are not text encodings |
| * :ref:`A ModuleSpec Type <whatsnew-pep-451>` for the Import System |
| (:pep:`451`). |
| |
| New library modules: |
| |
| * :mod:`asyncio`: New provisonal API for asynchronous IO (:pep:`3156`). |
| * :mod:`enum`: Support for enumeration types (:pep:`435`). |
| * :mod:`ensurepip`: Bootstrapping the pip installer (:pep:`453`). |
| * :mod:`pathlib`: Object-oriented filesystem paths (:pep:`428`). |
| * :mod:`selectors`: High-level and efficient I/O multiplexing, built upon the |
| :mod:`select` module primitives. |
| * :mod:`statistics`: A basic numerically stable statistics library (:pep:`450`). |
| * :mod:`tracemalloc`: Trace Python memory allocations (:pep:`454`). |
| |
| Significantly Improved Library Modules: |
| |
| * :ref:`Single-dispatch generic functions <whatsnew-singledispatch>` in |
| :mod:`functools` (:pep:`443`) |
| * New :mod:`pickle` :ref:`protocol 4 <whatsnew-protocol-4>` (:pep:`3154`) |
| * :ref:`SHA-3 (Keccak) support <whatsnew-sha3>` for :mod:`hashlib`. |
| * :ref:`TLSv1.1 and TLSv1.2 support <whatsnew-tls-11-12>` for :mod:`ssl`. |
| * :mod:`multiprocessing` now has :ref:`an option to avoid using os.fork |
| on Unix <whatsnew-multiprocessing-no-fork>` (:issue:`8713`). |
| * :mod:`email` has a new submodule, :mod:`~email.contentmanager`, and |
| a new :mod:`~email.message.Message` subclass |
| (:class:`~email.contentmanager.EmailMessage`) that :ref:`simplify MIME |
| handling <whatsnew_email_contentmanager>`. |
| |
| CPython implementation improvements: |
| |
| * :ref:`Safe object finalization <whatsnew-pep-442>` (:pep:`442`). |
| * Leveraging :pep:`442`, :ref:`module globals are no longer set to None |
| during finalization <whatsnew-pep-442>`, in most cases (:issue:`18214`). |
| * :ref:`Configurable memory allocators <whatsnew-pep-445>` (:pep:`445`). |
| * :ref:`Secure and interchangeable hash algorithm <whatsnew-pep-456>` |
| (:pep:`456`). |
| * :ref:`Argument Clinic <whatsnew-pep-436>` (:pep:`436`). |
| * The :mod:`marshal` format has been made :ref:`more compact and efficient |
| <whatsnew-marshal-3>` (:issue:`16475`). |
| |
| Please read on for a comprehensive list of user-facing changes. |
| |
| |
| New expected features for Python implementations |
| ================================================ |
| |
| PEP 453: Explicit bootstrapping of pip in Python installations |
| -------------------------------------------------------------- |
| |
| The new :mod:`ensurepip` module (defined in :pep:`453`) provides a standard |
| cross-platform mechanism to boostrap the pip installer into Python |
| installations and virtual environments. |
| |
| The :mod:`venv` module and the :command:`pyvenv` utility make use of this |
| module to make ``pip`` readily available in virtual environments. When |
| using the command line interface, ``pip`` is installed by default, while |
| for the module API installation of ``pip`` must be requested explicitly. |
| |
| For CPython source builds on POSIX systems, the ``make install`` and |
| ``make altinstall`` commands bootstrap ``pip`` by default. This behaviour |
| can be controlled through configure options, and overridden through |
| Makefile options. |
| |
| On Windows and Mac OS X, the CPython installers now offer the option to |
| install ``pip`` along with CPython itself. |
| |
| .. note:: |
| |
| The implementation of PEP 453 is still a work in progress. Refer to |
| :issue:`19347` for the progress on additional steps: |
| |
| * Having the binary installers install ``pip`` by default |
| * Recommending the use of ``pip`` in the "Installing Python Module" |
| documentation. |
| |
| .. seealso:: |
| |
| :pep:`453` - Explicit bootstrapping of pip in Python installations |
| PEP written by Donald Stufft and Nick Coghlan, implemented by |
| Donald Stufft, Nick Coghlan, Martin von Löwis and Ned Deily. |
| |
| |
| .. _whatsnew-pep-446: |
| |
| PEP 446: Make newly created file descriptors non-inheritable |
| ------------------------------------------------------------ |
| |
| :pep:`446` makes newly created file descriptors :ref:`non-inheritable |
| <fd_inheritance>`. New functions and methods: |
| |
| * :func:`os.get_inheritable`, :func:`os.set_inheritable` |
| * :func:`os.get_handle_inheritable`, :func:`os.set_handle_inheritable` |
| * :meth:`socket.socket.get_inheritable`, :meth:`socket.socket.set_inheritable` |
| |
| .. seealso:: |
| |
| :pep:`446` - Make newly created file descriptors non-inheritable |
| PEP written and implemented by Victor Stinner. |
| |
| |
| .. _codec-handling-improvements: |
| |
| Improvements to codec handling |
| ------------------------------ |
| |
| Since it was first introduced, the :mod:`codecs` module has always been |
| intended to operate as a type-neutral dynamic encoding and decoding |
| system. However, its close coupling with the Python text model, especially |
| the type restricted convenience methods on the builtin :class:`str`, |
| :class:`bytes` and :class:`bytearray` types, has historically obscured that |
| fact. |
| |
| As a key step in clarifying the situation, the :meth:`codecs.encode` and |
| :meth:`codecs.decode` convenience functions are now properly documented in |
| Python 2.7, 3.3 and 3.4. These functions have existed in the :mod:`codecs` |
| module (and have been covered by the regression test suite) since Python 2.4, |
| but were previously only discoverable through runtime introspection. |
| |
| Unlike the convenience methods on :class:`str`, :class:`bytes` and |
| :class:`bytearray`, these convenience functions support arbitrary codecs |
| in both Python 2 and Python 3, rather than being limited to Unicode text |
| encodings (in Python 3) or ``basestring`` <-> ``basestring`` conversions |
| (in Python 2). |
| |
| In Python 3.4, the interpreter is able to identify the known non-text |
| encodings provided in the standard library and direct users towards these |
| general purpose convenience functions when appropriate:: |
| |
| >>> b"abcdef".decode("hex") |
| Traceback (most recent call last): |
| File "<stdin>", line 1, in <module> |
| LookupError: 'hex' is not a text encoding; use codecs.decode() to handle arbitrary codecs |
| |
| >>> "hello".encode("rot13") |
| Traceback (most recent call last): |
| File "<stdin>", line 1, in <module> |
| LookupError: 'rot13' is not a text encoding; use codecs.encode() to handle arbitrary codecs |
| |
| In a related change, whenever it is feasible without breaking backwards |
| compatibility, exceptions raised during encoding and decoding operations |
| will be wrapped in a chained exception of the same type that mentions the |
| name of the codec responsible for producing the error:: |
| |
| >>> import codecs |
| |
| >>> codecs.decode(b"abcdefgh", "hex") |
| binascii.Error: Non-hexadecimal digit found |
| |
| The above exception was the direct cause of the following exception: |
| |
| Traceback (most recent call last): |
| File "<stdin>", line 1, in <module> |
| binascii.Error: decoding with 'hex' codec failed (Error: Non-hexadecimal digit found) |
| |
| >>> codecs.encode("hello", "bz2") |
| TypeError: 'str' does not support the buffer interface |
| |
| The above exception was the direct cause of the following exception: |
| |
| Traceback (most recent call last): |
| File "<stdin>", line 1, in <module> |
| TypeError: encoding with 'bz2' codec failed (TypeError: 'str' does not support the buffer interface) |
| |
| Finally, as the examples above show, these improvements have permitted |
| the restoration of the convenience aliases for the non-Unicode codecs that |
| were themselves restored in Python 3.2. This means that encoding binary data |
| to and from its hexadecimal representation (for example) can now be written |
| as:: |
| |
| >>> from codecs import encode, decode |
| >>> encode(b"hello", "hex") |
| b'68656c6c6f' |
| >>> decode(b"68656c6c6f", "hex") |
| b'hello' |
| |
| The binary and text transforms provided in the standard library are detailed |
| in :ref:`binary-transforms` and :ref:`text-transforms`. |
| |
| (Contributed by Nick Coghlan in :issue:`7475`, , :issue:`17827`, |
| :issue:`17828` and :issue:`19619`) |
| |
| .. _whatsnew-pep-451: |
| |
| PEP 451: A ModuleSpec Type for the Import System |
| ------------------------------------------------ |
| |
| :pep:`451` provides an encapsulation of the information about a module |
| that the import machinery will use to load it, (i.e. a module spec). |
| This helps simplify both the import implementation and several |
| import-related APIs. The change is also a stepping stone for several |
| future import-related improvements. |
| |
| https://mail.python.org/pipermail/python-dev/2013-November/130111.html |
| |
| The public-facing changes from the PEP are entirely backward-compatible. |
| Furthermore, they should be transparent to everyone but importer |
| authors. Key finder and loader methods have been deprecated, but they |
| will continue working. New importers should use the new methods |
| described in the PEP. Existing importers should be updated to implement |
| the new methods. |
| |
| |
| Other Language Changes |
| ---------------------- |
| |
| Some smaller changes made to the core Python language are: |
| |
| * Unicode database updated to UCD version 6.3. |
| |
| * :func:`min` and :func:`max` now accept a *default* argument that can be used |
| to specify the value they return if the iterable they are evaluating has no |
| elements. Contributed by Julian Berman in :issue:`18111`. |
| |
| * Module objects are now :mod:`weakref`'able. |
| |
| * Module ``__file__`` attributes (and related values) should now always |
| contain absolute paths by default, with the sole exception of |
| ``__main__.__file__`` when a script has been executed directly using |
| a relative path (Contributed by Brett Cannon in :issue:`18416`). |
| |
| * Now all the UTF-\* codecs (except UTF-7) reject surrogates during both |
| encoding and decoding unless the ``surrogatepass`` error handler is used, |
| with the exception of the UTF-16 decoder that accepts valid surrogate pairs, |
| and the UTF-16 encoder that produces them while encoding non-BMP characters. |
| Contributed by Victor Stinner, Kang-Hao (Kenny) Lu and Serhiy Storchaka in |
| :issue:`12892`. |
| |
| |
| New Modules |
| =========== |
| |
| |
| asyncio |
| ------- |
| |
| The new :mod:`asyncio` module (defined in :pep:`3156`) provides a standard |
| pluggable event loop model for Python, providing solid asynchronous IO |
| support in the standard library, and making it easier for other event loop |
| implementations to interoperate with the standard library and each other. |
| |
| For Python 3.4, this module is considered a :term:`provisional API`. |
| |
| .. seealso:: |
| |
| :pep:`3156` - Asynchronous IO Support Rebooted: the "asyncio" Module |
| PEP written and implementation led by Guido van Rossum. |
| |
| enum |
| ---- |
| |
| The new :mod:`enum` module (defined in :pep:`435`) provides a standard |
| implementation of enumeration types, allowing other modules (such as |
| :mod:`socket`) to provide more informative error messages and better |
| debugging support by replacing opaque integer constants with backwards |
| compatible enumeration values. |
| |
| .. seealso:: |
| |
| :pep:`435` - Adding an Enum type to the Python standard library |
| PEP written by Barry Warsaw, Eli Bendersky and Ethan Furman, |
| implemented by Ethan Furman. |
| |
| |
| pathlib |
| ------- |
| |
| The new :mod:`pathlib` module offers classes representing filesystem paths |
| with semantics appropriate for different operating systems. Path classes are |
| divided between *pure paths*, which provide purely computational operations |
| without I/O, and *concrete paths*, which inherit from pure paths but also |
| provide I/O operations. |
| |
| For Python 3.4, this module is considered a :term:`provisional API`. |
| |
| .. seealso:: |
| |
| :pep:`428` - The pathlib module -- object-oriented filesystem paths |
| PEP written and implemented by Antoine Pitrou. |
| |
| |
| selectors |
| --------- |
| |
| The new :mod:`selectors` module (created as part of implementing :pep:`3156`) |
| allows high-level and efficient I/O multiplexing, built upon the |
| :mod:`select` module primitives. |
| |
| |
| statistics |
| ---------- |
| |
| The new :mod:`statistics` module (defined in :pep:`450`) offers some core |
| statistics functionality directly in the standard library. This module |
| supports calculation of the mean, median, mode, variance and standard |
| deviation of a data series. |
| |
| .. seealso:: |
| |
| :pep:`450` - Adding A Statistics Module To The Standard Library |
| PEP written and implemented by Steven D'Aprano |
| |
| |
| tracemalloc |
| ----------- |
| |
| The new :mod:`tracemalloc` module (defined in :pep:`454`) is a debug tool to |
| trace memory blocks allocated by Python. It provides the following information: |
| |
| * Traceback where an object was allocated |
| * Statistics on allocated memory blocks per filename and per line number: |
| total size, number and average size of allocated memory blocks |
| * Compute the differences between two snapshots to detect memory leaks |
| |
| .. seealso:: |
| |
| :pep:`454` - Add a new tracemalloc module to trace Python memory allocations |
| PEP written and implemented by Victor Stinner |
| |
| |
| Improved Modules |
| ================ |
| |
| aifc |
| ---- |
| |
| The :meth:`~aifc.getparams` method now returns a namedtuple rather than a |
| plain tuple. (Contributed by Claudiu Popa in :issue:`17818`.) |
| |
| |
| audioop |
| ------- |
| |
| Added support for 24-bit samples (:issue:`12866`). |
| |
| Added the :func:`~audioop.byteswap` function to convert big-endian samples |
| to little-endian and vice versa (:issue:`19641`). |
| |
| |
| base64 |
| ------ |
| |
| The encoding and decoding functions in :mod:`base64` now accept any |
| :term:`bytes-like object` in cases where it previously required a |
| :class:`bytes` or :class:`bytearray` instance (:issue:`17839`). |
| |
| |
| colorsys |
| -------- |
| |
| The number of digits in the coefficients for the RGB --- YIQ conversions have |
| been expanded so that they match the FCC NTSC versions. The change in |
| results should be less than 1% and may better match results found elsewhere. |
| |
| |
| contextlib |
| ---------- |
| |
| The new :class:`contextlib.suppress` context manager helps to clarify the |
| intent of code that deliberately suppresses exceptions from a single |
| statement. (Contributed by Raymond Hettinger in :issue:`15806` and |
| Zero Piraeus in :issue:`19266`) |
| |
| The new :func:`contextlib.redirect_stdout` context manager makes it easier |
| for utility scripts to handle inflexible APIs that don't provide any |
| options to retrieve their output as a string or direct it to somewhere |
| other than :data:`sys.stdout`. In conjunction with :class:`io.StringIO`, |
| this context manager is also useful for checking expected output from |
| command line utilities. (Contribute by Raymond Hettinger in :issue:`15805`) |
| |
| The :mod:`contextlib` documentation has also been updated to include a |
| :ref:`discussion <single-use-reusable-and-reentrant-cms>` of the |
| differences between single use, reusable and reentrant context managers. |
| |
| |
| dis |
| --- |
| |
| The :mod:`dis` module is now built around an :class:`~dis.Instruction` class |
| that provides details of individual bytecode operations and a |
| :func:`~dis.get_instructions` iterator that emits the Instruction stream for a |
| given piece of Python code. The various display tools in the :mod:`dis` |
| module have been updated to be based on these new components. |
| |
| The new :class:`dis.Bytecode` class provides an object-oriented API for |
| inspecting bytecode, both in human-readable form and for iterating over |
| instructions. |
| |
| (Contributed by Nick Coghlan, Ryan Kelly and Thomas Kluyver in :issue:`11816` |
| and Claudiu Popa in :issue:`17916`) |
| |
| |
| doctest |
| ------- |
| |
| Added :data:`~doctest.FAIL_FAST` flag to halt test running as soon as the first |
| failure is detected. (Contributed by R. David Murray and Daniel Urban in |
| :issue:`16522`.) |
| |
| Updated the doctest command line interface to use :mod:`argparse`, and added |
| ``-o`` and ``-f`` options to the interface. ``-o`` allows doctest options to |
| be specified on the command line, and ``-f`` is a shorthand for ``-o |
| FAIL_FAST`` (to parallel the similar option supported by the :mod:`unittest` |
| CLI). (Contributed by R. David Murray in :issue:`11390`.) |
| |
| |
| email |
| ----- |
| |
| :meth:`~email.message.Message.as_string` now accepts a *policy* argument to |
| override the default policy of the message when generating a string |
| representation of it. This means that ``as_string`` can now be used in more |
| circumstances, instead of having to create and use a :mod:`~email.generator` in |
| order to pass formatting parameters to its ``flatten`` method. |
| |
| New method :meth:`~email.message.Message.as_bytes` added to produce a bytes |
| representation of the message in a fashion similar to how ``as_string`` |
| produces a string representation. It does not accept the *maxheaderlen* |
| argument, but does accept the *unixfrom* and *policy* arguments. The |
| :class:`~email.message.Message` :meth:`~email.message.Message.__bytes__` method |
| calls it, meaning that ``bytes(mymsg)`` will now produce the intuitive |
| result: a bytes object containing the fully formatted message. |
| |
| (Contributed by R. David Murray in :issue:`18600`.) |
| |
| .. _whatsnew_email_contentmanager: |
| |
| A pair of new subclasses of :class:`~email.message.Message` have been added, |
| along with a new sub-module, :mod:`~email.contentmanager`. All documentation |
| is currently in the new module, which is being added as part of the new |
| :term:`provisional <provisional package>` email API. These classes provide a |
| number of new methods that make extracting content from and inserting content |
| into email messages much easier. See the :mod:`~email.contentmanager` |
| documentation for details. |
| |
| These API additions complete the bulk of the work that was planned as part of |
| the email6 project. The currently provisional API is scheduled to become final |
| in Python 3.5 (possibly with a few minor additions in the area of error |
| handling). |
| |
| (Contributed by R. David Murray in :issue:`18891`.) |
| |
| |
| functools |
| --------- |
| |
| The new :func:`~functools.partialmethod` descriptor bring partial argument |
| application to descriptors, just as :func:`~functools.partial` provides |
| for normal callables. The new descriptor also makes it easier to get |
| arbitrary callables (including :func:`~functools.partial` instances) |
| to behave like normal instance methods when included in a class definition. |
| |
| (Contributed by Alon Horev and Nick Coghlan in :issue:`4331`) |
| |
| .. _whatsnew-singledispatch: |
| |
| The new :func:`~functools.singledispatch` decorator brings support for |
| single-dispatch generic functions to the Python standard library. Where |
| object oriented programming focuses on grouping multiple operations on a |
| common set of data into a class, a generic function focuses on grouping |
| multiple implementations of an operation that allows it to work with |
| *different* kinds of data. |
| |
| .. seealso:: |
| |
| :pep:`443` - Single-dispatch generic functions |
| PEP written and implemented by Łukasz Langa. |
| |
| |
| hashlib |
| ------- |
| |
| New :func:`hashlib.pbkdf2_hmac` function. |
| (Contributed by Christian Heimes in :issue:`18582`) |
| |
| .. _whatsnew-sha3: |
| |
| New :ref:`hash algorithms <hash-algorithms>` ``sah3_224()``, ``sha3_256()``, |
| ``sha3_384()``, and ``sha3_512()``. (Contributed by Christian Heimes in |
| :issue:`16113`.) |
| |
| |
| html |
| ---- |
| |
| Added a new :func:`html.unescape` function that converts HTML5 character |
| references to the corresponding Unicode characters. |
| (Contributed by Ezio Melotti in :issue:`2927`) |
| |
| Added a new *convert_charrefs* keyword argument to |
| :class:`~html.parser.HTMLParser` that, when ``True``, automatically converts |
| all character references. For backward-compatibility, its value defaults |
| to ``False``, but it will change to ``True`` in future versions, so you |
| are invited to set it explicitly and update your code to use this new feature. |
| (Contributed by Ezio Melotti in :issue:`13633`) |
| |
| The *strict* argument of :class:`~html.parser.HTMLParser` is now deprecated. |
| (Contributed by Ezio Melotti in :issue:`15114`) |
| |
| |
| inspect |
| ------- |
| |
| |
| The inspect module now offers a basic :ref:`command line interface |
| <inspect-module-cli>` to quickly display source code and other |
| information for modules, classes and functions. (Contributed by Claudiu Popa |
| and Nick Coghlan in :issue:`18626`) |
| |
| :func:`~inspect.unwrap` makes it easy to unravel wrapper function chains |
| created by :func:`functools.wraps` (and any other API that sets the |
| ``__wrapped__`` attribute on a wrapper function). (Contributed by |
| Daniel Urban, Aaron Iles and Nick Coghlan in :issue:`13266`) |
| |
| As part of the implementation of the new :mod:`enum` module, the |
| :mod:`inspect` module now has substantially better support for custom |
| ``__dir__`` methods and dynamic class attributes provided through |
| metaclasses (Contributed by Ethan Furman in :issue:`18929` and |
| :issue:`19030`) |
| |
| |
| .. _whatsnew-marshal-3: |
| |
| marshal |
| ------- |
| |
| The default :mod:`marshal` version has been bumped to 3. The code implementing |
| the new version restores the Python2 behavior of recording only one copy of |
| interned strings and preserving the interning on deserialization, and extends |
| this "one copy" ability to any object type (including handling recursive |
| references). This reduces both the size of ``.pyc`` files and the amount of |
| memory a module occupies in memory when it is loaded from a ``.pyc`` (or |
| ``.pyo``) file. (Contributed by Kristján Valur Jónsson in :issue:`16475`.) |
| |
| |
| mmap |
| ---- |
| |
| mmap objects can now be weakref'ed. |
| (Contributed by Valerie Lambert in :issue:`4885`.) |
| |
| |
| multiprocessing |
| --------------- |
| |
| .. _whatsnew-multiprocessing-no-fork: |
| |
| On Unix, two new :ref:`start methods <multiprocessing-start-methods>` |
| (``spawn`` and ``forkserver``) have been added for starting processes using |
| :mod:`multiprocessing`. These make the mixing of processes with threads more |
| robust, and the ``spawn`` method matches the semantics that multiprocessing has |
| always used on Windows. (Contributed by Richard Oudkerk in :issue:`8713`). |
| |
| Also, except when using the old *fork* start method, child processes |
| will no longer inherit unneeded handles/file descriptors from their parents |
| (part of :issue:`8713`). |
| |
| :mod:`multiprocessing` now relies on :mod:`runpy` (which implements the |
| ``-m`` switch) to initialise ``__main__`` appropriately in child processes |
| when using the ``spawn`` or ``forkserver`` start methods. This resolves some |
| edge cases where combining multiprocessing, the ``-m`` command line switch |
| and explicit relative imports could cause obscure failures in child |
| processes. (Contributed by Nick Coghlan in :issue:`19946`) |
| |
| |
| os |
| -- |
| |
| New functions to get and set the :ref:`inheritable flag <fd_inheritance>` of a file |
| descriptors or a Windows handle: |
| |
| * :func:`os.get_inheritable`, :func:`os.set_inheritable` |
| * :func:`os.get_handle_inheritable`, :func:`os.set_handle_inheritable` |
| |
| |
| pdb |
| --- |
| |
| The ``print`` command has been removed from :mod:`pdb`, restoring access to the |
| ``print`` function. |
| |
| Rationale: Python2's ``pdb`` did not have a ``print`` command; instead, |
| entering ``print`` executed the ``print`` statement. In Python3 ``print`` was |
| mistakenly made an alias for the pdb :pdbcmd:`p` command. ``p``, however, |
| prints the ``repr`` of its argument, not the ``str`` like the Python2 ``print`` |
| command did. Worse, the Python3 ``pdb print`` command shadowed the Python3 |
| ``print`` function, making it inaccessible at the ``pdb`` prompt. |
| |
| (Contributed by Connor Osborn in :issue:`18764`.) |
| |
| |
| .. _whatsnew-protocol-4: |
| |
| Pickle |
| ------ |
| |
| protocol 4 |
| |
| :mod:`pickle` now supports (but does not use by default) a new pickle protocol, |
| protocol 4. This new protocol addresses a number of issues that were present |
| in previous protocols, such as the serialization of nested classes, very large |
| strings and containers, or classes whose :meth:`__new__` method takes |
| keyword-only arguments. It also provides some efficiency improvements. |
| |
| .. seealso:: |
| |
| :pep:`3154` - Pickle protocol 4 |
| PEP written by Antoine Pitrou and implemented by Alexandre Vassalotti. |
| |
| |
| poplib |
| ------ |
| |
| New :meth:`~poplib.POP3.stls` method to switch a clear-text POP3 session into |
| an encrypted POP3 session. |
| |
| New :meth:`~poplib.POP3.capa` method to query the capabilities advertised by the |
| POP3 server. |
| |
| (Contributed by Lorenzo Catucci in :issue:`4473`.) |
| |
| |
| pprint |
| ------ |
| |
| The :mod:`pprint` module now supports *compact* mode for formatting long |
| sequences (:issue:`19132`). |
| |
| |
| pydoc |
| ----- |
| |
| While significant changes have not been made to :mod:`pydoc` directly, |
| its handling of custom ``__dir__`` methods and various descriptor |
| behaviours has been improved substantially by the underlying changes in |
| the :mod:`inspect` module. |
| |
| |
| re |
| -- |
| |
| Added :func:`re.fullmatch` function and :meth:`regex.fullmatch` method, |
| which anchor the pattern at both ends of the string to match. |
| (Contributed by Matthew Barnett in :issue:`16203`.) |
| |
| The repr of :ref:`regex objects <re-objects>` now includes the pattern |
| and the flags; the repr of :ref:`match objects <match-objects>` now |
| includes the start, end, and the part of the string that matched. |
| (Contributed by Serhiy Storchaka in :issue:`13592` and :issue:`17087`.) |
| |
| |
| resource |
| -------- |
| |
| New :func:`resource.prlimit` function and Linux specific constants. |
| (Contributed by Christian Heimes in :issue:`16595` and :issue:`19324`.) |
| |
| smtplib |
| ------- |
| |
| :exc:`~smtplib.SMTPException` is now a subclass of :exc:`OSError`, which allows |
| both socket level errors and SMTP protocol level errors to be caught in one |
| try/except statement by code that only cares whether or not an error occurred. |
| (:issue:`2118`). |
| |
| |
| socket |
| ------ |
| |
| Socket objects have new methods to get or set their :ref:`inheritable flag |
| <fd_inheritance>`: |
| |
| * :meth:`socket.socket.get_inheritable`, :meth:`socket.socket.set_inheritable` |
| |
| The ``socket.AF_*`` and ``socket.SOCK_*`` constants are enumeration values, |
| using the new :mod:`enum` module. This allows descriptive reporting during |
| debugging, instead of seeing integer "magic numbers". |
| |
| ssl |
| --- |
| |
| .. _whatsnew-tls-11-12: |
| |
| :data:`~ssl.PROTOCOL_TLSv1_1` and :data:`~ssl.PROTOCOL_TLSv1_2` (TLSv1.1 and |
| TLSv1.2 support) have been added; support for these protocols is only available if |
| Python is linked with OpenSSL 1.0.1 or later. (Contributed by Michele Orrù and |
| Antoine Pitrou in :issue:`16692`) |
| |
| New diagnostic functions :func:`~ssl.get_default_verify_paths`, |
| :meth:`~ssl.SSLContext.cert_store_stats` and |
| :meth:`~ssl.SSLContext.get_ca_certs` (Contributed by Christian Heimes |
| in :issue:`18143` and :issue:`18147`) |
| |
| Add :func:`ssl.enum_cert_store` to retrieve certificates and CRL from Windows' |
| cert store. (Contributed by Christian Heimes in :issue:`17134`.) |
| |
| Support for server-side SNI using the new |
| :meth:`ssl.SSLContext.set_servername_callback` method. |
| (Contributed by Daniel Black in :issue:`8109`.) |
| |
| |
| stat |
| ---- |
| |
| The :mod:`stat` module is now backed by a C implementation in :mod:`_stat`. A C |
| implementation is required as most of the values aren't standardized and |
| platform-dependent. (Contributed by Christian Heimes in :issue:`11016`.) |
| |
| The module supports new file types: door, event port and whiteout. |
| |
| |
| struct |
| ------ |
| |
| Streaming struct unpacking using :func:`struct.iter_unpack`. |
| (Contributed by Antoine Pitrou in :issue:`17804`.) |
| |
| |
| sunau |
| ----- |
| |
| The :meth:`~sunau.getparams` method now returns a namedtuple rather than a |
| plain tuple. (Contributed by Claudiu Popa in :issue:`18901`.) |
| |
| :meth:`sunau.open` now supports the context manager protocol (:issue:`18878`). |
| |
| |
| traceback |
| --------- |
| |
| A new :func:`traceback.clear_frames` function takes a traceback object |
| and clears the local variables in all of the frames it references, |
| reducing the amount of memory consumed (:issue:`1565525`). |
| |
| |
| urllib |
| ------ |
| |
| Add support.for ``data:`` URLs in :mod:`urllib.request`. |
| (Contributed by Mathias Panzenböck in :issue:`16423`.) |
| |
| |
| unittest |
| -------- |
| |
| Support for easy dynamically-generated subtests using the |
| :meth:`~unittest.TestCase.subTest` context manager. |
| (Contributed by Antoine Pitrou in :issue:`16997`.) |
| |
| |
| wave |
| ---- |
| |
| The :meth:`~wave.getparams` method now returns a namedtuple rather than a |
| plain tuple. (Contributed by Claudiu Popa in :issue:`17487`.) |
| |
| :meth:`wave.open` now supports the context manager protocol. (Contributed |
| by Claudiu Popa in :issue:`17616`.) |
| |
| |
| weakref |
| ------- |
| |
| New :class:`~weakref.WeakMethod` class simulates weak references to bound |
| methods. (Contributed by Antoine Pitrou in :issue:`14631`.) |
| |
| New :class:`~weakref.finalize` class makes it possible to register a callback |
| to be invoked when an object is garbage collected, without needing to |
| carefully manage the lifecycle of the weak reference itself. (Contributed by |
| Richard Oudkerk in :issue:`15528`) |
| |
| |
| xml.etree |
| --------- |
| |
| Add an event-driven parser for non-blocking applications, |
| :class:`~xml.etree.ElementTree.XMLPullParser`. |
| (Contributed by Antoine Pitrou in :issue:`17741`.) |
| |
| |
| zipfile.PyZipfile |
| ----------------- |
| |
| Add a filter function to ignore some packages (tests for instance), |
| :meth:`~zipfile.PyZipFile.writepy`. |
| (Contributed by Christian Tismer in :issue:`19274`.) |
| |
| |
| Other improvements |
| ================== |
| |
| Tab-completion is now enabled by default in the interactive interpreter. |
| (Contributed by Antoine Pitrou and Éric Araujo in :issue:`5845`.) |
| |
| Python invocation changes |
| ========================= |
| |
| Invoking the Python interpreter with ``--version`` now outputs the version to |
| standard output instead of standard error (:issue:`18338`). Similar changes |
| were made to :mod:`argparse` (:issue:`18920`) and other modules that have |
| script-like invocation capabilities (:issue:`18922`). |
| |
| Optimizations |
| ============= |
| |
| Major performance enhancements have been added: |
| |
| * The UTF-32 decoder is now 3x to 4x faster. |
| |
| * The cost of hash collisions for sets is now reduced. Each hash table |
| probe now checks a series of consecutive, adjacent key/hash pairs before |
| continuing to make random probes through the hash table. This exploits |
| cache locality to make collision resolution less expensive. |
| |
| The collision resolution scheme can be described as a hybrid of linear |
| probing and open addressing. The number of additional linear probes |
| defaults to nine. This can be changed at compile-time by defining |
| LINEAR_PROBES to be any value. Set LINEAR_PROBES=0 to turn-off |
| linear probing entirely. |
| |
| (Contributed by Raymond Hettinger in :issue:`18771`.) |
| |
| * The interpreter starts about 30% faster. A couple of measures lead to the |
| speedup. The interpreter loads fewer modules on startup, e.g. the :mod:`re`, |
| :mod:`collections` and :mod:`locale` modules and their dependencies are no |
| longer imported by default. The marshal module has been improved to load |
| compiled Python code faster. |
| |
| (Contributed by Antoine Pitrou, Christian Heimes and Victor Stinner in |
| :issue:`19219`, :issue:`19218`, :issue:`19209`, :issue:`19205` and |
| :issue:`9548`) |
| |
| |
| CPython Implementation Changes |
| ============================== |
| |
| |
| .. _whatsnew-pep-445: |
| |
| PEP 445: Customization of CPython memory allocators |
| --------------------------------------------------- |
| |
| :pep:`445` adds new C level interfaces to customize memory allocation in |
| the CPython interpreter. |
| |
| .. seealso:: |
| |
| :pep:`445` - Add new APIs to customize Python memory allocators |
| PEP written and implemented by Victor Stinner. |
| |
| |
| .. _whatsnew-pep-442: |
| |
| PEP 442: Safe object finalization |
| --------------------------------- |
| |
| :pep:`442` removes the current limitations and quirks of object finalization |
| in CPython. With it, objects with :meth:`__del__` methods, as well as |
| generators with :keyword:`finally` clauses, can be finalized when they are |
| part of a reference cycle. |
| |
| As part of this change, module globals are no longer forcibly set to |
| :const:`None` during interpreter shutdown in most cases, instead relying |
| on the normal operation of the cyclic garbage collector. This avoids a |
| whole class of interpreter-shutdown-time errors, usually involving |
| ``__del__`` methods, that have plagued Python since the cyclic GC |
| was first introduced. |
| |
| .. seealso:: |
| |
| :pep:`442` - Safe object finalization |
| PEP written and implemented by Antoine Pitrou. |
| |
| |
| .. _whatsnew-pep-456: |
| |
| PEP 456: Secure and Interchangeable Hash Algorithm |
| -------------------------------------------------- |
| |
| :pep:`456` follows up on earlier security fix work done on Python's hash |
| algorithm to address certain DOS attacks to which public facing APIs backed by |
| dictionary lookups may be subject. (See :issue:`14621` for the start of the |
| current round of improvements.) The PEP unifies CPython's hash code to make it |
| easier for a packager to substitute a different hash algorithm, and switches |
| Python's default implementation to a SipHash implementation on platforms that |
| have a 64 bit data type. Any performance differences in comparison with the |
| older FNV algorithm are trivial. |
| |
| The PEP adds additional fields to the :func:`sys.hash_info` struct sequence to |
| describe the hash algorithm in use by the currently executing binary. Otherwise, |
| the PEP does not alter any existing CPython APIs. |
| |
| |
| Other build and C API changes |
| ----------------------------- |
| |
| Changes to Python's build process and to the C API include: |
| |
| * The new :c:func:`Py_SetStandardStreamEncoding` pre-initialization API |
| allows applications embedding the CPython interpreter to reliably force |
| a particular encoding and error handler for the standard streams |
| (Contributed by Bastien Montagne and Nick Coghlan in :issue:`16129`) |
| |
| * Most Python C APIs that don't mutate string arguments are now correctly |
| marked as accepting ``const char *`` rather than ``char *`` (Contributed |
| by Serhiy Storchaka in :issue:`1772673`). |
| |
| .. _whatsnew-pep-436: |
| |
| * "Argument Clinic" (:pep:`436`) is now part of the CPython build process |
| and can be used to simplify the process of defining and maintaining |
| accurate signatures for builtins and standard library extension modules |
| implemented in C. |
| |
| .. note:: |
| The Argument Clinic PEP is not fully up to date with the state of the |
| implementation. This has been deemed acceptable by the release manager |
| and core development team in this case, as Argument Clinic will not |
| be made available as a public API for third party use in Python 3.4. |
| |
| |
| Deprecated |
| ========== |
| |
| Unsupported Operating Systems |
| ----------------------------- |
| |
| * OS/2 |
| * Windows 2000 |
| * VMS |
| |
| |
| Deprecated Python modules, functions and methods |
| ------------------------------------------------ |
| |
| * :meth:`difflib.SequenceMatcher.isbjunk` and |
| :meth:`difflib.SequenceMatcher.isbpopular` were removed: use ``x in sm.bjunk`` and |
| ``x in sm.bpopular``, where *sm* is a :class:`~difflib.SequenceMatcher` object. |
| |
| * :func:`importlib.util.module_for_loader` is pending deprecation. Using |
| :func:`importlib.util.module_to_load` and |
| :meth:`importlib.abc.Loader.init_module_attrs` allows subclasses of a loader |
| to more easily customize module loading. |
| |
| * The :mod:`imp` module is pending deprecation. To keep compatibility with |
| Python 2/3 code bases, the module's removal is currently not scheduled. |
| |
| * The :mod:`formatter` module is pending deprecation and is slated for removal |
| in Python 3.6. |
| |
| * MD5 as default digestmod for :mod:`hmac` is deprecated. Python 3.6 will |
| require an explicit digest name or constructor as *digestmod* argument. |
| |
| |
| Deprecated functions and types of the C API |
| ------------------------------------------- |
| |
| * The ``PyThreadState.tick_counter`` field has been removed: its value was |
| meaningless since Python 3.2 ("new GIL"). |
| |
| |
| Deprecated features |
| ------------------- |
| |
| * The site module adding a "site-python" directory to sys.path, if it |
| exists, is deprecated (:issue:`19375`). |
| |
| |
| Porting to Python 3.4 |
| ===================== |
| |
| Changes in the Python API |
| ------------------------- |
| |
| This section lists previously described changes and other bugfixes |
| that may require changes to your code. |
| |
| * The ABCs defined in :mod:`importlib.abc` now either raise the appropriate |
| exception or return a default value instead of raising |
| :exc:`NotImplementedError` blindly. This will only affect code calling |
| :func:`super` and falling through all the way to the ABCs. For compatibility, |
| catch both :exc:`NotImplementedError` or the appropriate exception as needed. |
| |
| * The module type now initializes the :attr:`__package__` and :attr:`__loader__` |
| attributes to ``None`` by default. To determine if these attributes were set |
| in a backwards-compatible fashion, use e.g. |
| ``getattr(module, '__loader__', None) is not None``. |
| |
| * :meth:`importlib.util.module_for_loader` now sets ``__loader__`` and |
| ``__package__`` unconditionally to properly support reloading. If this is not |
| desired then you will need to set these attributes manually. You can use |
| :func:`importlib.util.module_to_load` for module management. |
| |
| * Import now resets relevant attributes (e.g. ``__name__``, ``__loader__``, |
| ``__package__``, ``__file__``, ``__cached__``) unconditionally when reloading. |
| |
| * Frozen packages no longer set ``__path__`` to a list containing the package |
| name but an empty list instead. Determing if a module is a package should be |
| done using ``hasattr(module, '__path__')``. |
| |
| * :func:`py_compile.compile` now raises :exc:`FileExistsError` if the file path |
| it would write to is a symlink or a non-regular file. This is to act as a |
| warning that import will overwrite those files with a regular file regardless |
| of what type of file path they were originally. |
| |
| * :meth:`importlib.abc.SourceLoader.get_source` no longer raises |
| :exc:`ImportError` when the source code being loaded triggers a |
| :exc:`SyntaxError` or :exc:`UnicodeDecodeError`. As :exc:`ImportError` is |
| meant to be raised only when source code cannot be found but it should, it was |
| felt to be over-reaching/overloading of that meaning when the source code is |
| found but improperly structured. If you were catching ImportError before and |
| wish to continue to ignore syntax or decoding issues, catch all three |
| exceptions now. |
| |
| * :func:`functools.update_wrapper` and :func:`functools.wraps` now correctly |
| set the ``__wrapped__`` attribute to the function being wrapper, even if |
| that function also had its ``__wrapped__`` attribute set. This means |
| ``__wrapped__`` attributes now correctly link a stack of decorated |
| functions rather than every ``__wrapped__`` attribute in the chain |
| referring to the innermost function. Introspection libraries that |
| assumed the previous behaviour was intentional can use |
| :func:`inspect.unwrap` to access the first function in the chain that has |
| no ``__wrapped__`` attribute. |
| |
| * :class:`importlib.machinery.PathFinder` now passes on the current working |
| directory to objects in :data:`sys.path_hooks` for the empty string. This |
| results in :data:`sys.path_importer_cache` never containing ``''``, thus |
| iterating through :data:`sys.path_importer_cache` based on :data:`sys.path` |
| will not find all keys. A module's ``__file__`` when imported in the current |
| working directory will also now have an absolute path, including when using |
| ``-m`` with the interpreter (this does not influence when the path to a file |
| is specified on the command-line). |
| |
| Changes in the C API |
| -------------------- |
| |
| * :c:func:`PyErr_SetImportError` now sets :exc:`TypeError` when its **msg** |
| argument is not set. Previously only ``NULL`` was returned with no exception |
| set. |
| |
| * The result of the :c:data:`PyOS_ReadlineFunctionPointer` callback must |
| now be a string allocated by :c:func:`PyMem_RawMalloc` or |
| :c:func:`PyMem_RawRealloc`, or *NULL* if an error occurred, instead of a |
| string allocated by :c:func:`PyMem_Malloc` or :c:func:`PyMem_Realloc`. |
| |
| * :c:func:`PyThread_set_key_value` now always set the value. In Python |
| 3.3, the function did nothing if the key already exists (if the current |
| value is a non-NULL pointer). |
| |
| * The ``f_tstate`` (thread state) field of the :c:type:`PyFrameObject` |
| structure has been removed to fix a bug: see :issue:`14432` for the |
| rationale. |
| |