| **************************** |
| What's New In Python 3.4 |
| **************************** |
| |
| :Author: R. David Murray <rdmurray@bitdance.com> (Editor) |
| |
| .. Rules for maintenance: |
| |
| * Anyone can add text to this document, but the maintainer reserves the |
| right to rewrite any additions. In particular, for obscure or esoteric |
| features, the maintainer may reduce any addition to a simple reference to |
| the new documentation rather than explaining the feature inline. |
| |
| * While the maintainer will periodically go through Misc/NEWS |
| and add changes, it's best not to rely on this. We know from experience |
| that any changes that aren't in the What's New documentation around the |
| time of the original release will remain largely unknown to the community |
| for years, even if they're added later. We also know from experience that |
| other priorities can arise, and the maintainer will run out of time to do |
| updates -- in such cases, end users will be much better served by partial |
| notifications that at least give a hint about new features to |
| investigate. |
| |
| * This is not a complete list of every single change; completeness |
| is the purpose of Misc/NEWS. The What's New should focus on changes that |
| are visible to Python *users* and that *require* a feature release (i.e. |
| most bug fixes should only be recorded in Misc/NEWS) |
| |
| * PEPs should not be marked Final until they have an entry in What's New. |
| A placeholder entry that is just a section header and a link to the PEP |
| (e.g ":pep:`397` has been implemented") is acceptable. If a PEP has been |
| implemented and noted in What's New, don't forget to mark it as Final! |
| |
| * If you want to draw your new text to the attention of the |
| maintainer, add 'XXX' to the beginning of the paragraph or |
| section. |
| |
| * It's OK to add just a very brief note about a change. For |
| example: "The :ref:`~socket.transmogrify()` function was added to the |
| :mod:`socket` module." The maintainer will research the change and |
| write the necessary text (if appropriate). The advantage of doing this |
| is that even if no more descriptive text is ever added, readers will at |
| least have a notification that the new feature exists and a link to the |
| relevant documentation. |
| |
| * You can comment out your additions if you like, but it's not |
| necessary (especially when a final release is some months away). |
| |
| * Credit the author of a patch or bugfix. Just the name is |
| sufficient; the e-mail address isn't necessary. |
| |
| * It's helpful to add the bug/patch number as a comment: |
| |
| The :ref:`~socket.transmogrify()` function was added to the |
| :mod:`socket` module. (Contributed by P.Y. Developer in :issue:`12345`.) |
| |
| This saves the maintainer the effort of going through the Mercurial log |
| when researching a change. |
| |
| * Cross referencing tip: :ref:`mod.attr` will display as ``mod.attr``, |
| while :ref:`~mod.attr` will display as ``attr``. |
| |
| This article explains the new features in Python 3.4, compared to 3.3. |
| Python 3.4 was released on March 16, 2014. For full details, see the |
| `changelog <http://docs.python.org/3.4/whatsnew/changelog.html>`_. |
| |
| |
| .. seealso:: |
| |
| :pep:`429` -- Python 3.4 Release Schedule |
| |
| |
| |
| Summary -- Release Highlights |
| ============================= |
| |
| .. This section singles out the most important changes in Python 3.4. |
| Brevity is key. |
| |
| New syntax features: |
| |
| * No new syntax features were added in Python 3.4. |
| |
| Other new features: |
| |
| * :ref:`pip should always be available <whatsnew-pep-453>` (:pep:`453`). |
| * :ref:`Newly created file descriptors are non-inheritable <whatsnew-pep-446>` |
| (:pep:`446`). |
| * command line option for :ref:`isolated mode <whatsnew-isolated-mode>` |
| (:issue:`16499`). |
| * :ref:`improvements in the handling of codecs <codec-handling-improvements>` |
| that are not text encodings (multiple issues). |
| * :ref:`A ModuleSpec Type <whatsnew-pep-451>` for the Import System |
| (:pep:`451`). (Affects importer authors.) |
| * The :mod:`marshal` format has been made :ref:`more compact and efficient |
| <whatsnew-marshal-3>` (:issue:`16475`). |
| |
| New library modules: |
| |
| * :mod:`asyncio`: :ref:`New provisional API for asynchronous IO |
| <whatsnew-asyncio>` (:pep:`3156`). |
| * :mod:`ensurepip`: :ref:`Bootstrapping the pip installer <whatsnew-ensurepip>` |
| (:pep:`453`). |
| * :mod:`enum`: :ref:`Support for enumeration types <whatsnew-enum>` |
| (:pep:`435`). |
| * :mod:`pathlib`: :ref:`Object-oriented filesystem paths <whatsnew-pathlib>` |
| (:pep:`428`). |
| * :mod:`selectors`: :ref:`High-level and efficient I/O multiplexing |
| <whatsnew-selectors>`, built upon the :mod:`select` module primitives (part |
| of :pep:`3156`). |
| * :mod:`statistics`: A basic :ref:`numerically stable statistics library |
| <whatsnew-statistics>` (:pep:`450`). |
| * :mod:`tracemalloc`: :ref:`Trace Python memory allocations |
| <whatsnew-tracemalloc>` (:pep:`454`). |
| |
| Significantly improved library modules: |
| |
| * :ref:`Single-dispatch generic functions <whatsnew-singledispatch>` in |
| :mod:`functools` (:pep:`443`). |
| * New :mod:`pickle` :ref:`protocol 4 <whatsnew-protocol-4>` (:pep:`3154`). |
| * :mod:`multiprocessing` now has :ref:`an option to avoid using os.fork |
| on Unix <whatsnew-multiprocessing-no-fork>` (:issue:`8713`). |
| * :mod:`email` has a new submodule, :mod:`~email.contentmanager`, and |
| a new :mod:`~email.message.Message` subclass |
| (:class:`~email.contentmanager.EmailMessage`) that :ref:`simplify MIME |
| handling <whatsnew_email_contentmanager>` (:issue:`18891`). |
| * The :mod:`inspect` and :mod:`pydoc` modules are now capable of |
| correct introspection of a much wider variety of callable objects, |
| which improves the output of the Python :func:`help` system. |
| * The :mod:`ipaddress` module API has been declared stable |
| |
| Security improvements: |
| |
| * :ref:`Secure and interchangeable hash algorithm <whatsnew-pep-456>` |
| (:pep:`456`). |
| * :ref:`Make newly created file descriptors non-inheritable <whatsnew-pep-446>` |
| (:pep:`446`) to avoid leaking file descriptors to child processes. |
| * New command line option for :ref:`isolated mode <whatsnew-isolated-mode>`, |
| (:issue:`16499`). |
| * :mod:`multiprocessing` now has :ref:`an option to avoid using os.fork |
| on Unix <whatsnew-multiprocessing-no-fork>`. *spawn* and *forkserver* are |
| more secure because they avoid sharing data with child processes. |
| * :mod:`multiprocessing` child processes on Windows no longer inherit |
| all of the parent's inheritable handles, only the necessary ones. |
| * A new :func:`hashlib.pbkdf2_hmac` function provides |
| the `PKCS#5 password-based key derivation function 2 |
| <http://en.wikipedia.org/wiki/PBKDF2>`_. |
| * :ref:`TLSv1.1 and TLSv1.2 support <whatsnew-tls-11-12>` for :mod:`ssl`. |
| * :ref:`Retrieving certificates from the Windows system cert store support |
| <whatsnew34-win-cert-store>` for :mod:`ssl`. |
| * :ref:`Server-side SNI (Server Name Indication) support |
| <whatsnew34-sni>` for :mod:`ssl`. |
| * The :class:`ssl.SSLContext` class has a :ref:`lot of improvements |
| <whatsnew34-sslcontext>`. |
| * All modules in the standard library that support SSL now support server |
| certificate verification, including hostname matching |
| (:func:`ssl.match_hostname`) and CRLs (Certificate Revocation lists, see |
| :func:`ssl.SSLContext.load_verify_locations`). |
| |
| CPython implementation improvements: |
| |
| * :ref:`Safe object finalization <whatsnew-pep-442>` (:pep:`442`). |
| * Leveraging :pep:`442`, in most cases :ref:`module globals are no longer set |
| to None during finalization <whatsnew-pep-442>` (:issue:`18214`). |
| * :ref:`Configurable memory allocators <whatsnew-pep-445>` (:pep:`445`). |
| * :ref:`Argument Clinic <whatsnew-pep-436>` (:pep:`436`). |
| |
| Please read on for a comprehensive list of user-facing changes, including many |
| other smaller improvements, CPython optimizations, deprecations, and potential |
| porting issues. |
| |
| |
| |
| New Features |
| ============ |
| |
| .. _whatsnew-pep-453: |
| |
| PEP 453: Explicit Bootstrapping of PIP in Python Installations |
| -------------------------------------------------------------- |
| |
| Bootstrapping pip By Default |
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| |
| The new :mod:`ensurepip` module (defined in :pep:`453`) provides a standard |
| cross-platform mechanism to bootstrap the pip installer into Python |
| installations and virtual environments. The version of ``pip`` included |
| with Python 3.4.0 is ``pip`` 1.5.4, and future 3.4.x maintenance releases |
| will update the bundled version to the latest version of ``pip`` that is |
| available at the time of creating the release candidate. |
| |
| By default, the commands ``pipX`` and ``pipX.Y`` will be installed on all |
| platforms (where X.Y stands for the version of the Python installation), |
| along with the ``pip`` Python package and its dependencies. On Windows and |
| in virtual environments on all platforms, the unversioned ``pip`` command |
| will also be installed. On other platforms, the system wide unversioned |
| ``pip`` command typically refers to the separately installed Python 2 |
| version. |
| |
| The :ref:`pyvenv <scripts-pyvenv>` command line utility and the :mod:`venv` |
| module make use of the :mod:`ensurepip` module to make ``pip`` readily |
| available in virtual environments. When using the command line utility, |
| ``pip`` is installed by default, while when using the :mod:`venv` module |
| :ref:`venv-api` installation of ``pip`` must be requested explicitly. |
| |
| For CPython :ref:`source builds on POSIX systems <building-python-on-unix>`, |
| the ``make install`` and ``make altinstall`` commands bootstrap ``pip`` by |
| default. This behaviour can be controlled through configure options, and |
| overridden through Makefile options. |
| |
| On Windows and Mac OS X, the CPython installers now default to installing |
| ``pip`` along with CPython itself (users may opt out of installing it |
| during the installation process). Window users will need to opt in to the |
| automatic ``PATH`` modifications to have ``pip`` available from the command |
| line by default, otherwise it can still be accessed through the Python |
| launcher for Windows as ``py -m pip``. |
| |
| As `discussed in the PEP`__, platform packagers may choose not to install |
| these commands by default, as long as, when invoked, they provide clear and |
| simple directions on how to install them on that platform (usually using |
| the system package manager). |
| |
| __ http://www.python.org/dev/peps/pep-0453/#recommendations-for-downstream-distributors |
| |
| .. note:: |
| |
| To avoid conflicts between parallel Python 2 and Python 3 installations, |
| only the versioned ``pip3`` and ``pip3.4`` commands are bootstrapped by |
| default when ``ensurepip`` is invoked directly - the ``--default-pip`` |
| option is needed to also request the unversioned ``pip`` command. |
| ``pyvenv`` and the Windows installer ensure that the unqualified ``pip`` |
| command is made available in those environments, and ``pip`` can always be |
| invoked via the ``-m`` switch rather than directly to avoid ambiguity on |
| systems with multiple Python installations. |
| |
| |
| Documentation Changes |
| ~~~~~~~~~~~~~~~~~~~~~ |
| |
| As part of this change, the :ref:`installing-index` and |
| :ref:`distributing-index` sections of the documentation have been |
| completely redesigned as short getting started and FAQ documents. Most |
| packaging documentation has now been moved out to the Python Packaging |
| Authority maintained `Python Packaging User Guide |
| <http://packaging.python.org>`__ and the documentation of the individual |
| projects. |
| |
| However, as this migration is currently still incomplete, the legacy |
| versions of those guides remaining available as :ref:`install-index` |
| and :ref:`distutils-index`. |
| |
| .. seealso:: |
| |
| :pep:`453` -- Explicit bootstrapping of pip in Python installations |
| PEP written by Donald Stufft and Nick Coghlan, implemented by |
| Donald Stufft, Nick Coghlan, Martin von Löwis and Ned Deily. |
| |
| |
| .. _whatsnew-pep-446: |
| |
| PEP 446: Newly Created File Descriptors Are Non-Inheritable |
| ----------------------------------------------------------- |
| |
| :pep:`446` makes newly created file descriptors :ref:`non-inheritable |
| <fd_inheritance>`. In general, this is the behavior an application will |
| want: when launching a new process, having currently open files also |
| open in the new process can lead to all sorts of hard to find bugs, |
| and potentially to security issues. |
| |
| However, there are occasions when inheritance is desired. To support |
| these cases, the following new functions and methods are available: |
| |
| * :func:`os.get_inheritable`, :func:`os.set_inheritable` |
| * :func:`os.get_handle_inheritable`, :func:`os.set_handle_inheritable` |
| * :meth:`socket.socket.get_inheritable`, :meth:`socket.socket.set_inheritable` |
| |
| .. seealso:: |
| |
| :pep:`446` -- Make newly created file descriptors non-inheritable |
| PEP written and implemented by Victor Stinner. |
| |
| |
| .. _codec-handling-improvements: |
| |
| Improvements to Codec Handling |
| ------------------------------ |
| |
| Since it was first introduced, the :mod:`codecs` module has always been |
| intended to operate as a type-neutral dynamic encoding and decoding |
| system. However, its close coupling with the Python text model, especially |
| the type restricted convenience methods on the builtin :class:`str`, |
| :class:`bytes` and :class:`bytearray` types, has historically obscured that |
| fact. |
| |
| As a key step in clarifying the situation, the :meth:`codecs.encode` and |
| :meth:`codecs.decode` convenience functions are now properly documented in |
| Python 2.7, 3.3 and 3.4. These functions have existed in the :mod:`codecs` |
| module (and have been covered by the regression test suite) since Python 2.4, |
| but were previously only discoverable through runtime introspection. |
| |
| Unlike the convenience methods on :class:`str`, :class:`bytes` and |
| :class:`bytearray`, the :mod:`codecs` convenience functions support arbitrary |
| codecs in both Python 2 and Python 3, rather than being limited to Unicode text |
| encodings (in Python 3) or ``basestring`` <-> ``basestring`` conversions (in |
| Python 2). |
| |
| In Python 3.4, the interpreter is able to identify the known non-text |
| encodings provided in the standard library and direct users towards these |
| general purpose convenience functions when appropriate:: |
| |
| >>> b"abcdef".decode("hex") |
| Traceback (most recent call last): |
| File "<stdin>", line 1, in <module> |
| LookupError: 'hex' is not a text encoding; use codecs.decode() to handle arbitrary codecs |
| |
| >>> "hello".encode("rot13") |
| Traceback (most recent call last): |
| File "<stdin>", line 1, in <module> |
| LookupError: 'rot13' is not a text encoding; use codecs.encode() to handle arbitrary codecs |
| |
| >>> open("foo.txt", encoding="hex") |
| Traceback (most recent call last): |
| File "<stdin>", line 1, in <module> |
| LookupError: 'hex' is not a text encoding; use codecs.open() to handle arbitrary codecs |
| |
| In a related change, whenever it is feasible without breaking backwards |
| compatibility, exceptions raised during encoding and decoding operations |
| are wrapped in a chained exception of the same type that mentions the |
| name of the codec responsible for producing the error:: |
| |
| >>> import codecs |
| |
| >>> codecs.decode(b"abcdefgh", "hex") |
| Traceback (most recent call last): |
| File "/usr/lib/python3.4/encodings/hex_codec.py", line 20, in hex_decode |
| return (binascii.a2b_hex(input), len(input)) |
| binascii.Error: Non-hexadecimal digit found |
| |
| The above exception was the direct cause of the following exception: |
| |
| Traceback (most recent call last): |
| File "<stdin>", line 1, in <module> |
| binascii.Error: decoding with 'hex' codec failed (Error: Non-hexadecimal digit found) |
| |
| >>> codecs.encode("hello", "bz2") |
| Traceback (most recent call last): |
| File "/usr/lib/python3.4/encodings/bz2_codec.py", line 17, in bz2_encode |
| return (bz2.compress(input), len(input)) |
| File "/usr/lib/python3.4/bz2.py", line 498, in compress |
| return comp.compress(data) + comp.flush() |
| TypeError: 'str' does not support the buffer interface |
| |
| The above exception was the direct cause of the following exception: |
| |
| Traceback (most recent call last): |
| File "<stdin>", line 1, in <module> |
| TypeError: encoding with 'bz2' codec failed (TypeError: 'str' does not support the buffer interface) |
| |
| Finally, as the examples above show, these improvements have permitted |
| the restoration of the convenience aliases for the non-Unicode codecs that |
| were themselves restored in Python 3.2. This means that encoding binary data |
| to and from its hexadecimal representation (for example) can now be written |
| as:: |
| |
| >>> from codecs import encode, decode |
| >>> encode(b"hello", "hex") |
| b'68656c6c6f' |
| >>> decode(b"68656c6c6f", "hex") |
| b'hello' |
| |
| The binary and text transforms provided in the standard library are detailed |
| in :ref:`binary-transforms` and :ref:`text-transforms`. |
| |
| (Contributed by Nick Coghlan in :issue:`7475`, :issue:`17827`, |
| :issue:`17828` and :issue:`19619`) |
| |
| |
| .. _whatsnew-pep-451: |
| |
| PEP 451: A ModuleSpec Type for the Import System |
| ------------------------------------------------ |
| |
| :pep:`451` provides an encapsulation of the information about a module that the |
| import machinery will use to load it (that is, a module specification). This |
| helps simplify both the import implementation and several import-related APIs. |
| The change is also a stepping stone for `several future import-related |
| improvements`__. |
| |
| __ https://mail.python.org/pipermail/python-dev/2013-November/130111.html |
| |
| The public-facing changes from the PEP are entirely backward-compatible. |
| Furthermore, they should be transparent to everyone but importer authors. Key |
| finder and loader methods have been deprecated, but they will continue working. |
| New importers should use the new methods described in the PEP. Existing |
| importers should be updated to implement the new methods. See the |
| :ref:`deprecated-3.4` section for a list of methods that should be replaced and |
| their replacements. |
| |
| |
| Other Language Changes |
| ---------------------- |
| |
| Some smaller changes made to the core Python language are: |
| |
| * Unicode database updated to UCD version 6.3. |
| |
| * :func:`min` and :func:`max` now accept a *default* keyword-only argument that |
| can be used to specify the value they return if the iterable they are |
| evaluating has no elements. (Contributed by Julian Berman in |
| :issue:`18111`.) |
| |
| * Module objects are now :mod:`weakref`'able. |
| |
| * Module ``__file__`` attributes (and related values) should now always |
| contain absolute paths by default, with the sole exception of |
| ``__main__.__file__`` when a script has been executed directly using |
| a relative path (Contributed by Brett Cannon in :issue:`18416`). |
| |
| * All the UTF-\* codecs (except UTF-7) now reject surrogates during both |
| encoding and decoding unless the ``surrogatepass`` error handler is used, |
| with the exception of the UTF-16 decoder (which accepts valid surrogate pairs) |
| and the UTF-16 encoder (which produces them while encoding non-BMP characters). |
| Contributed by Victor Stinner, Kang-Hao (Kenny) Lu and Serhiy Storchaka in |
| :issue:`12892`. |
| |
| * New German EBCDIC :ref:`codec <standard-encodings>` ``cp273``. (Contributed |
| by Michael Bierenfeld and Andrew Kuchling in :issue:`1097797`.) |
| |
| * New Ukrainian :ref:`codec <standard-encodings>` ``cp1125``. (Contributed by |
| Serhiy Storchaka in :issue:`19668`.) |
| |
| * :class:`bytes`.join() and :class:`bytearray`.join() now accept arbitrary |
| buffer objects as arguments. (Contributed by Antoine Pitrou in |
| :issue:`15958`.) |
| |
| * The :class:`int` constructor now accepts any object that has an ``__index__`` |
| method for its *base* argument. (Contributed by Mark Dickinson in |
| :issue:`16772`.) |
| |
| * Frame objects now have a :func:`~frame.clear` method that clears all |
| references to local variables from the frame. (Contributed by Antoine Pitrou |
| in :issue:`17934`.) |
| |
| * :class:`memoryview` is now registered as a :class:`Sequence <collections.abc>`, |
| and supports the :func:`reversed` builtin. (Contributed by Nick Coghlan |
| and Claudiu Popa in :issue:`18690` and :issue:`19078`.) |
| |
| * Signatures reported by :func:`help` have been modified and improved in |
| several cases as a result of the introduction of Argument Clinic and other |
| changes to the :mod:`inspect` and :mod:`pydoc` modules. |
| |
| * :meth:`~object.__length_hint__` is now part of the formal language |
| specification (see :pep:`424`). (Contributed by Armin Ronacher in |
| :issue:`16148`.) |
| |
| |
| New Modules |
| =========== |
| |
| |
| .. _whatsnew-asyncio: |
| |
| asyncio |
| ------- |
| |
| The new :mod:`asyncio` module (defined in :pep:`3156`) provides a standard |
| pluggable event loop model for Python, providing solid asynchronous IO |
| support in the standard library, and making it easier for other event loop |
| implementations to interoperate with the standard library and each other. |
| |
| For Python 3.4, this module is considered a :term:`provisional API`. |
| |
| .. seealso:: |
| |
| :pep:`3156` -- Asynchronous IO Support Rebooted: the "asyncio" Module |
| PEP written and implementation led by Guido van Rossum. |
| |
| |
| .. _whatsnew-ensurepip: |
| |
| ensurepip |
| --------- |
| |
| The new :mod:`ensurepip` module is the primary infrastructure for the |
| :pep:`453` implementation. In the normal course of events end users will not |
| need to interact with this module, but it can be used to manually bootstrap |
| ``pip`` if the automated bootstrapping into an installation or virtual |
| environment was declined. |
| |
| :mod:`ensurepip` includes a bundled copy of ``pip``, up-to-date as of the first |
| release candidate of the release of CPython with which it ships (this applies |
| to both maintenance releases and feature releases). ``ensurepip`` does not |
| access the internet. If the installation has Internet access, after |
| ``ensurepip`` is run the bundled ``pip`` can be used to upgrade ``pip`` to a |
| more recent release than the bundled one. (Note that such an upgraded version |
| of ``pip`` is considered to be a separately installed package and will not be |
| removed if Python is uninstalled.) |
| |
| The module is named *ensure*\ pip because if called when ``pip`` is already |
| installed, it does nothing. It also has an ``--upgrade`` option that will |
| cause it to install the bundled copy of ``pip`` if the existing installed |
| version of ``pip`` is older than the bundled copy. |
| |
| |
| .. _whatsnew-enum: |
| |
| enum |
| ---- |
| |
| The new :mod:`enum` module (defined in :pep:`435`) provides a standard |
| implementation of enumeration types, allowing other modules (such as |
| :mod:`socket`) to provide more informative error messages and better |
| debugging support by replacing opaque integer constants with backwards |
| compatible enumeration values. |
| |
| .. seealso:: |
| |
| :pep:`435` -- Adding an Enum type to the Python standard library |
| PEP written by Barry Warsaw, Eli Bendersky and Ethan Furman, |
| implemented by Ethan Furman. |
| |
| |
| .. _whatsnew-pathlib: |
| |
| pathlib |
| ------- |
| |
| The new :mod:`pathlib` module offers classes representing filesystem paths |
| with semantics appropriate for different operating systems. Path classes are |
| divided between *pure paths*, which provide purely computational operations |
| without I/O, and *concrete paths*, which inherit from pure paths but also |
| provide I/O operations. |
| |
| For Python 3.4, this module is considered a :term:`provisional API`. |
| |
| .. seealso:: |
| |
| :pep:`428` -- The pathlib module -- object-oriented filesystem paths |
| PEP written and implemented by Antoine Pitrou. |
| |
| |
| .. _whatsnew-selectors: |
| |
| selectors |
| --------- |
| |
| The new :mod:`selectors` module (created as part of implementing :pep:`3156`) |
| allows high-level and efficient I/O multiplexing, built upon the |
| :mod:`select` module primitives. |
| |
| |
| .. _whatsnew-statistics: |
| |
| statistics |
| ---------- |
| |
| The new :mod:`statistics` module (defined in :pep:`450`) offers some core |
| statistics functionality directly in the standard library. This module |
| supports calculation of the mean, median, mode, variance and standard |
| deviation of a data series. |
| |
| .. seealso:: |
| |
| :pep:`450` -- Adding A Statistics Module To The Standard Library |
| PEP written and implemented by Steven D'Aprano |
| |
| .. _whatsnew-tracemalloc: |
| |
| |
| tracemalloc |
| ----------- |
| |
| The new :mod:`tracemalloc` module (defined in :pep:`454`) is a debug tool to |
| trace memory blocks allocated by Python. It provides the following information: |
| |
| * Trace where an object was allocated |
| * Statistics on allocated memory blocks per filename and per line number: |
| total size, number and average size of allocated memory blocks |
| * Compute the differences between two snapshots to detect memory leaks |
| |
| .. seealso:: |
| |
| :pep:`454` -- Add a new tracemalloc module to trace Python memory allocations |
| PEP written and implemented by Victor Stinner |
| |
| |
| |
| Improved Modules |
| ================ |
| |
| |
| abc |
| --- |
| |
| New function :func:`abc.get_cache_token` can be used to know when to invalidate |
| caches that are affected by changes in the object graph. (Contributed |
| by Łukasz Langa in :issue:`16832`.) |
| |
| New class :class:`~abc.ABC` has :class:`~abc.ABCMeta` as its meta class. |
| Using ``ABC`` as a base class has essentially the same effect as specifying |
| ``metaclass=abc.ABCMeta``, but is simpler to type and easier to read. |
| (Contributed by Bruno Dupuis in :issue:`16049`.) |
| |
| |
| aifc |
| ---- |
| |
| The :meth:`~aifc.aifc.getparams` method now returns a namedtuple rather than a |
| plain tuple. (Contributed by Claudiu Popa in :issue:`17818`.) |
| |
| :func:`aifc.open` now supports the context manager protocol: when used in a |
| :keyword:`with` block, the :meth:`~aifc.aifc.close` method of the returned |
| object will be called automatically at the end of the block. (Contributed by |
| Serhiy Storchacha in :issue:`16486`.) |
| |
| The :meth:`~aifc.aifc.writeframesraw` and :meth:`~aifc.aifc.writeframes` |
| methods now accept any :term:`bytes-like object`. (Contributed by Serhiy |
| Storchaka in :issue:`8311`.) |
| |
| |
| argparse |
| -------- |
| |
| The :class:`~argparse.FileType` class now accepts *encoding* and |
| *errors* arguments, which are passed through to :func:`open`. (Contributed |
| by Lucas Maystre in :issue:`11175`.) |
| |
| |
| audioop |
| ------- |
| |
| :mod:`audioop` now supports 24-bit samples. (Contributed by Serhiy Storchaka |
| in :issue:`12866`.) |
| |
| New :func:`~audioop.byteswap` function converts big-endian samples to |
| little-endian and vice versa (Contributed by Serhiy Storchaka in |
| :issue:`19641`). |
| |
| All :mod:`audioop` functions now accept any :term:`bytes-like object`. Strings |
| are not accepted: they didn't work before, now they raise an error right away. |
| (Contributed by Serhiy Storchaka in :issue:`16685`.) |
| |
| |
| base64 |
| ------ |
| |
| The encoding and decoding functions in :mod:`base64` now accept any |
| :term:`bytes-like object` in cases where it previously required a |
| :class:`bytes` or :class:`bytearray` instance. (Contributed by Nick Coghlan in |
| :issue:`17839`.) |
| |
| New functions :func:`~base64.a85encode`, :func:`~base64.a85decode`, |
| :func:`~base64.b85encode`, and :func:`~base64.b85decode` provide the ability to |
| encode and decode binary data from and to ``Ascii85`` and the git/mercurial |
| ``Base85`` formats, respectively. The ``a85`` functions have options that can |
| be used to make them compatible with the variants of the ``Ascii85`` encoding, |
| including the Adobe variant. (Contributed by Martin Morrison, the Mercurial |
| project, Serhiy Storchaka, and Antoine Pitrou in :issue:`17618`.) |
| |
| |
| collections |
| ----------- |
| |
| The :meth:`.ChainMap.new_child` method now accepts an *m* argument specifying |
| the child map to add to the chain. This allows an existing mapping and/or a |
| custom mapping type to be used for the child. (Contributed by Vinay Sajip in |
| :issue:`16613`.) |
| |
| |
| colorsys |
| -------- |
| |
| The number of digits in the coefficients for the RGB --- YIQ conversions have |
| been expanded so that they match the FCC NTSC versions. The change in |
| results should be less than 1% and may better match results found elsewhere. |
| (Contributed by Brian Landers and Serhiy Storchaka in :issue:`14323`.) |
| |
| |
| contextlib |
| ---------- |
| |
| The new :class:`contextlib.suppress` context manager helps to clarify the |
| intent of code that deliberately suppresses exceptions from a single |
| statement. (Contributed by Raymond Hettinger in :issue:`15806` and |
| Zero Piraeus in :issue:`19266`) |
| |
| The new :func:`contextlib.redirect_stdout` context manager makes it easier |
| for utility scripts to handle inflexible APIs that write their output to |
| :data:`sys.stdout` and don't provide any options to redirect it. Using the |
| context manager, the :data:`sys.stdout` output can be redirected to any |
| other stream or, in conjunction with :class:`io.StringIO`, to a string. |
| The latter can be especially useful, for example, to capture output |
| from a function that was written to implement a command line interface. |
| It is recommended only for utility scripts because it affects the |
| global state of :data:`sys.stdout`. (Contributed by Raymond Hettinger |
| in :issue:`15805`) |
| |
| The :mod:`contextlib` documentation has also been updated to include a |
| :ref:`discussion <single-use-reusable-and-reentrant-cms>` of the |
| differences between single use, reusable and reentrant context managers. |
| |
| |
| dbm |
| --- |
| |
| :func:`dbm.open` objects now support the context management protocol. When |
| used in a :keyword:`with` statement, the ``close`` method of the database |
| object will be called automatically at the end of the block. (Contributed by |
| Claudiu Popa and Nick Coghlan in :issue:`19282`.) |
| |
| |
| dis |
| --- |
| |
| Functions :func:`~dis.show_code`, :func:`~dis.dis`, :func:`~dis.distb`, and |
| :func:`~dis.disassemble` now accept a keyword-only *file* argument that |
| controls where they write their output. |
| |
| The :mod:`dis` module is now built around an :class:`~dis.Instruction` class |
| that provides object oriented access to the details of each individual bytecode |
| operation. |
| |
| A new method, :func:`~dis.get_instructions`, provides an iterator that emits |
| the Instruction stream for a given piece of Python code. Thus it is now |
| possible to write a program that inspects and manipulates a bytecode |
| object in ways different from those provided by the :mod:`~dis` module |
| itself. For example:: |
| |
| >>> import dis |
| >>> for instr in dis.get_instructions(lambda x: x + 1): |
| ... print(instr.opname) |
| LOAD_FAST |
| LOAD_CONST |
| BINARY_ADD |
| RETURN_VALUE |
| |
| The various display tools in the :mod:`dis` module have been rewritten to use |
| these new components. |
| |
| In addition, a new application-friendly class :class:`~dis.Bytecode` provides |
| an object-oriented API for inspecting bytecode in both in human-readable form |
| and for iterating over instructions. The :class:`~dis.Bytecode` constructor |
| takes the same arguments that :func:`~dis.get_instruction` does (plus an |
| optional *current_offset*), and the resulting object can be iterated to produce |
| :class:`~dis.Instruction` objects. But it also has a :mod:`~dis.Bytecode.dis` |
| method, equivalent to calling :mod:`~dis.dis` on the constructor argument, but |
| returned as a multi-line string:: |
| |
| >>> bytecode = dis.Bytecode(lambda x: x +1, current_offset=3) |
| >>> for instr in bytecode: |
| ... print('{} ({})'.format(instr.opname, instr.opcode)) |
| LOAD_FAST (124) |
| LOAD_CONST (100) |
| BINARY_ADD (23) |
| RETURN_VALUE (83) |
| >>> bytecode.dis().splitlines() # doctest: +NORMALIZE_WHITESPACE |
| [' 1 0 LOAD_FAST 0 (x)', |
| ' --> 3 LOAD_CONST 1 (1)', |
| ' 6 BINARY_ADD', |
| ' 7 RETURN_VALUE'] |
| |
| :class:`~dis.Bytecode` also has a class method, |
| :meth:`~dis.Bytecode.from_traceback`, that provides the ability to manipulate a |
| traceback (that is, ``print(Bytecode.from_traceback(tb).dis())`` is equivalent |
| to ``distb(tb)``). |
| |
| (Contributed by Nick Coghlan, Ryan Kelly and Thomas Kluyver in :issue:`11816` |
| and Claudiu Popa in :issue:`17916`) |
| |
| New function :func:`~dis.stack_effect` computes the effect on the Python stack |
| of a given opcode and argument, information that is not otherwise available. |
| (Contributed by Larry Hastings in :issue:`19722`.) |
| |
| |
| doctest |
| ------- |
| |
| A new :ref:`option flag <doctest-options>`, :data:`~doctest.FAIL_FAST`, halts |
| test running as soon as the first failure is detected. (Contributed by R. |
| David Murray and Daniel Urban in :issue:`16522`.) |
| |
| The :mod:`doctest` command line interface now uses :mod:`argparse`, and has two |
| new options, ``-o`` and ``-f``. ``-o`` allows :ref:`doctest options |
| <doctest-options>` to be specified on the command line, and ``-f`` is a |
| shorthand for ``-o FAIL_FAST`` (to parallel the similar option supported by the |
| :mod:`unittest` CLI). (Contributed by R. David Murray in :issue:`11390`.) |
| |
| :mod:`doctest` will now find doctests in extension module ``__doc__`` strings. |
| (Contributed by Zachary Ware in :issue:`3158`.) |
| |
| |
| email |
| ----- |
| |
| :meth:`~email.message.Message.as_string` now accepts a *policy* argument to |
| override the default policy of the message when generating a string |
| representation of it. This means that ``as_string`` can now be used in more |
| circumstances, instead of having to create and use a :mod:`~email.generator` in |
| order to pass formatting parameters to its ``flatten`` method. (Contributed by |
| R. David Murray in :issue:`18600`.) |
| |
| New method :meth:`~email.message.Message.as_bytes` added to produce a bytes |
| representation of the message in a fashion similar to how ``as_string`` |
| produces a string representation. It does not accept the *maxheaderlen* |
| argument, but does accept the *unixfrom* and *policy* arguments. The |
| :class:`~email.message.Message` :meth:`~email.message.Message.__bytes__` method |
| calls it, meaning that ``bytes(mymsg)`` will now produce the intuitive |
| result: a bytes object containing the fully formatted message. (Contributed |
| by R. David Murray in :issue:`18600`.) |
| |
| The :meth:`.Message.set_param` message now accepts a *replace* keyword argument. |
| When specified, the associated header will be updated without changing |
| its location in the list of headers. For backward compatibility, the default |
| is ``False``. (Contributed by R. David Murray in :issue:`18891`.) |
| |
| |
| .. _whatsnew_email_contentmanager: |
| |
| A pair of new subclasses of :class:`~email.message.Message` have been added |
| (:class:`.EmailMessage` and :class:`.MIMEPart`), along with a new sub-module, |
| :mod:`~email.contentmanager` and a new :mod:`~email.policy` attribute |
| :attr:`~email.policy.EmailPolicy.content_manager`. All documentation is |
| currently in the new module, which is being added as part of email's new |
| :term:`provisional API`. These classes provide a number of new methods that |
| make extracting content from and inserting content into email messages much |
| easier. For details, see the :mod:`~email.contentmanager` documentation and |
| the :ref:`email-contentmanager-api-examples`. These API additions complete the |
| bulk of the work that was planned as part of the email6 project. The currently |
| provisional API is scheduled to become final in Python 3.5 (possibly with a few |
| minor additions in the area of error handling). (Contributed by R. David |
| Murray in :issue:`18891`.) |
| |
| |
| filecmp |
| ------- |
| |
| A new :func:`~filecmp.clear_cache` function provides the ability to clear the |
| :mod:`filecmp` comparison cache, which uses :func:`os.stat` information to |
| determine if the file has changed since the last compare. This can be used, |
| for example, if the file might have been changed and re-checked in less time |
| than the resolution of a particular filesystem's file modification time field. |
| (Contributed by Mark Levitt in :issue:`18149`.) |
| |
| New module attribute :data:`~filecmp.DEFAULT_IGNORES` provides the list of |
| directories that are used as the default value for the *ignore* parameter of |
| the :func:`~filecmp.dircmp` function. (Contributed by Eli Bendersky in |
| :issue:`15442`.) |
| |
| |
| functools |
| --------- |
| |
| The new :func:`~functools.partialmethod` descriptor brings partial argument |
| application to descriptors, just as :func:`~functools.partial` provides |
| for normal callables. The new descriptor also makes it easier to get |
| arbitrary callables (including :func:`~functools.partial` instances) |
| to behave like normal instance methods when included in a class definition. |
| (Contributed by Alon Horev and Nick Coghlan in :issue:`4331`) |
| |
| .. _whatsnew-singledispatch: |
| |
| The new :func:`~functools.singledispatch` decorator brings support for |
| single-dispatch generic functions to the Python standard library. Where |
| object oriented programming focuses on grouping multiple operations on a |
| common set of data into a class, a generic function focuses on grouping |
| multiple implementations of an operation that allows it to work with |
| *different* kinds of data. |
| |
| .. seealso:: |
| |
| :pep:`443` -- Single-dispatch generic functions |
| PEP written and implemented by Łukasz Langa. |
| |
| :func:`~functools.total_ordering` now supports a return value of |
| :const:`NotImplemented` from the underlying comparison function. (Contributed |
| by Katie Miller in :issue:`10042`.) |
| |
| A pure-python version of the :func:`~functools.partial` function is now in the |
| stdlib; in CPython it is overridden by the C accelerated version, but it is |
| available for other implementations to use. (Contributed by Brian Thorne in |
| :issue:`12428`.) |
| |
| |
| gc |
| -- |
| |
| New function :func:`~gc.get_stats` returns a list of three per-generation |
| dictionaries containing the collections statistics since interpreter startup. |
| (Contributed by Antoine Pitrou in :issue:`16351`.) |
| |
| |
| glob |
| ---- |
| |
| A new function :func:`~glob.escape` provides a way to escape special characters |
| in a filename so that they do not become part of the globbing expansion but are |
| instead matched literally. (Contributed by Serhiy Storchaka in :issue:`8402`.) |
| |
| |
| hashlib |
| ------- |
| |
| A new :func:`hashlib.pbkdf2_hmac` function provides |
| the `PKCS#5 password-based key derivation function 2 |
| <http://en.wikipedia.org/wiki/PBKDF2>`_. (Contributed by Christian |
| Heimes in :issue:`18582`) |
| |
| The :attr:`~hashlib.hash.name` attribute of :mod:`hashlib` hash objects is now |
| a formally supported interface. It has always existed in CPython's |
| :mod:`hashlib` (although it did not return lower case names for all supported |
| hashes), but it was not a public interface and so some other Python |
| implementations have not previously supported it. (Contributed by Jason R. |
| Coombs in :issue:`18532`.) |
| |
| |
| hmac |
| ---- |
| |
| :mod:`hmac` now accepts ``bytearray`` as well as ``bytes`` for the *key* |
| argument to the :func:`~hmac.new` function, and the *msg* parameter to both the |
| :func:`~hmac.new` function and the :meth:`~hmac.HMAC.update` method now |
| accepts any type supported by the :mod:`hashlib` module. (Contributed |
| by Jonas Borgström in :issue:`18240`.) |
| |
| The *digestmod* argument to the :func:`hmac.new` function may now be any hash |
| digest name recognized by :mod:`hashlib`. In addition, the current behavior in |
| which the value of *digestmod* defaults to ``MD5`` is deprecated: in a |
| future version of Python there will be no default value. (Contributed by |
| Christian Heimes in :issue:`17276`.) |
| |
| With the addition of :attr:`~hmac.HMAC.block_size` and :attr:`~hmac.HMAC.name` |
| attributes (and the formal documentation of the :attr:`~hmac.HMAC.digest_size` |
| attribute), the :mod:`hmac` module now conforms fully to the :pep:`247` API. |
| (Contributed by Christian Heimes in :issue:`18775`.) |
| |
| |
| html |
| ---- |
| |
| New function :func:`~html.unescape` function converts HTML5 character references to |
| the corresponding Unicode characters. (Contributed by Ezio Melotti in |
| :issue:`2927`) |
| |
| :class:`~html.parser.HTMLParser` accepts a new keyword argument |
| *convert_charrefs* that, when ``True``, automatically converts all character |
| references. For backward-compatibility, its value defaults to ``False``, but |
| it will change to ``True`` in a future version of Python, so you are invited to |
| set it explicitly and update your code to use this new feature. (Contributed |
| by Ezio Melotti in :issue:`13633`) |
| |
| The *strict* argument of :class:`~html.parser.HTMLParser` is now deprecated. |
| (Contributed by Ezio Melotti in :issue:`15114`) |
| |
| |
| http |
| ---- |
| |
| :meth:`~http.server.BaseHTTPRequestHandler.send_error` now accepts an |
| optional additional *explain* parameter which can be used to provide an |
| extended error description, overriding the hardcoded default if there is one. |
| This extended error description will be formatted using the |
| :attr:`~http.server.HTTP.error_message_format` attribute and sent as the body |
| of the error response. (Contributed by Karl Cow in :issue:`12921`.) |
| |
| The :mod:`http.server` :ref:`command line interface <http-server-cli>` now has |
| a ``-b/--bind`` option that causes the server to listen on a specific address. |
| (Contributed by Malte Swart in :issue:`17764`.) |
| |
| |
| importlib |
| --------- |
| |
| The :class:`~importlib.abc.InspectLoader` ABC defines a new method, |
| :meth:`~importlib.abc.InspectLoader.source_to_code` that accepts source |
| data and a path and returns a code object. The default implementation |
| is equivalent to ``compile(data, path, 'exec', dont_inherit=True)``. |
| (Contributed by Eric Snow and Brett Cannon in :issue:`15627`.) |
| |
| :class:`~importlib.abc.InspectLoader` also now has a default implementation |
| for the :meth:`~importlib.abc.InspectLoader.get_code` method. However, |
| it will normally be desirable to override the default implementation |
| for performance reasons. (Contributed by Brett Cannon in :issue:`18072`.) |
| |
| The :func:`~importlib.reload` function has been moved from :mod:`imp` to |
| :mod:`importlib` as part of the :mod:`imp` module deprecation. (Contributed by |
| Berker Peksag in :issue:`18193`.) |
| |
| :mod:`importlib.util` now has a :data:`~importlib.util.MAGIC_NUMBER` attribute |
| providing access to the bytecode version number. This replaces the |
| :func:`~imp.get_magic` function in the deprecated :mod:`imp` module. |
| (Contributed by Brett Cannon in :issue:`18192`.) |
| |
| New :mod:`importlib.util` functions :func:`~importlib.util.cache_from_source` |
| and :func:`~importlib.util.source_from_cache` replace the same-named functions |
| in the deprecated :mod:`imp` module. (Contributed by Brett Cannon in |
| :issue:`18194`.) |
| |
| The :mod:`importlib` bootstrap :class:`.NamespaceLoader` now conforms to |
| the :class:`.InspectLoader` ABC, which means that ``runpy`` and |
| ``python -m`` can now be used with namespace packages. (Contributed |
| by Brett Cannon in :issue:`18058`.) |
| |
| :mod:`importlib.util` has a new function :func:`~importlib.util.decode_source` |
| that decodes source from bytes using universal newline processing. This is |
| useful for implementing :meth:`.InspectLoader.get_source` methods. |
| |
| :class:`importlib.machinery.ExtensionFileLoader` now has a |
| :meth:`~importlib.machinery.ExtensionFileLoader.get_filename` method. This was |
| inadvertently omitted in the original implementation. (Contributed by Eric |
| Snow in :issue:`19152`.) |
| |
| |
| inspect |
| ------- |
| |
| The :mod:`inspect` module now offers a basic :ref:`command line interface |
| <inspect-module-cli>` to quickly display source code and other |
| information for modules, classes and functions. (Contributed by Claudiu Popa |
| and Nick Coghlan in :issue:`18626`) |
| |
| :func:`~inspect.unwrap` makes it easy to unravel wrapper function chains |
| created by :func:`functools.wraps` (and any other API that sets the |
| ``__wrapped__`` attribute on a wrapper function). (Contributed by |
| Daniel Urban, Aaron Iles and Nick Coghlan in :issue:`13266`) |
| |
| As part of the implementation of the new :mod:`enum` module, the |
| :mod:`inspect` module now has substantially better support for custom |
| ``__dir__`` methods and dynamic class attributes provided through |
| metaclasses (Contributed by Ethan Furman in :issue:`18929` and |
| :issue:`19030`) |
| |
| :func:`~inspect.getfullargspec` and :func:`~inspect.getargspec` |
| now use the :func:`~inspect.signature` API. This allows them to |
| support a much broader range of callables, including those with |
| ``__signature__`` attributes, those with metadata provided by argument |
| clinic, :func:`functools.partial` objects and more. Note that, unlike |
| :func:`~inspect.signature`, these functions still ignore ``__wrapped__`` |
| attributes, and report the already bound first argument for bound methods, |
| so it is still necessary to update your code to use |
| :func:`~inspect.signature` directly if those features are desired. |
| (Contributed by Yury Selivanov in :issue:`17481`) |
| |
| :func:`~inspect.signature` now supports duck types of CPython functions, |
| which adds support for functions compiled with Cython. (Contributed |
| by Stefan Behnel and Yury Selivanov in :issue:`17159`) |
| |
| |
| ipaddress |
| --------- |
| |
| :mod:`ipaddress` was added to the standard library in Python 3.3 as a |
| :term:`provisional API`. With the release of Python 3.4, this qualification |
| has been removed: :mod:`ipaddress` is now considered a stable API, covered |
| by the normal standard library requirements to maintain backwards |
| compatibility. |
| |
| A new :attr:`~ipaddress.IPv4Address.is_global` property is ``True`` if |
| an address is globally routeable. (Contributed by Peter Moody in |
| :issue:`17400`.) |
| |
| |
| logging |
| ------- |
| |
| The :class:`~logging.handlers.TimedRotatingFileHandler` has a new *atTime* |
| parameter that can be used to specify the time of day when rollover should |
| happen. (Contributed by Ronald Oussoren in :issue:`9556`.) |
| |
| :class:`~logging.handlers.SocketHandler` and |
| :class:`~logging.handlers.DatagramHandler` now support Unix domain sockets (by |
| setting *port* to ``None``). (Contributed by Vinay Sajip in commit |
| ce46195b56a9.) |
| |
| :func:`~logging.config.fileConfig` now accepts a |
| :class:`configparser.RawConfigParser` subclass instance for the *fname* |
| parameter. This facilitates using a configuration file when logging |
| configuration is just a part of the overall application configuration, or where |
| the application modifies the configuration before passing it to |
| :func:`~logging.config.fileConfig`. (Contributed by Vinay Sajip in |
| :issue:`16110`.) |
| |
| Logging configuration data received from a socket via the |
| :func:`logging.config.listen` function can now be validated before being |
| processed by supplying a verification function as the argument to the new |
| *verify* keyword argument. (Contributed by Vinay Sajip in :issue:`15452`.) |
| |
| |
| .. _whatsnew-marshal-3: |
| |
| marshal |
| ------- |
| |
| The default :mod:`marshal` version has been bumped to 3. The code implementing |
| the new version restores the Python2 behavior of recording only one copy of |
| interned strings and preserving the interning on deserialization, and extends |
| this "one copy" ability to any object type (including handling recursive |
| references). This reduces both the size of ``.pyc`` files and the amount of |
| memory a module occupies in memory when it is loaded from a ``.pyc`` (or |
| ``.pyo``) file. (Contributed by Kristján Valur Jónsson in :issue:`16475`, |
| with additional speedups by Antoine Pitrou in :issue:`19219`.) |
| |
| |
| mmap |
| ---- |
| |
| mmap objects can now be :mod:`weakref`\ ed. (Contributed by Valerie Lambert in |
| :issue:`4885`.) |
| |
| |
| multiprocessing |
| --------------- |
| |
| .. _whatsnew-multiprocessing-no-fork: |
| |
| On Unix two new :ref:`start methods <multiprocessing-start-methods>`, |
| (``spawn`` and ``forkserver``, have been added for starting processes using |
| :mod:`multiprocessing`. These make the mixing of processes with threads more |
| robust, and the ``spawn`` method matches the semantics that multiprocessing has |
| always used on Windows. New function |
| :func:`~multiprocessing.get_all_start_methods` reports all start methods |
| available on the platform, :func:`~multiprocessing.get_start_method` reports |
| the current start method, and :func:`~multiprocessing.set_start_method` sets |
| the start method. (Contributed by Richard Oudkerk in :issue:`8713`). |
| |
| :mod:`multiprocessing` also now has the concept of a ``context``, which |
| determines how child processes are created. New function |
| :func:`~multiprocessing.get_context` returns a context that uses a specified |
| start method. It has the same API as the :mod:`multiprocessing` module itself, |
| so you can use it to create :class:`~multiprocessing.pool.Pool`\ s and other |
| objects that will operate within that context. This allows a framework and an |
| application or different parts of the same application to use multiprocessing |
| without interfering with each other. (Contributed by Richard Oudkerk in |
| :issue:`18999`.) |
| |
| Except when using the old *fork* start method, child processes no longer |
| inherit unneeded handles/file descriptors from their parents (part of |
| :issue:`8713`). |
| |
| :mod:`multiprocessing` now relies on :mod:`runpy` (which implements the |
| ``-m`` switch) to initialise ``__main__`` appropriately in child processes |
| when using the ``spawn`` or ``forkserver`` start methods. This resolves some |
| edge cases where combining multiprocessing, the ``-m`` command line switch, |
| and explicit relative imports could cause obscure failures in child |
| processes. (Contributed by Nick Coghlan in :issue:`19946`) |
| |
| |
| operator |
| -------- |
| |
| New function :func:`~operator.length_hint` provides an implementation of the |
| specification for how the :meth:`~object.__length_hint__` special method should |
| be used, as part of the :pep:`424` formal specification of this language |
| feature. (Contributed by Armin Ronacher in :issue:`16148`.) |
| |
| There is now a pure-python version of the :mod:`operator` module available for |
| reference and for use by alternate implementations of Python. (Contributed by |
| Zachary Ware in :issue:`16694`.) |
| |
| |
| os |
| -- |
| |
| There are new functions to get and set the :ref:`inheritable flag |
| <fd_inheritance>` of a file descriptor (:func:`os.get_inheritable`, |
| :func:`os.set_inheritable`) or a Windows handle |
| (:func:`os.get_handle_inheritable`, :func:`os.set_handle_inheritable`). |
| |
| New function :func:`~os.cpu_count` reports the number of CPUs available on the |
| platform on which Python is running (or ``None`` if the count can't be |
| determined). The :func:`multiprocessing.cpu_count` function is now implemented |
| in terms of this function). (Contributed by Trent Nelson, Yogesh Chaudhari, |
| Victor Stinner, and Charles-François Natali in :issue:`17914`.) |
| |
| :func:`os.path.samestat` is now available on the Windows platform (and the |
| :func:`os.path.samefile` implementation is now shared between Unix and |
| Windows). (Contributed by Brian Curtin in :issue:`11939`.) |
| |
| :func:`os.path.ismount` now recognizes volumes mounted below a drive |
| root on Windows. (Contributed by Tim Golden in :issue:`9035`.) |
| |
| :func:`os.open` supports two new flags on platforms that provide them, |
| :data:`~os.O_PATH` (un-opened file descriptor), and :data:`~os.O_TMPFILE` |
| (unnamed temporary file; as of 3.4.0 release available only on Linux systems |
| with a kernel version of 3.11 or newer that have uapi headers). (Contributed |
| by Christian Heimes in :issue:`18673` and Benjamin Peterson, respectively.) |
| |
| |
| pdb |
| --- |
| |
| :mod:`pdb` has been enhanced to handle generators, :keyword:`yield`, and |
| ``yield from`` in a more useful fashion. This is especially helpful when |
| debugging :mod:`asyncio` based programs. (Contributed by Andrew Svetlov and |
| Xavier de Gaye in :issue:`16596`.) |
| |
| The ``print`` command has been removed from :mod:`pdb`, restoring access to the |
| Python :func:`print` function from the pdb command line. Python2's ``pdb`` did |
| not have a ``print`` command; instead, entering ``print`` executed the |
| ``print`` statement. In Python3 ``print`` was mistakenly made an alias for the |
| pdb :pdbcmd:`p` command. ``p``, however, prints the ``repr`` of its argument, |
| not the ``str`` like the Python2 ``print`` command did. Worse, the Python3 |
| ``pdb print`` command shadowed the Python3 ``print`` function, making it |
| inaccessible at the ``pdb`` prompt. (Contributed by Connor Osborn in |
| :issue:`18764`.) |
| |
| |
| .. _whatsnew-protocol-4: |
| |
| pickle |
| ------ |
| |
| :mod:`pickle` now supports (but does not use by default) a new pickle protocol, |
| protocol 4. This new protocol addresses a number of issues that were present |
| in previous protocols, such as the serialization of nested classes, very large |
| strings and containers, and classes whose :meth:`__new__` method takes |
| keyword-only arguments. It also provides some efficiency improvements. |
| |
| .. seealso:: |
| |
| :pep:`3154` -- Pickle protocol 4 |
| PEP written by Antoine Pitrou and implemented by Alexandre Vassalotti. |
| |
| |
| plistlib |
| -------- |
| |
| :mod:`plistlib` now has an API that is similar to the standard pattern for |
| stdlib serialization protocols, with new :func:`~plistlib.load`, |
| :func:`~plistlib.dump`, :func:`~plistlib.loads`, and :func:`~plistlib.dumps` |
| functions. (The older API is now deprecated.) In addition to the already |
| supported XML plist format (:data:`~plistlib.FMT_XML`), it also now supports |
| the binary plist format (:data:`~plistlib.FMT_BINARY`). (Contributed by Ronald |
| Oussoren and others in :issue:`14455`). |
| |
| |
| poplib |
| ------ |
| |
| Two new methods have been added to :mod:`poplib`: :meth:`~poplib.POP3.capa`, |
| which returns the list of capabilities advertised by the POP server, and |
| :meth:`~poplib.POP3.stls`, which switches a clear-text POP3 session into an |
| encrypted POP3 session if the POP server supports it. (Contributed by Lorenzo |
| Catucci in :issue:`4473`.) |
| |
| |
| pprint |
| ------ |
| |
| The :mod:`pprint` module's :class:`~pprint.PrettyPrinter` class and its |
| :func:`~pprint.pformat`, and :func:`~pprint.pprint` functions have a new |
| option, *compact*, that controls how the output is formatted. Currently |
| setting *compact* to ``True`` means that sequences will be printed with as many |
| sequence elements as will fit within *width* on each (indented) line. |
| (Contributed by Serhiy Storchaka in :issue:`19132`.) |
| |
| Long strings are now wrapped using Python's normal line continuation |
| syntax. (Contributed by Antoine Pitrou in :issue:`17150`). |
| |
| |
| pty |
| --- |
| |
| :func:`pty.spawn` now returns the status value from :func:`os.waitpid` on |
| the child process, instead of ``None``. (Contributed by Gregory P. Smith.) |
| |
| |
| pydoc |
| ----- |
| |
| The :mod:`pydoc` module is now based directly on the :func:`inspect.signature` |
| introspection API, allowing it to provide signature information for a wider |
| variety of callable objects. This change also means that ``__wrapped__`` |
| attributes are now taken into account when displaying help information |
| (Contributed by Larry Hastings in :issue:`19674`) |
| |
| The :mod:`pydoc` module no longer displays the ``self`` parameter for |
| already bound methods. Instead, it aims to always display the exact current |
| signature of the supplied callable (Contributed by Larry Hastings in |
| :issue:`20710`) |
| |
| In addition to the changes that have been made to :mod:`pydoc` directly, |
| its handling of custom ``__dir__`` methods and various descriptor |
| behaviours has also been improved substantially by the underlying changes in |
| the :mod:`inspect` module. |
| |
| As the :func:`help` builtin is based on :mod:`pydoc`, the above changes also |
| affect the behaviour of :func:`help`. |
| |
| |
| re |
| -- |
| |
| New :func:`~re.fullmatch` function and :meth:`.regex.fullmatch` method anchor |
| the pattern at both ends of the string to match. This provides a way to be |
| explicit about the goal of the match, which avoids a class of subtle bugs where |
| ``$`` characters get lost during code changes or the addition of alternatives |
| to an existing regular expression. (Contributed by Matthew Barnett in |
| :issue:`16203`.) |
| |
| The repr of :ref:`regex objects <re-objects>` now includes the pattern |
| and the flags; the repr of :ref:`match objects <match-objects>` now |
| includes the start, end, and the part of the string that matched. (Contributed |
| by Hugo Lopes Tavares and Serhiy Storchaka in :issue:`13592` and |
| :issue:`17087`.) |
| |
| |
| resource |
| -------- |
| |
| New :func:`~resource.prlimit` function, available on Linux platforms with a |
| kernel version of 2.6.36 or later and glibc of 2.13 or later, provides the |
| ability to query or set the resource limits for processes other than the one |
| making the call. (Contributed by Christian Heimes in :issue:`16595`.) |
| |
| On Linux kernel version 2.6.36 or later, there are there are also some new |
| Linux specific constants: :attr:`~resource.RLIMIT_MSGQUEUE`, |
| :attr:`~resource.RLIMIT_NICE`, :attr:`~resource.RLIMIT_RTPRIO`, |
| :attr:`~resource.RLIMIT_RTTIME`, and :attr:`~resource.RLIMIT_SIGPENDING`. |
| (Contributed by Christian Heimes in :issue:`19324`.) |
| |
| On FreeBSD version 9 and later, there some new FreeBSD specific constants: |
| :attr:`~resource.RLIMIT_SBSIZE`, :attr:`~resource.RLIMIT_SWAP`, and |
| :attr:`~resource.RLIMIT_NPTS`. (Contributed by Claudiu Popa in |
| :issue:`19343`.) |
| |
| |
| select |
| ------ |
| |
| :class:`~select.epoll` objects now support the context management protocol. |
| When used in a :keyword:`with` statement, the :meth:`~select.epoll.close` |
| method will be called automatically at the end of the block. (Contributed |
| by Serhiy Storchaka in :issue:`16488`.) |
| |
| :class:`~select.devpoll` objects now have :meth:`~select.devpoll.fileno` and |
| :meth:`~select.devpoll.close` methods, as well as a new attribute |
| :attr:`~select.devpoll.closed`. (Contributed by Victor Stinner in |
| :issue:`18794`.) |
| |
| |
| shelve |
| ------ |
| |
| :class:`~shelve.Shelf` instances may now be used in :keyword:`with` statements, |
| and will be automatically closed at the end of the :keyword:`with` block. |
| (Contributed by Filip Gruszczyński in :issue:`13896`.) |
| |
| |
| shutil |
| ------ |
| |
| :func:`~shutil.copyfile` now raises a specific :exc:`~shutil.Error` subclass, |
| :exc:`~shutil.SameFileError`, when the source and destination are the same |
| file, which allows an application to take appropriate action on this specific |
| error. (Contributed by Atsuo Ishimoto and Hynek Schlawack in |
| :issue:`1492704`.) |
| |
| |
| smtpd |
| ----- |
| |
| The :class:`~smtpd.SMTPServer` and :class:`~smtpd.SMTPChannel` classes now |
| accept a *map* keyword argument which, if specified, is passed in to |
| :class:`asynchat.async_chat` as its *map* argument. This allows an application |
| to avoid affecting the global socket map. (Contributed by Vinay Sajip in |
| :issue:`11959`.) |
| |
| |
| smtplib |
| ------- |
| |
| :exc:`~smtplib.SMTPException` is now a subclass of :exc:`OSError`, which allows |
| both socket level errors and SMTP protocol level errors to be caught in one |
| try/except statement by code that only cares whether or not an error occurred. |
| (Contributed by Ned Jackson Lovely in :issue:`2118`). |
| |
| |
| socket |
| ------ |
| |
| The socket module now supports the :data:`~socket.CAN_BCM` protocol on |
| platforms that support it. (Contributed by Brian Thorne in :issue:`15359`.) |
| |
| Socket objects have new methods to get or set their :ref:`inheritable flag |
| <fd_inheritance>`, :meth:`~socket.socket.get_inheritable` and |
| :meth:`~socket.socket.set_inheritable`. |
| |
| The ``socket.AF_*`` and ``socket.SOCK_*`` constants are now enumeration values |
| using the new :mod:`enum` module. This allows meaningful names to be printed |
| during debugging, instead of integer "magic numbers". |
| |
| The :data:`~socket.AF_LINK` constant is now available on BSD and OSX. |
| |
| :func:`~socket.inet_pton` and :func:`~socket.inet_ntop` are now supported |
| on Windows. (Contributed by Atsuo Ishimoto in :issue:`7171`.) |
| |
| |
| sqlite3 |
| ------- |
| |
| A new boolean parameter to the :func:`~sqlite3.connect` function, *uri*, can be |
| used to indicate that the *database* parameter is a ``uri`` (see the `SQLite |
| URI documentation <http://www.sqlite.org/uri.html>`_). (Contributed by poq in |
| :issue:`13773`.) |
| |
| |
| ssl |
| --- |
| |
| .. _whatsnew-tls-11-12: |
| |
| :data:`~ssl.PROTOCOL_TLSv1_1` and :data:`~ssl.PROTOCOL_TLSv1_2` (TLSv1.1 and |
| TLSv1.2 support) have been added; support for these protocols is only available if |
| Python is linked with OpenSSL 1.0.1 or later. (Contributed by Michele Orrù and |
| Antoine Pitrou in :issue:`16692`) |
| |
| .. _whatsnew34-sslcontext: |
| |
| New function :func:`~ssl.create_default_context` provides a standard way to |
| obtain an :class:`~ssl.SSLContext` whose settings are intended to be a |
| reasonable balance between compatibility and security. These settings are |
| more stringent than the defaults provided by the :class:`~ssl.SSLContext` |
| constructor, and may be adjusted in the future, without prior deprecation, if |
| best-practice security requirements change. The new recommended best |
| practice for using stdlib libraries that support SSL is to use |
| :func:`~ssl.create_default_context` to obtain an :class:`~ssl.SSLContext` |
| object, modify it if needed, and then pass it as the *context* argument |
| of the appropriate stdlib API. (Contributed by Christian Heimes |
| in :issue:`19689`.) |
| |
| :class:`~ssl.SSLContext` method :meth:`~ssl.SSLContext.load_verify_locations` |
| accepts a new optional argument *cadata*, which can be used to provide PEM or |
| DER encoded certificates directly via strings or bytes, respectively. |
| (Contributed by Christian Heimes in :issue:`18138`.) |
| |
| New function :func:`~ssl.get_default_verify_paths` returns |
| a named tuple of the paths and environment variables that the |
| :meth:`~ssl.SSLContext.set_default_verify_paths` method uses to set |
| OpenSSL's default ``cafile`` and ``capath``. This can be an aid in |
| debugging default verification issues. (Contributed by Christian Heimes |
| in :issue:`18143`.) |
| |
| :class:`~ssl.SSLContext` has a new method, |
| :meth:`~ssl.SSLContext.cert_store_stats`, that reports the number of loaded |
| ``X.509`` certs, ``X.509 CA`` certs, and certificate revocation lists (``crl``\ |
| s), as well as a :meth:`~ssl.SSLContext.get_ca_certs` method that returns a |
| list of the loaded ``CA`` certificates. (Contributed by Christian Heimes in |
| :issue:`18147`.) |
| |
| If OpenSSL 0.9.8 or later is available, :class:`~ssl.SSLContext` has an new |
| attribute :attr:`~ssl.SSLContext.verify_flags` that can be used to control the |
| certificate verification process by setting it to some combination of the new |
| constants :data:`~ssl.VERIFY_DEFAULT`, :data:`~ssl.VERIFY_CRL_CHECK_LEAF`, |
| :data:`~ssl.VERIFY_CRL_CHECK_CHAIN`, or :data:`~ssl.VERIFY_X509_STRICT`. |
| OpenSSL does not do any CRL verification by default. (Contributed by |
| Christien Heimes in :issue:`8813`.) |
| |
| New :class:`~ssl.SSLContext` method :meth:`~ssl.SSLContext.load_default_certs` |
| loads a set of default "certificate authority" (CA) certificates from default |
| locations, which vary according to the platform. It can be used to load both |
| TLS web server authentication certificates |
| (``purpose=``:data:`~ssl.Purpose.SERVER_AUTH`) for a client to use to verify a |
| server, and certificates for a server to use in verifying client certificates |
| (``purpose=``:data:`~ssl.Purpose.CLIENT_AUTH`). (Contributed by Christian |
| Heimes in :issue:`19292`.) |
| |
| .. _whatsnew34-win-cert-store: |
| |
| Two new windows-only functions, :func:`~ssl.enum_certificates` and |
| :func:`~ssl.enum_crls` provide the ability to retrieve certificates, |
| certificate information, and CRLs from the Windows cert store. (Contributed |
| by Christian Heimes in :issue:`17134`.) |
| |
| .. _whatsnew34-sni: |
| |
| Support for server-side SNI (Server Name Indication) using the new |
| :meth:`ssl.SSLContext.set_servername_callback` method. |
| (Contributed by Daniel Black in :issue:`8109`.) |
| |
| The dictionary returned by :meth:`.SSLSocket.getpeercert` contains additional |
| ``X509v3`` extension items: ``crlDistributionPoints``, ``calIssuers``, and |
| ``OCSP`` URIs. (Contributed by Christian Heimes in :issue:`18379`.) |
| |
| |
| stat |
| ---- |
| |
| The :mod:`stat` module is now backed by a C implementation in :mod:`_stat`. A C |
| implementation is required as most of the values aren't standardized and |
| are platform-dependent. (Contributed by Christian Heimes in :issue:`11016`.) |
| |
| The module supports new :mod:`~stat.ST_MODE` flags, :mod:`~stat.S_IFDOOR`, |
| :attr:`~stat.S_IFPORT`, and :attr:`~stat.S_IFWHT`. (Contributed by |
| Christian Hiemes in :issue:`11016`.) |
| |
| |
| struct |
| ------ |
| |
| New function :mod:`~struct.iter_unpack` and a new |
| :meth:`struct.Struct.iter_unpack` method on compiled formats provide streamed |
| unpacking of a buffer containing repeated instances of a given format of data. |
| (Contributed by Antoine Pitrou in :issue:`17804`.) |
| |
| |
| subprocess |
| ---------- |
| |
| :func:`~subprocess.check_output` now accepts an *input* argument that can |
| be used to provide the contents of ``stdin`` for the command that is run. |
| (Contributed by Zack Weinberg in :issue:`16624`.) |
| |
| :func:`~subprocess.getstatus` and :func:`~subprocess.getstatusoutput` now |
| work on Windows. This change was actually inadvertently made in 3.3.4. |
| (Contributed by Tim Golden in :issue:`10197`.) |
| |
| |
| sunau |
| ----- |
| |
| The :meth:`~sunau.getparams` method now returns a namedtuple rather than a |
| plain tuple. (Contributed by Claudiu Popa in :issue:`18901`.) |
| |
| :meth:`sunau.open` now supports the context manager protocol: when used in a |
| :keyword:`with` block, the ``close`` method of the returned object will be |
| called automatically at the end of the block. (Contributed by Serhiy Storchaka |
| in :issue:`18878`.) |
| |
| :meth:`.AU_write.setsampwidth` now supports 24 bit samples, thus adding |
| support for writing 24 sample using the module. (Contributed by |
| Serhiy Storchaka in :issue:`19261`.) |
| |
| The :meth:`~sunau.AU_write.writeframesraw` and |
| :meth:`~sunau.AU_write.writeframes` methods now accept any :term:`bytes-like |
| object`. (Contributed by Serhiy Storchaka in :issue:`8311`.) |
| |
| |
| sys |
| --- |
| |
| New function :func:`sys.getallocatedblocks` returns the current number of |
| blocks allocated by the interpreter. (In CPython with the default |
| ``--with-pymalloc`` setting, this is allocations made through the |
| :c:func:`PyObject_Malloc` API.) This can be useful for tracking memory leaks, |
| especially if automated via a test suite. (Contributed by Antoine Pitrou |
| in :issue:`13390`.) |
| |
| When the Python interpreter starts in :ref:`interactive mode |
| <tut-interactive>`, it checks for an :data:`~sys.__interactivehook__` attribute |
| on the :mod:`sys` module. If the attribute exists, its value is called with no |
| arguments just before interactive mode is started. The check is made after the |
| :envvar:`PYTHONSTARTUP` file is read, so it can be set there. The :mod:`site` |
| module :ref:`sets it <rlcompleter-config>` to a function that enables tab |
| completion and history saving (in :file:`~/.python-history`) if the platform |
| supports :mod:`readline`. If you do not want this (new) behavior, you can |
| override it in :envvar:`PYTHONSTARTUP`, :mod:`sitecustomize`, or |
| :mod:`usercustomize` by deleting this attribute from :mod:`sys` (or setting it |
| to some other callable). (Contributed by Éric Araujo and Antoine Pitrou in |
| :issue:`5845`.) |
| |
| |
| tarfile |
| ------- |
| |
| The :mod:`tarfile` module now supports a simple :ref:`tarfile-commandline` when |
| called as a script directly or via ``-m``. This can be used to create and |
| extract tarfile archives. (Contributed by Berker Peksag in :issue:`13477`.) |
| |
| |
| textwrap |
| -------- |
| |
| The :class:`~textwrap.TextWrapper` class has two new attributes/constructor |
| arguments: :attr:`~textwrap.TextWrapper.max_lines`, which limits the number of |
| lines in the output, and :attr:`~textwrap.TextWrapper.placeholder`, which is a |
| string that will appear at the end of the output if it has been truncated |
| because of *max_lines*. Building on these capabilities, a new convenience |
| function :func:`~textwrap.shorten` collapses all of the whitespace in the input |
| to single spaces and produces a single line of a given *width* that ends with |
| the *placeholder* (by default, ``[...]``). (Contributed by Antoine Pitrou and |
| Serhiy Storchaka in :issue:`18585` and :issue:`18725`.) |
| |
| |
| threading |
| --------- |
| |
| The :class:`~threading.Thread` object representing the main thread can be |
| obtained from the new :func:`~threading.main_thread` function. In normal |
| conditions this will be the thread from which the Python interpreter was |
| started. (Contributed by Andrew Svetlov in :issue:`18882`.) |
| |
| |
| traceback |
| --------- |
| |
| A new :func:`traceback.clear_frames` function takes a traceback object |
| and clears the local variables in all of the frames it references, |
| reducing the amount of memory consumed. (Contributed by Andrew Kuchling in |
| :issue:`1565525`). |
| |
| |
| types |
| ----- |
| |
| A new :func:`~types.DynamicClassAttribute` descriptor provides a way to define |
| an attribute that acts normally when looked up through an instance object, but |
| which is routed to the *class* ``__getattr__`` when looked up through the |
| class. This allows one to have properties active on a class, and have virtual |
| attributes on the class with the same name (see :mod:`Enum` for an example). |
| (Contributed by Ethan Furman in :issue:`19030`.) |
| |
| |
| urllib |
| ------ |
| |
| :mod:`urllib.request` now supports ``data:`` URLs via the |
| :class:`~urllib.request.DataHandler` class. (Contributed by Mathias Panzenböck |
| in :issue:`16423`.) |
| |
| The http method that will be used by a :class:`~urllib.request.Request` class |
| can now be specified by setting a :class:`~urllib.request.Request.method` |
| class attribute on the subclass. (Contributed by Jason R Coombs in |
| :issue:`18978`.) |
| |
| :class:`~urllib.request.Request` objects are now reusable: if the |
| :attr:`~urllib.request.Request.full_url` or :attr:`~urllib.request.Request.data` |
| attributes are modified, all relevant internal properties are updated. This |
| means, for example, that it is now possible to use the same |
| :class:`~urllib.request.Request` object in more than one |
| :meth:`.OpenerDirector.open` call with different *data* arguments, or to |
| modify a :class:`~urllib.request.Request`\ 's ``url`` rather than recomputing it |
| from scratch. There is also a new |
| :meth:`~urllib.request.Request.remove_header` method that can be used to remove |
| headers from a :class:`~urllib.request.Request`. (Contributed by Alexey |
| Kachayev in :issue:`16464`, Daniel Wozniak in :issue:`17485`, and Damien Brecht |
| and Senthil Kumaran in :issue:`17272`.) |
| |
| :class:`~urllib.error.HTTPError` objects now have a |
| :attr:`~urllib.error.HTTPError.headers` attribute that provides access to the |
| HTTP response headers associated with the error. (Contributed by |
| Berker Peksag in :issue:`15701`.) |
| |
| |
| unittest |
| -------- |
| |
| The :class:`~unittest.TestCase` class has a new method, |
| :meth:`~unittest.TestCase.subTest`, that produces a context manager whose |
| :keyword:`with` block becomes a "sub-test". This context manager allows a test |
| method to dynamically generate subtests by, say, calling the ``subTest`` |
| context manager inside a loop. A single test method can thereby produce an |
| indefinite number of separately-identified and separately-counted tests, all of |
| which will run even if one or more of them fail. For example:: |
| |
| class NumbersTest(unittest.TestCase): |
| def test_even(self): |
| for i in range(6): |
| with self.subTest(i=i): |
| self.assertEqual(i % 2, 0) |
| |
| will result in six subtests, each identified in the unittest verbose output |
| with a label consisting of the variable name ``i`` and a particular value for |
| that variable (``i=0``, ``i=1``, etc). See :ref:`subtests` for the full |
| version of this example. (Contributed by Antoine Pitrou in :issue:`16997`.) |
| |
| :func:`unittest.main` now accepts an iterable of test names for |
| *defaultTest*, where previously it only accepted a single test name as a |
| string. (Contributed by Jyrki Pulliainen in :issue:`15132`.) |
| |
| If :class:`~unittest.SkipTest` is raised during test discovery (that is, at the |
| module level in the test file), it is now reported as a skip instead of an |
| error. (Contributed by Zach Ware in :issue:`16935`.) |
| |
| :meth:`~unittest.TestLoader.discover` now sorts the discovered files to provide |
| consistent test ordering. (Contributed by Martin Melin and Jeff Ramnani in |
| :issue:`16709`.) |
| |
| :class:`~unittest.TestSuite` now drops references to tests as soon as the test |
| has been run, if the test is successful. On Python interpreters that do |
| garbage collection, this allows the tests to be garbage collected if nothing |
| else is holding a reference to the test. It is possible to override this |
| behavior by creating a :class:`~unittest.TestSuite` subclass that defines a |
| custom ``_removeTestAtIndex`` method. (Contributed by Tom Wardill, Matt |
| McClure, and Andrew Svetlov in :issue:`11798`.) |
| |
| A new test assertion context-manager, :meth:`~unittest.TestCase.assertLogs`, |
| will ensure that a given block of code emits a log message using the |
| :mod:`logging` module. By default the message can come from any logger and |
| have a priority of ``INFO`` or higher, but both the logger name and an |
| alternative minimum logging level may be specified. The object returned by the |
| context manager can be queried for the :class:`~logging.LogRecord`\ s and/or |
| formatted messages that were logged. (Contributed by Antoine Pitrou in |
| :issue:`18937`.) |
| |
| Test discovery now works with namespace packages (Contributed by Claudiu Popa |
| in :issue:`17457`.) |
| |
| :mod:`unittest.mock` objects now inspect their specification signatures when |
| matching calls, which means an argument can now be matched by either position |
| or name, instead of only by position. (Contributed by Antoine Pitrou in |
| :issue:`17015`.) |
| |
| :func:`~mock.mock_open` objects now have ``readline`` and ``readlines`` |
| methods. (Contributed by Toshio Kuratomi in :issue:`17467`.) |
| |
| |
| venv |
| ---- |
| |
| :mod:`venv` now includes activation scripts for the ``csh`` and ``fish`` |
| shells (Contributed by Andrew Svetlov in :issue:`15417`.) |
| |
| :class:`~venv.EnvBuilder` and the :func:`~venv.create` convenience function |
| take a new keyword argument *with_pip*, which defaults to ``False``, that |
| controls whether or not :class:`~venv.EnvBuilder` ensures that ``pip`` is |
| installed in the virtual environment. (Contributed by Nick Coghlan in |
| :issue:`19552` as part of the :pep:`453` implementation.) |
| |
| |
| wave |
| ---- |
| |
| The :meth:`~wave.getparams` method now returns a namedtuple rather than a |
| plain tuple. (Contributed by Claudiu Popa in :issue:`17487`.) |
| |
| :meth:`wave.open` now supports the context manager protocol. (Contributed |
| by Claudiu Popa in :issue:`17616`.) |
| |
| :mod:`wave` can now :ref:`write output to unseekable files |
| <wave-write-objects>`. (Contributed by David Jones, Guilherme Polo, and Serhiy |
| Storchaka in :issue:`5202`.) |
| |
| The :meth:`~wave.Wave_write.writeframesraw` and |
| :meth:`~wave.Wave_write.writeframes` methods now accept any :term:`bytes-like |
| object`. (Contributed by Serhiy Storchaka in :issue:`8311`.) |
| |
| |
| weakref |
| ------- |
| |
| New :class:`~weakref.WeakMethod` class simulates weak references to bound |
| methods. (Contributed by Antoine Pitrou in :issue:`14631`.) |
| |
| New :class:`~weakref.finalize` class makes it possible to register a callback |
| to be invoked when an object is garbage collected, without needing to |
| carefully manage the lifecycle of the weak reference itself. (Contributed by |
| Richard Oudkerk in :issue:`15528`) |
| |
| The callback, if any, associated with a :class:`~weakref.ref` is now |
| exposed via the :attr:`~weakref.ref.__callback__` attribute. (Contributed |
| by Mark Dickinson in :issue:`17643`.) |
| |
| |
| xml.etree |
| --------- |
| |
| A new parser, :class:`~xml.etree.ElementTree.XMLPullParser`, allows a |
| non-blocking applications to parse XML documents. An example can be |
| seen at :ref:`elementtree-pull-parsing`. (Contributed by Antoine |
| Pitrou in :issue:`17741`.) |
| |
| The :mod:`xml.etree.ElementTree` :func:`~xml.etree.ElementTree.tostring` and |
| :func:`~xml.etree.ElementTree.tostringlist` functions, and the |
| :class:`~xml.etree.ElementTree.ElementTree` |
| :meth:`~xml.etree.ElementTree.ElementTree.write` method, now have a |
| *short_empty_elements* :ref:`keyword-only parameter <keyword-only_parameter>` |
| providing control over whether elements with no content are written in |
| abbreviated (``<tag />``) or expanded (``<tag></tag>``) form. (Contributed by |
| Ariel Poliak and Serhiy Storchaka in :issue:`14377`.) |
| |
| |
| zipfile |
| ------- |
| |
| The :meth:`~zipfile.PyZipFile.writepy` method of the |
| :class:`~zipfile.PyZipFile` class has a new *filterfunc* option that can be |
| used to control which directories and files are added to the archive. For |
| example, this could be used to exclude test files from the archive. |
| (Contributed by Christian Tismer in :issue:`19274`.) |
| |
| The *allowZip64* parameter to :class:`~zipfile.ZipFile` and |
| :class:`~zipfile.PyZipfile` is now ``True`` by default. (Contributed by |
| William Mallard in :issue:`17201`.) |
| |
| |
| |
| CPython Implementation Changes |
| ============================== |
| |
| |
| .. _whatsnew-pep-445: |
| |
| PEP 445: Customization of CPython Memory Allocators |
| --------------------------------------------------- |
| |
| :pep:`445` adds new C level interfaces to customize memory allocation in |
| the CPython interpreter. |
| |
| .. seealso:: |
| |
| :pep:`445` -- Add new APIs to customize Python memory allocators |
| PEP written and implemented by Victor Stinner. |
| |
| |
| .. _whatsnew-pep-442: |
| |
| PEP 442: Safe Object Finalization |
| --------------------------------- |
| |
| :pep:`442` removes the current limitations and quirks of object finalization |
| in CPython. With it, objects with :meth:`__del__` methods, as well as |
| generators with :keyword:`finally` clauses, can be finalized when they are |
| part of a reference cycle. |
| |
| As part of this change, module globals are no longer forcibly set to |
| :const:`None` during interpreter shutdown in most cases, instead relying |
| on the normal operation of the cyclic garbage collector. This avoids a |
| whole class of interpreter-shutdown-time errors, usually involving |
| ``__del__`` methods, that have plagued Python since the cyclic GC |
| was first introduced. |
| |
| .. seealso:: |
| |
| :pep:`442` -- Safe object finalization |
| PEP written and implemented by Antoine Pitrou. |
| |
| |
| .. _whatsnew-pep-456: |
| |
| PEP 456: Secure and Interchangeable Hash Algorithm |
| -------------------------------------------------- |
| |
| :pep:`456` follows up on earlier security fix work done on Python's hash |
| algorithm to address certain DOS attacks to which public facing APIs backed by |
| dictionary lookups may be subject. (See :issue:`14621` for the start of the |
| current round of improvements.) The PEP unifies CPython's hash code to make it |
| easier for a packager to substitute a different hash algorithm, and switches |
| Python's default implementation to a SipHash implementation on platforms that |
| have a 64 bit data type. Any performance differences in comparison with the |
| older FNV algorithm are trivial. |
| |
| The PEP adds additional fields to the :func:`sys.hash_info` struct sequence to |
| describe the hash algorithm in use by the currently executing binary. Otherwise, |
| the PEP does not alter any existing CPython APIs. |
| |
| |
| .. _whatsnew-pep-436: |
| |
| PEP 436: Argument Clinic |
| ------------------------ |
| |
| "Argument Clinic" (:pep:`436`) is now part of the CPython build process |
| and can be used to simplify the process of defining and maintaining |
| accurate signatures for builtins and standard library extension modules |
| implemented in C. |
| |
| Some standard library extension modules have been converted to use Argument |
| Clinic in Python 3.4, and :mod:`pydoc` and :mod:`inspect` have been updated |
| accordingly. |
| |
| It is expected that signature metadata for programmatic introspection will |
| be added to additional callables implemented in C as part of Python 3.4 |
| maintenance releases. |
| |
| .. note:: |
| The Argument Clinic PEP is not fully up to date with the state of the |
| implementation. This has been deemed acceptable by the release manager |
| and core development team in this case, as Argument Clinic will not |
| be made available as a public API for third party use in Python 3.4. |
| |
| .. seealso:: |
| |
| :pep:`436` -- The Argument Clinic DSL |
| PEP written and implemented by Larry Hastings. |
| |
| |
| Other Build and C API Changes |
| ----------------------------- |
| |
| * The new :c:func:`PyType_GetSlot` function has been added to the stable ABI, |
| allowing retrieval of function pointers from named type slots when using |
| the limited API. (Contributed by Martin von Löwis in :issue:`17162`) |
| |
| * The new :c:func:`Py_SetStandardStreamEncoding` pre-initialization API |
| allows applications embedding the CPython interpreter to reliably force |
| a particular encoding and error handler for the standard streams |
| (Contributed by Bastien Montagne and Nick Coghlan in :issue:`16129`) |
| |
| * Most Python C APIs that don't mutate string arguments are now correctly |
| marked as accepting ``const char *`` rather than ``char *`` (Contributed |
| by Serhiy Storchaka in :issue:`1772673`). |
| |
| * A new shell version of ``python-config`` can be used even when a python |
| interpreter is not available (for example, in cross compilation scenarios). |
| |
| * :c:func:`PyUnicode_FromFormat` now supports width and precision |
| specifications for ``%s``, ``%A``, ``%U``, ``%V``, ``%S``, and ``%R``. |
| (Contributed by Ysj Ray and Victor Stinner in :issue:`7330`.) |
| |
| * New function :c:func:`PyStructSequence_InitType2` supplements the |
| existing :c:func:`PyStructSequence_InitType` function. The difference |
| is that it returns ``0`` on success and ``-1`` on failure. |
| |
| * The CPython source can now be compiled using the address sanity checking |
| features of recent versions of GCC and clang: the false alarms in the small |
| object allocator have been silenced. (Contributed by Dhiru Kholia in |
| :issue:`18596`.) |
| |
| * The Windows build now uses `Address Space Layout Randomization |
| <http://en.wikipedia.org/wiki/ASLR>`_ and `Data Execution Prevention |
| <http://en.wikipedia.org/wiki/Data_Execution_Prevention>`_. (Contributed by |
| Christian Heimes in :issue:`16632`.) |
| |
| * New function :c:func:`PyObject_LengthHint` is the C API equivalent |
| of :func:`operator.length_hint`. (Contributed by Armin Ronacher in |
| :issue:`16148`.) |
| |
| |
| .. _other-improvements-3.4: |
| |
| Other Improvements |
| ------------------ |
| |
| .. _whatsnew-isolated-mode: |
| |
| * The :ref:`python <using-on-cmdline>` command has a new :ref:`option |
| <using-on-misc-options>`, ``-I``, which causes it to run in "isolated mode", |
| which means that :data:`sys.path` contains neither the script's directory nor |
| the user's ``site-packages`` directory, and all :envvar:`PYTHON*` environment |
| variables are ignored (it implies both ``-s`` and ``-E``). Other |
| restrictions may also be applied in the future, with the goal being to |
| isolate the execution of a script from the user's environment. This is |
| appropriate, for example, when Python is used to run a system script. On |
| most POSIX systems it can and should be used in the ``#!`` line of system |
| scripts. (Contributed by Christian Heimes in :issue:`16499`.) |
| |
| * Tab-completion is now enabled by default in the interactive interpreter |
| on systems that support :mod:`readline`. History is also enabled by default, |
| and is written to (and read from) the file :file:`~/.python-history`. |
| (Contributed by Antoine Pitrou and Éric Araujo in :issue:`5845`.) |
| |
| * Invoking the Python interpreter with ``--version`` now outputs the version to |
| standard output instead of standard error (:issue:`18338`). Similar changes |
| were made to :mod:`argparse` (:issue:`18920`) and other modules that have |
| script-like invocation capabilities (:issue:`18922`). |
| |
| * The CPython Windows installer now adds ``.py`` to the :envvar:`PATHEXT` |
| variable when extensions are registered, allowing users to run a python |
| script at the windows command prompt by just typing its name without the |
| ``.py`` extension. (Contributed by Paul Moore in :issue:`18569`.) |
| |
| * A new ``make`` target `coverage-report |
| <http://docs.python.org/devguide/coverage.html#measuring-coverage-of-c-code-with-gcov-and-lcov>`_ |
| will build python, run the test suite, and generate an HTML coverage report |
| for the C codebase using ``gcov`` and `lcov |
| <http://ltp.sourceforge.net/coverage/lcov.php>`_. |
| |
| * The ``-R`` option to the :ref:`python regression test suite <regrtest>` now |
| also checks for memory allocation leaks, using |
| :func:`sys.getallocatedblocks()`. (Contributed by Antoine Pitrou in |
| :issue:`13390`). |
| |
| * ``python -m`` now works with namespace packages. |
| |
| * The :mod:`stat` module is now implemented in C, which means it gets the |
| values for its constants from the C header files, instead of having the |
| values hard-coded in the python module as was previously the case. |
| |
| * Loading multiple python modules from a single OS module (``.so``, ``.dll``) |
| now works correctly (previously it silently returned the first python |
| module in the file). (Contributed by Václav Šmilauer in :issue:`16421`.) |
| |
| * A new opcode, :opcode:`LOAD_CLASSDEREF`, has been added to fix a bug in the |
| loading of free variables in class bodies that could be triggered by certain |
| uses of :ref:`__prepare__ <prepare>`. (Contributed by Benjamin Peterson in |
| :issue:`17853`.) |
| |
| * A number of MemoryError-related crashes were identified and fixed by Victor |
| Stinner using his :pep:`445`-based ``pyfailmalloc`` tool (:issue:`18408`, |
| :issue:`18520`). |
| |
| * The :ref:`pyvenv <scripts-pyvenv>` command now accepts a ``--copies`` option |
| to use copies rather than symlinks even on systems where symlinks are the |
| default. (Contributed by Vinay Sajip in :issue:`18807`.) |
| |
| * The :ref:`pyvenv <scripts-pyvenv>` command also accepts a ``--without-pip`` |
| option to suppress the otherwise-automatic bootstrapping of pip into |
| the virtual environment. (Contributed by Nick Coghlan in :issue:`19552` |
| as part of the :pep:`453` implementation.) |
| |
| * The encoding name is now optional in the value set for the |
| :envvar:`PYTHONIOENCODING` environment variable. This makes it possible to |
| set just the error handler, without changing the default encoding. |
| (Contributed by Serhiy Storchaka in :issue:`18818`.) |
| |
| * The :mod:`bz2`, :mod:`lzma`, and :mod:`gzip` module ``open`` functions now |
| support ``x`` (exclusive creation) mode. (Contributed by Tim Heaney and |
| Vajrasky Kok in :issue:`19201`, :issue:`19222`, and :issue:`19223`.) |
| |
| |
| Significant Optimizations |
| ------------------------- |
| |
| * The UTF-32 decoder is now 3x to 4x faster. (Contributed by Serhiy Storchaka |
| in :issue:`14625`.) |
| |
| * The cost of hash collisions for sets is now reduced. Each hash table |
| probe now checks a series of consecutive, adjacent key/hash pairs before |
| continuing to make random probes through the hash table. This exploits |
| cache locality to make collision resolution less expensive. |
| The collision resolution scheme can be described as a hybrid of linear |
| probing and open addressing. The number of additional linear probes |
| defaults to nine. This can be changed at compile-time by defining |
| LINEAR_PROBES to be any value. Set LINEAR_PROBES=0 to turn-off |
| linear probing entirely. (Contributed by Raymond Hettinger in |
| :issue:`18771`.) |
| |
| * The interpreter starts about 30% faster. A couple of measures lead to the |
| speedup. The interpreter loads fewer modules on startup, e.g. the :mod:`re`, |
| :mod:`collections` and :mod:`locale` modules and their dependencies are no |
| longer imported by default. The marshal module has been improved to load |
| compiled Python code faster. (Contributed by Antoine Pitrou, Christian |
| Heimes and Victor Stinner in :issue:`19219`, :issue:`19218`, :issue:`19209`, |
| :issue:`19205` and :issue:`9548`) |
| |
| * :class:`bz2.BZ2File` is now as fast or faster than the Python2 version for |
| most cases. :class:`lzma.LZMAFile` has also been optimized. (Contributed by |
| Serhiy Storchaka and Nadeem Vawda in :issue:`16034`.) |
| |
| * :func:`random.getrandbits` is 20%-40% faster for small integers (the most |
| common use case). (Contributed by Serhiy Storchaka in :issue:`16674`). |
| |
| * By taking advantage of the new storage format for strings, pickling of |
| strings is now significantly faster. (Contributed by Victor Stinner and |
| Antoine Pitrou in :issue:`15596`.) |
| |
| * A performance issue in :meth:`io.FileIO.readall` has been solved. This |
| particularly affects Windows, and significantly speeds up the case of piping |
| significant amounts of data through :mod:`subprocess`. (Contributed |
| by Richard Oudkerk in :issue:`15758`.) |
| |
| * :func:`html.escape` is now 10x faster. (Contributed by Matt Bryant in |
| :issue:`18020`.) |
| |
| * On Windows, the native ``VirtualAlloc`` is now used instead of the CRT |
| ``malloc`` in ``obmalloc``. Artificial benchmarks show about a 3% memory |
| savings. |
| |
| * :func:`os.urandom` now uses a lazily-opened persistent file descriptor |
| so as to avoid using many file descriptors when run in parallel from |
| multiple threads. (Contributed by Antoine Pitrou in :issue:`18756`.) |
| |
| |
| .. _deprecated-3.4: |
| |
| Deprecated |
| ========== |
| |
| This section covers various APIs and other features that have been deprecated |
| in Python 3.4, and will be removed in Python 3.5 or later. In most (but not |
| all) cases, using the deprecated APIs will produce a :exc:`DeprecationWarning` |
| when the interpreter is run with deprecation warnings enabled (for example, by |
| using ``-Wd``). |
| |
| |
| Deprecations in the Python API |
| ------------------------------ |
| |
| * As mentioned in :ref:`whatsnew-pep-451`, a number of :mod:`importilb` |
| methods and functions are deprecated: :meth:`importlib.find_loader` is |
| replaced by :func:`importlib.util.find_spec`; |
| :meth:`importlib.machinery.PathFinder.find_module` is replaced by |
| :meth:`importlib.machinery.PathFinder.find_spec`; |
| :meth:`importlib.abc.MetaPathFinder.find_module` is replaced by |
| :meth:`importlib.abc.MetaPathFinder.find_spec`; |
| :meth:`importlib.abc.PathEntryFinder.find_loader` and |
| :meth:`~importlib.abc.PathEntryFinder.find_module` are replaced by |
| :meth:`importlib.abc.PathEntryFinder.find_spec`; all of the ``xxxLoader`` ABC |
| ``load_module`` methods (:meth:`importlib.abc.Loader.load_module`, |
| :meth:`importlib.abc.InspectLoader.load_module`, |
| :meth:`importlib.abc.FileLoader.load_module`, |
| :meth:`importlib.abc.SourceLoader.load_module`) should no longer be |
| implemented, instead loaders should implement an |
| ``exec_module`` method |
| (:meth:`importlib.abc.Loader.exec_module`, |
| :meth:`importlib.abc.InspectLoader.exec_module` |
| :meth:`importlib.abc.SourceLoader.exec_module`) and let the import system |
| take care of the rest; and |
| :meth:`importlib.abc.Loader.module_repr`, |
| :meth:`importlib.util.module_for_loader`, :meth:`importlib.util.set_loader`, |
| and :meth:`importlib.util.set_package` are no longer needed because their |
| functions are now handled automatically by the import system. |
| |
| * The :mod:`imp` module is pending deprecation. To keep compatibility with |
| Python 2/3 code bases, the module's removal is currently not scheduled. |
| |
| * The :mod:`formatter` module is pending deprecation and is slated for removal |
| in Python 3.6. |
| |
| * ``MD5`` as the default *digestmod* for the :func:`hmac.new` function is |
| deprecated. Python 3.6 will require an explicit digest name or constructor as |
| *digestmod* argument. |
| |
| * The internal ``Netrc`` class in the :mod:`ftplib` module has been documented |
| as deprecated in its docstring for quite some time. It now emits a |
| :exc:`DeprecationWarning` and will be removed completely in Python 3.5. |
| |
| * The undocumented *endtime* argument to :meth:`subprocess.Popen.wait` should |
| not have been exposed and is hopefully not in use; it is deprecated and |
| will mostly likely be removed in Python 3.5. |
| |
| * The *strict* argument of :class:`~html.parser.HTMLParser` is deprecated. |
| |
| * The :mod:`plistlib` :func:`~plistlib.readPlist`, |
| :func:`~plistlib.writePlist`, :func:`~plistlib.readPlistFromBytes`, and |
| :func:`~plistlib.writePlistToBytes` functions are deprecated in favor of the |
| corresponding new functions :func:`~plistlib.load`, :func:`~plistlib.dump`, |
| :func:`~plistlib.loads`, and :func:`~plistlib.dumps`. :func:`~plistlib.Data` |
| is deprecated in favor of just using the :class:`bytes` constructor. |
| |
| * The :mod:`sysconfig` key ``SO`` is deprecated, it has been replaced by |
| ``EXT_SUFFIX``. |
| |
| * The ``U`` mode accepted by various ``open`` functions is deprecated. |
| In Python3 it does not do anything useful, and should be replaced by |
| appropriate uses of :class:`io.TextIOWrapper` (if needed) and its *newline* |
| argument. |
| |
| * The *parser* argument of :func:`xml.etree.ElementTree.iterparse` has |
| been deprecated, as has the *html* argument of |
| :func:`~xml.etree.ElementTree.XMLParser`. To prepare for the removal of the |
| latter, all arguments to ``XMLParser`` should be passed by keyword. |
| |
| |
| Deprecated Features |
| ------------------- |
| |
| * Running :ref:`idle` with the ``-n`` flag (no subprocess) is deprecated. |
| However, the feature will not be removed until :issue:`18823` is resolved. |
| |
| * The site module adding a "site-python" directory to sys.path, if it |
| exists, is deprecated (:issue:`19375`). |
| |
| |
| |
| Removed |
| ======= |
| |
| |
| Operating Systems No Longer Supported |
| ------------------------------------- |
| |
| Support for the following operating systems has been removed from the source |
| and build tools: |
| |
| * OS/2 (:issue:`16135`). |
| * Windows 2000 (changeset e52df05b496a). |
| * Windows systems where ``COMSPEC`` points to ``command.com`` (:issue:`14470`). |
| * VMS (:issue:`16136`). |
| |
| |
| API and Feature Removals |
| ------------------------ |
| |
| The following obsolete and previously deprecated APIs and features have been |
| removed: |
| |
| * The unmaintained ``Misc/TextMate`` and ``Misc/vim`` directories have been |
| removed (see the `devguide <http://docs.python.org/devguide>`_ |
| for suggestions on what to use instead). |
| |
| * The ``SO`` makefile macro is removed (it was replaced by the |
| ``SHLIB_SUFFIX`` and ``EXT_SUFFIX`` macros) (:issue:`16754`). |
| |
| * The ``PyThreadState.tick_counter`` field has been removed; its value has |
| been meaningless since Python 3.2, when the "new GIL" was introduced |
| (:issue:`19199`). |
| |
| * ``PyLoader`` and ``PyPycLoader`` have been removed from :mod:`importlib`. |
| (Contributed by Taras Lyapun in :issue:`15641`.) |
| |
| * The *strict* argument to :class:`~http.client.HTTPConnection` and |
| :class:`~http.client.HTTPSConnection` has been removed. HTTP 0.9-style |
| "Simple Responses" are no longer supported. |
| |
| * The deprecated :mod:`urllib.request.Request` getter and setter methods |
| ``add_data``, ``has_data``, ``get_data``, ``get_type``, ``get_host``, |
| ``get_selector``, ``set_proxy``, ``get_origin_req_host``, and |
| ``is_unverifiable`` have been removed (use direct attribute access instead). |
| |
| * Support for loading the deprecated ``TYPE_INT64`` has been removed from |
| :mod:`marshal`. (Contributed by Dan Riti in :issue:`15480`.) |
| |
| * :class:`inspect.Signature`: positional-only parameters are now required |
| to have a valid name. |
| |
| * :meth:`object.__format__` no longer accepts non-empty format strings, it now |
| raises a :exc:`TypeError` instead. Using a non-empty string has been |
| deprecated since Python 3.2. This change has been made to prevent a |
| situation where previously working (but incorrect) code would start failing |
| if an object gained a __format__ method, which means that your code may now |
| raise a :exc:`TypeError` if you are using an ``'s'`` format code with objects |
| that do not have a __format__ method that handles it. See :issue:`7994` for |
| background. |
| |
| * :meth:`difflib.SequenceMatcher.isbjunk` and |
| :meth:`difflib.SequenceMatcher.isbpopular` were deprecated in 3.2, and have |
| now been removed: use ``x in sm.bjunk`` and |
| ``x in sm.bpopular``, where *sm* is a :class:`~difflib.SequenceMatcher` object |
| (:issue:`13248`). |
| |
| |
| Code Cleanups |
| ------------- |
| |
| * The unused and undocumented internal ``Scanner`` class has been removed from |
| the :mod:`pydoc` module. |
| |
| * The private and effectively unused ``_gestalt`` module has been removed, |
| along with the private :mod:`platform` functions ``_mac_ver_lookup``, |
| ``_mac_ver_gstalt``, and ``_bcd2str``, which would only have ever been called |
| on badly broken OSX systems (see :issue:`18393`). |
| |
| * The hardcoded copies of certain :mod:`stat` constants that were included in |
| the :mod:`tarfile` module namespace have been removed. |
| |
| |
| |
| Porting to Python 3.4 |
| ===================== |
| |
| This section lists previously described changes and other bugfixes |
| that may require changes to your code. |
| |
| |
| Changes in 'python' Command Behavior |
| ------------------------------------ |
| |
| * In a posix shell, setting the :envvar:`PATH` environment variable to |
| an empty value is equivalent to not setting it at all. However, setting |
| :envvar:`PYTHONPATH` to an empty value was *not* equivalent to not setting it |
| at all: setting :envvar:`PYTHONPATH` to an empty value was equivalent to |
| setting it to ``.``, which leads to confusion when reasoning by analogy to |
| how :envvar:`PATH` works. The behavior now conforms to the posix convention |
| for :envvar:`PATH`. |
| |
| * The [X refs, Y blocks] output of a debug (``--with-pydebug``) build of the |
| CPython interpreter is now off by default. It can be re-enabled using the |
| ``-X showrefcount`` option. (Contributed by Ezio Melotti in :issue:`17323`.) |
| |
| * The python command and most stdlib scripts (as well as :mod:`argparse`) now |
| output ``--version`` information to ``stdout`` instead of ``stderr`` (for |
| issue list see :ref:`other-improvements-3.4` above). |
| |
| |
| Changes in the Python API |
| ------------------------- |
| |
| * The ABCs defined in :mod:`importlib.abc` now either raise the appropriate |
| exception or return a default value instead of raising |
| :exc:`NotImplementedError` blindly. This will only affect code calling |
| :func:`super` and falling through all the way to the ABCs. For compatibility, |
| catch both :exc:`NotImplementedError` or the appropriate exception as needed. |
| |
| * The module type now initializes the :attr:`__package__` and :attr:`__loader__` |
| attributes to ``None`` by default. To determine if these attributes were set |
| in a backwards-compatible fashion, use e.g. |
| ``getattr(module, '__loader__', None) is not None``. (:issue:`17115`.) |
| |
| * :meth:`importlib.util.module_for_loader` now sets ``__loader__`` and |
| ``__package__`` unconditionally to properly support reloading. If this is not |
| desired then you will need to set these attributes manually. You can use |
| :func:`importlib.util.module_to_load` for module management. |
| |
| * Import now resets relevant attributes (e.g. ``__name__``, ``__loader__``, |
| ``__package__``, ``__file__``, ``__cached__``) unconditionally when reloading. |
| Note that this restores a pre-3.3 behavior in that it means a module is |
| re-found when re-loaded (:issue:`19413`). |
| |
| * Frozen packages no longer set ``__path__`` to a list containing the package |
| name, they now set it to an empty list. The previous behavior could cause |
| the import system to do the wrong thing on submodule imports if there was |
| also a directory with the same name as the frozen package. The correct way |
| to determine if a module is a package or not is to use ``hasattr(module, |
| '__path__')`` (:issue:`18065`). |
| |
| * Frozen modules no longer define a ``__file__`` attribute. It's semantically |
| incorrect for frozen modules to set the attribute as they are not loaded from |
| any explicit location. If you must know that a module comes from frozen code |
| then you can see if the module's ``__spec__.location`` is set to ``'frozen'``, |
| check if the loader is a subclass of |
| :class:`importlib.machinery.FrozenImporter`, |
| or if Python 2 compatibility is necessary you can use :func:`imp.is_frozen`. |
| |
| * :func:`py_compile.compile` now raises :exc:`FileExistsError` if the file path |
| it would write to is a symlink or a non-regular file. This is to act as a |
| warning that import will overwrite those files with a regular file regardless |
| of what type of file path they were originally. |
| |
| * :meth:`importlib.abc.SourceLoader.get_source` no longer raises |
| :exc:`ImportError` when the source code being loaded triggers a |
| :exc:`SyntaxError` or :exc:`UnicodeDecodeError`. As :exc:`ImportError` is |
| meant to be raised only when source code cannot be found but it should, it was |
| felt to be over-reaching/overloading of that meaning when the source code is |
| found but improperly structured. If you were catching ImportError before and |
| wish to continue to ignore syntax or decoding issues, catch all three |
| exceptions now. |
| |
| * :func:`functools.update_wrapper` and :func:`functools.wraps` now correctly |
| set the ``__wrapped__`` attribute to the function being wrapped, even if |
| that function also had its ``__wrapped__`` attribute set. This means |
| ``__wrapped__`` attributes now correctly link a stack of decorated |
| functions rather than every ``__wrapped__`` attribute in the chain |
| referring to the innermost function. Introspection libraries that |
| assumed the previous behaviour was intentional can use |
| :func:`inspect.unwrap` to access the first function in the chain that has |
| no ``__wrapped__`` attribute. |
| |
| * :func:`inspect.getfullargspec` has been reimplemented on top of |
| :func:`inspect.signature` and hence handles a much wider variety of callable |
| objects than it did in the past. It is expected that additional builtin and |
| extension module callables will gain signature metadata over the course of |
| the Python 3.4 series. Code that assumes that |
| :func:`inspect.getfullargspec` will fail on non-Python callables may need |
| to be adjusted accordingly. |
| |
| * :class:`importlib.machinery.PathFinder` now passes on the current working |
| directory to objects in :data:`sys.path_hooks` for the empty string. This |
| results in :data:`sys.path_importer_cache` never containing ``''``, thus |
| iterating through :data:`sys.path_importer_cache` based on :data:`sys.path` |
| will not find all keys. A module's ``__file__`` when imported in the current |
| working directory will also now have an absolute path, including when using |
| ``-m`` with the interpreter (except for ``__main__.__file__`` when a script |
| has been executed directly using a relative path) (Contributed by Brett |
| Cannon in :issue:`18416`). is specified on the command-line) |
| (:issue:`18416`). |
| |
| * The removal of the *strict* argument to :class:`~http.client.HTTPConnection` |
| and :class:`~http.client.HTTPSConnection` changes the meaning of the |
| remaining arguments if you are specifying them positionally rather than by |
| keyword. If you've been paying attention to deprecation warnings your code |
| should already be specifying any additional arguments via keywords. |
| |
| * Strings between ``from __future__ import ...`` statements now *always* raise |
| a :exc:`SyntaxError`. Previously if there was no leading docstring, an |
| interstitial string would sometimes be ignored. This brings CPython into |
| compliance with the language spec; Jython and PyPy already were. |
| (:issue:`17434`). |
| |
| * :meth:`ssl.SSLSocket.getpeercert` and :meth:`ssl.SSLSocket.do_handshake` |
| now raise an :exc:`OSError` with ``ENOTCONN`` when the ``SSLSocket`` is not |
| connected, instead of the previous behavior of raising an |
| :exc:`AttributError`. In addition, :meth:`~ssl.SSLSocket.getpeercert` |
| will raise a :exc:`ValueError` if the handshake has not yet been done. |
| |
| * :func:`base64.b32decode` now raises a :exc:`binascii.Error` when the |
| input string contains non-b32-alphabet characters, instead of a |
| :exc:`TypeError`. This particular :exc:`TypeError` was missed when the other |
| :exc:`TypeError`\ s were converted. (Contributed by Serhiy Storchaka in |
| :issue:`18011`.) Note: this change was also inadvertently applied in Python |
| 3.3.3. |
| |
| * The :attr:`~cgi.FieldStorage.file` attribute is now automatically closed when |
| the creating :class:`cgi.FieldStorage` instance is garbage collected. If you |
| were pulling the file object out separately from the :class:`cgi.FieldStorage` |
| instance and not keeping the instance alive, then you should either store the |
| entire :class:`cgi.FieldStorage` instance or read the contents of the file |
| before the :class:`cgi.FieldStorage` instance is garbage collected. |
| |
| * Calling ``read`` or ``write`` on a closed SSL socket now raises an |
| informative :exc:`ValueError` rather than the previous more mysterious |
| :exc:`AttributeError` (:issue:`9177`). |
| |
| * :meth:`slice.indices` no longer produces an :exc:`OverflowError` for huge |
| values. As a consequence of this fix, :meth:`slice.indices` now raises a |
| :exc:`ValueError` if given a negative length; previously it returned nonsense |
| values (:issue:`14794`). |
| |
| * The :class:`complex` constructor, unlike the :mod:`cmath` functions, was |
| incorrectly accepting :class:`float` values if an object's ``__complex__`` |
| special method returned one. This now raises a :exc:`TypeError`. |
| (:issue:`16290`.) |
| |
| * The :class:`int` constructor in 3.2 and 3.3 erroneously accepts :class:`float` |
| values for the *base* parameter. It is unlikely anyone was doing this, but |
| if so, it will now raise a :exc:`TypeError` (:issue:`16772`). |
| |
| * Defaults for keyword-only arguments are now evaluated *after* defaults for |
| regular keyword arguments, instead of before. Hopefully no one wrote any |
| code that depends on the previous buggy behavior (:issue:`16967`). |
| |
| * Stale thread states are now cleared after :func:`~os.fork`. This may cause |
| some system resources to be released that previously were incorrectly kept |
| perpetually alive (for example, database connections kept in thread-local |
| storage). (:issue:`17094`.) |
| |
| * Parameter names in ``__annotations__`` dicts are now mangled properly, |
| similarly to ``__kwdefaults__``. (Contributed by Yury Selivanov in |
| :issue:`20625`). |
| |
| * :attr:`hashlib.hash.name` now always returns the identifier in lower case. |
| Previously some builtin hashes had uppercase names, but now that it is a |
| formal public interface the naming has been made consistent (:issue:`18532`). |
| |
| * Because :mod:`unittest.TestSuite` now drops references to tests after they |
| are run, test harnesses that re-use a :class:`~unittest.TestSuite` to re-run |
| a set of tests may fail. Test suites should not be re-used in this fashion |
| since it means state is retained between test runs, breaking the test |
| isolation that :mod:`unittest` is designed to provide. However, if the lack |
| of isolation is considered acceptable, the old behavior can be restored by |
| creating a :mod:`~unittest.TestSuite` subclass that defines a |
| ``_removeTestAtIndex`` method that does nothing (see |
| :meth:`.TestSuite.__iter__`) (:issue:`11798`). |
| |
| * :mod:`unittest` now uses :mod:`argparse` for command line parsing. There are |
| certain invalid command forms that used to work that are no longer allowed; |
| in theory this should not cause backward compatibility issues since the |
| disallowed command forms didn't make any sense and are unlikely to be in use. |
| |
| * The :func:`re.split`, :func:`re.findall`, and :func:`re.sub` functions, and |
| the :meth:`~re.match.group` and :meth:`~re.match.groups` methods of |
| ``match`` objects now always return a *bytes* object when the string |
| to be matched is a :term:`bytes-like object`. Previously the return type |
| matched the input type, so if your code was depending on the return value |
| being, say, a ``bytearray``, you will need to change your code. |
| |
| * :mod:`audioop` functions now raise an error immediately if passed string |
| input, instead of failing randomly later on (:issue:`16685`). |
| |
| * The new *convert_charrefs* argument to :class:`~html.parser.HTMLParser` |
| currently defaults to ``False`` for backward compatibility, but will |
| eventually be changed to default to ``True``. It is recommended that you add |
| this keyword, with the appropriate value, to any |
| :class:`~html.parser.HTMLParser` calls in your code (:issue:`13633`). |
| |
| * Since the *digestmod* argument to the :func:`hmac.new` function will in the |
| future have no default, all calls to :func:`hmac.new` should be changed to |
| explicitly specify a *digestmod* (:issue:`17276`). |
| |
| * Calling :func:`sysconfig.get_config_var` with the ``SO`` key, or looking |
| ``SO`` up in the results of a call to :func:`sysconfig.get_config_vars` |
| is deprecated. This key should be replaced by ``EXT_SUFFIX`` or |
| ``SHLIB_SUFFIX``, depending on the context (:issue:`19555`). |
| |
| * Any calls to ``open`` functions that specify ``U`` should be modified. |
| ``U`` is ineffective in Python3 and will eventually raise an error if used. |
| Depending on the function, the equivalent of its old Python2 behavior can be |
| achieved using either a *newline* argument, or if necessary by wrapping the |
| stream in :mod:`~io.TextIOWrapper` to use its *newline* argument |
| (:issue:`15204`). |
| |
| * If you use :ref:`pyvenv <scripts-pyvenv>` in a script and desire that pip |
| *not* be installed, you must add ``--without-pip`` to your command |
| invocation. |
| |
| * The default behavior of :func:`json.dump` and :func:`json.dumps` when |
| an indent is specified has changed: it no longer produces trailing |
| spaces after the item separating commas at the ends of lines. This |
| will matter only if you have tests that are doing white-space-sensitive |
| comparisons of such output (:issue:`16333`). |
| |
| * :mod:`doctest` now looks for doctests in extension module ``__doc__`` |
| strings, so if your doctest test discovery includes extension modules that |
| have things that look like doctests in them you may see test failures you've |
| never seen before when running your tests (:issue:`3158`). |
| |
| * The :mod:`collections.abc` module has been slightly refactored as |
| part of the Python startup improvements. As a consequence of this, it is no |
| longer the case that importing :mod:`collections` automatically imports |
| :mod:`collections.abc`. If your program depended on the (undocumented) |
| implicit import, you will need to add an explicit ``import collections.abc`` |
| (:issue:`20784`). |
| |
| |
| Changes in the C API |
| -------------------- |
| |
| * :c:func:`PyEval_EvalFrameEx`, :c:func:`PyObject_Repr`, and |
| :c:func:`PyObject_Str`, along with some other internal C APIs, now include |
| a debugging assertion that ensures they are not used in situations where |
| they may silently discard a currently active exception. In cases where |
| discarding the active exception is expected and desired (for example, |
| because it has already been saved locally with :c:func:`PyErr_Fetch` or |
| is being deliberately replaced with a different exception), an explicit |
| :c:func:`PyErr_Clear` call will be needed to avoid triggering the |
| assertion when invoking these operations (directly or indirectly) and |
| running against a version of Python that is compiled with assertions |
| enabled. |
| |
| * :c:func:`PyErr_SetImportError` now sets :exc:`TypeError` when its **msg** |
| argument is not set. Previously only ``NULL`` was returned with no exception |
| set. |
| |
| * The result of the :c:data:`PyOS_ReadlineFunctionPointer` callback must |
| now be a string allocated by :c:func:`PyMem_RawMalloc` or |
| :c:func:`PyMem_RawRealloc`, or *NULL* if an error occurred, instead of a |
| string allocated by :c:func:`PyMem_Malloc` or :c:func:`PyMem_Realloc` |
| (:issue:`16742`) |
| |
| * :c:func:`PyThread_set_key_value` now always set the value. In Python |
| 3.3, the function did nothing if the key already exists (if the current |
| value is a non-NULL pointer). |
| |
| * The ``f_tstate`` (thread state) field of the :c:type:`PyFrameObject` |
| structure has been removed to fix a bug: see :issue:`14432` for the |
| rationale. |