bpo-43795: PEP 652 user documentation (GH-25668) (GH-26034)
- Reformat the C API and ABI Versioning page (and extend/clarify a bit)
- Rewrite the stable ABI docs into a general text on C API Compatibility
- Add a list of Limited API contents, and notes for the individual items.
- Replace `Include/README.rst` with a link to a devguide page with the same info
(cherry picked from commit b05955d6f5f149523b5855a335444b7c6324bdb7)
Co-authored-by: Petr Viktorin <encukou@gmail.com>
Co-authored-by: Petr Viktorin <encukou@gmail.com>
diff --git a/Doc/c-api/apiabiversion.rst b/Doc/c-api/apiabiversion.rst
index b8a8f2f..04050f7 100644
--- a/Doc/c-api/apiabiversion.rst
+++ b/Doc/c-api/apiabiversion.rst
@@ -6,34 +6,57 @@
API and ABI Versioning
***********************
-``PY_VERSION_HEX`` is the Python version number encoded in a single integer.
+CPython exposes its version number in the following macros.
+Note that these correspond to the version code is **built** with,
+not necessarily the version used at **run time**.
-For example if the ``PY_VERSION_HEX`` is set to ``0x030401a2``, the underlying
-version information can be found by treating it as a 32 bit number in
-the following manner:
+See :ref:`stable` for a discussion of API and ABI stability across versions.
- +-------+-------------------------+------------------------------------------------+
- | Bytes | Bits (big endian order) | Meaning |
- +=======+=========================+================================================+
- | ``1`` | ``1-8`` | ``PY_MAJOR_VERSION`` (the ``3`` in |
- | | | ``3.4.1a2``) |
- +-------+-------------------------+------------------------------------------------+
- | ``2`` | ``9-16`` | ``PY_MINOR_VERSION`` (the ``4`` in |
- | | | ``3.4.1a2``) |
- +-------+-------------------------+------------------------------------------------+
- | ``3`` | ``17-24`` | ``PY_MICRO_VERSION`` (the ``1`` in |
- | | | ``3.4.1a2``) |
- +-------+-------------------------+------------------------------------------------+
- | ``4`` | ``25-28`` | ``PY_RELEASE_LEVEL`` (``0xA`` for alpha, |
- | | | ``0xB`` for beta, ``0xC`` for release |
- | | | candidate and ``0xF`` for final), in this |
- | | | case it is alpha. |
- +-------+-------------------------+------------------------------------------------+
- | | ``29-32`` | ``PY_RELEASE_SERIAL`` (the ``2`` in |
- | | | ``3.4.1a2``, zero for final releases) |
- +-------+-------------------------+------------------------------------------------+
+.. c:macro:: PY_MAJOR_VERSION
-Thus ``3.4.1a2`` is hexversion ``0x030401a2``.
+ The ``3`` in ``3.4.1a2``.
+
+.. c:macro:: PY_MINOR_VERSION
+
+ The ``4`` in ``3.4.1a2``.
+
+.. c:macro:: PY_MICRO_VERSION
+
+ The ``1`` in ``3.4.1a2``.
+
+.. c:macro:: PY_RELEASE_LEVEL
+
+ The ``a`` in ``3.4.1a2``.
+ This can be ``0xA`` for alpha, ``0xB`` for beta, ``0xC`` for release
+ candidate or ``0xF`` for final.
+
+.. c:macro:: PY_RELEASE_SERIAL
+
+ The ``2`` in ``3.4.1a2``. Zero for final releases.
+
+.. c:macro:: PY_VERSION_HEX
+
+ The Python version number encoded in a single integer.
+
+ The underlying version information can be found by treating it as a 32 bit
+ number in the following manner:
+
+ +-------+-------------------------+-------------------------+--------------------------+
+ | Bytes | Bits (big endian order) | Meaning | Value for ``3.4.1a2`` |
+ +=======+=========================+=========================+==========================+
+ | 1 | 1-8 | ``PY_MAJOR_VERSION`` | ``0x03`` |
+ +-------+-------------------------+-------------------------+--------------------------+
+ | 2 | 9-16 | ``PY_MINOR_VERSION`` | ``0x04`` |
+ +-------+-------------------------+-------------------------+--------------------------+
+ | 3 | 17-24 | ``PY_MICRO_VERSION`` | ``0x01`` |
+ +-------+-------------------------+-------------------------+--------------------------+
+ | 4 | 25-28 | ``PY_RELEASE_LEVEL`` | ``0xA`` |
+ + +-------------------------+-------------------------+--------------------------+
+ | | 29-32 | ``PY_RELEASE_SERIAL`` | ``0x2`` |
+ +-------+-------------------------+-------------------------+--------------------------+
+
+ Thus ``3.4.1a2`` is hexversion ``0x030401a2`` and ``3.10.0`` is
+ hexversion ``0x030a00f0``.
+
All the given macros are defined in :source:`Include/patchlevel.h`.
-
diff --git a/Doc/c-api/stable.rst b/Doc/c-api/stable.rst
index 9c05cb3..2f67b09 100644
--- a/Doc/c-api/stable.rst
+++ b/Doc/c-api/stable.rst
@@ -2,37 +2,157 @@
.. _stable:
-***********************************
+***************
+C API Stability
+***************
+
+Python's C API is covered by the Backwards Compatibility Policy, :pep:`387`.
+While the C API will change with every minor release (e.g. from 3.9 to 3.10),
+most changes will be source-compatible, typically by only adding new API.
+Changing existing API or removing API is only done after a deprecation period
+or to fix serious issues.
+
+CPython's Application Binary Interface (ABI) is forward- and
+backwards-compatible across a minor release (if these are compiled the same
+way; see :ref:`stable-abi-platform` below).
+So, code compiled for Python 3.10.0 will work on 3.10.8 and vice versa,
+but will need to be compiled separately for 3.9.x and 3.10.x.
+
+Names prefixed by an underscore, such as ``_Py_InternalState``,
+are private API that can change without notice even in patch releases.
+
+
Stable Application Binary Interface
-***********************************
+===================================
-Traditionally, the C API of Python will change with every release. Most changes
-will be source-compatible, typically by only adding API, rather than changing
-existing API or removing API (although some interfaces do get removed after
-being deprecated first).
+Python 3.2 introduced the *Limited API*, a subset of Python's C API.
+Extensions that only use the Limited API can be
+compiled once and work with multiple versions of Python.
+Contents of the Limited API are :ref:`listed below <stable-abi-list>`.
-Unfortunately, the API compatibility does not extend to binary compatibility
-(the ABI). The reason is primarily the evolution of struct definitions, where
-addition of a new field, or changing the type of a field, might not break the
-API, but can break the ABI. As a consequence, extension modules need to be
-recompiled for every Python release (although an exception is possible on Unix
-when none of the affected interfaces are used). In addition, on Windows,
-extension modules link with a specific pythonXY.dll and need to be recompiled to
-link with a newer one.
+To enable this, Python provides a *Stable ABI*: a set of symbols that will
+remain compatible across Python 3.x versions. The Stable ABI contains symbols
+exposed in the Limited API, but also other ones – for example, functions
+necessary to support older versions of the Limited API.
-Since Python 3.2, a subset of the API has been declared to guarantee a stable
-ABI. Extension modules wishing to use this API (called "limited API") need to
-define ``Py_LIMITED_API``. A number of interpreter details then become hidden
-from the extension module; in return, a module is built that works on any 3.x
-version (x>=2) without recompilation.
+(For simplicity, this document talks about *extensions*, but the Limited API
+and Stable ABI work the same way for all uses of the API – for example,
+embedding Python.)
-In some cases, the stable ABI needs to be extended with new functions.
-Extension modules wishing to use these new APIs need to set ``Py_LIMITED_API``
-to the ``PY_VERSION_HEX`` value (see :ref:`apiabiversion`) of the minimum Python
-version they want to support (e.g. ``0x03030000`` for Python 3.3). Such modules
-will work on all subsequent Python releases, but fail to load (because of
-missing symbols) on the older releases.
+.. c:macro:: Py_LIMITED_API
-As of Python 3.2, the set of functions available to the limited API is
-documented in :pep:`384`. In the C API documentation, API elements that are not
-part of the limited API are marked as "Not part of the limited API."
+ Define this macro ``Py_LIMITED_API`` before including ``Python.h`` to
+ opt in to only use the Limited API.
+
+ Defining ``Py_LIMITED_API`` to ``3`` will limit the available API so that
+ the extension will work without recompilation with all Python 3.x releases
+ (x>=2) on the particular :ref:`platform <stable-abi-platform>`.
+
+ Defining ``Py_LIMITED_API`` to a value of :c:data:`PY_VERSION_HEX` will
+ limit the available API so that the extension will work without
+ recompilation with all Python 3 releases from the specified one.
+ This will allow using additional API introduced up to this version,
+ but the extension will lose compatibility with earlier Python versions.
+ Rather than using the ``PY_VERSION_HEX`` macro directly, hardcode a minimum
+ minor version (e.g. ``0x030A0000`` for Python 3.10) for stability when
+ compiling with future Python versions.
+
+On Windows, extensions that use the Stable ABI should be linked against
+``python3.dll`` rather than a version-specific library such as
+``python39.dll``.
+
+On some platforms, Python will look for and load shared library files named
+with the ``abi3`` tag (e.g. ``mymodule.abi3.so``).
+It does not check if such extensions conform to a Stable ABI.
+The user (or their packaging tools) need to ensure that, for example,
+extensions built with the 3.10+ Limited API are not installed for lower
+versions of Python.
+
+All functions in the Stable ABI are present as functions in Python's shared
+library, not solely as macros. This makes them usable from languages that don't
+use the C preprocessor.
+
+
+Limited API Scope and Performance
+---------------------------------
+
+The goal for the Limited API is to allow everything that is possible with the
+full C API, but possibly with a performance penalty.
+
+For example, while :c:func:`PyList_GetItem` is available, its “unsafe” macro
+variant :c:func:`PyList_GET_ITEM` is not.
+The macro can be faster because it can rely on version-specific implementation
+details of the list object.
+
+Without ``Py_LIMITED_API`` defined, some C API functions are inlined or
+replaced by macros.
+Defining ``Py_LIMITED_API`` disables this inlining, allowing stability as
+Python's data structures are improved, but possibly reducing performance.
+
+By leaving out the ``Py_LIMITED_API`` definition, it is possible to compile
+a Limited API extension with a version-specific ABI. This can improve
+performance for that Python version, but will limit compatibility.
+Compiling with ``Py_LIMITED_API`` will then yield an extension that can be
+distributed where a version-specific one is not available – for example,
+for prereleases of an upcoming Python version.
+
+
+Limited API Caveats
+-------------------
+
+Note that compiling with ``Py_LIMITED_API`` is *not* a complete guarantee that
+code conforms to the Limited API or the Stable ABI. ``Py_LIMITED_API`` only
+covers definitions, but an API also includes other issues, such as expected
+semantics.
+
+One issue that ``Py_LIMITED_API`` does not guard against is calling a function
+with arguments that are invalid in a lower Python version.
+For example, consider a function that starts accepting ``NULL`` for an
+argument. In Python 3.9, ``NULL`` now selects a default behavior, but in
+Python 3.8, the argument will be used directly, causing a ``NULL`` dereference
+and crash. A similar argument works for fields of structs.
+
+Another issue is that some struct fields are currently not hidden when
+``Py_LIMITED_API`` is defined, even though they're part of the Limited API.
+
+For these reasons, we recommend testing an extension with *all* minor Python
+versions it supports, and preferably to build with the *lowest* such version.
+
+We also recommend reviewing documentation of all used API to check
+if it is explicitly part of the Limited API. Even with ``Py_LIMITED_API``
+defined, a few private declarations are exposed for technical reasons (or
+even unintentionally, as bugs).
+
+Also note that the Limited API is not necessarily stable: compiling with
+``Py_LIMITED_API`` with Python 3.8 means that the extension will
+run with Python 3.12, but it will not necessarily *compile* with Python 3.12.
+In particular, parts of the Limited API may be deprecated and removed,
+provided that the Stable ABI stays stable.
+
+
+.. _stable-abi-platform:
+
+Platform Considerations
+=======================
+
+ABI stability depends not only on Python, but also on the compiler used,
+lower-level libraries and compiler options. For the purposes of the Stable ABI,
+these details define a “platform”. They usually depend on the OS
+type and processor architecture
+
+It is the responsibility of each particular distributor of Python
+to ensure that all Python versions on a particular platform are built
+in a way that does not break the Stable ABI.
+This is the case with Windows and macOS releases from ``python.org`` and many
+third-party distributors.
+
+
+.. _stable-abi-list:
+
+Contents of Limited API
+=======================
+
+
+Currently, the Limited API includes the following items:
+
+.. limited-api-list::