PEP 489: Multi-phase extension module initialization
Known limitations of the current implementation:
- documentation changes are incomplete
- there's a reference leak I haven't tracked down yet
The leak is most visible by running:
./python -m test -R3:3 test_importlib
However, you can also see it by running:
./python -X showrefcount
Importing the array or _testmultiphase modules, and
then deleting them from both sys.modules and the local
namespace shows significant increases in the total
number of active references each cycle. By contrast,
with _testcapi (which continues to use single-phase
initialisation) the global refcounts stabilise after
a couple of cycles.
diff --git a/Doc/c-api/module.rst b/Doc/c-api/module.rst
index 985a347..df9301f 100644
--- a/Doc/c-api/module.rst
+++ b/Doc/c-api/module.rst
@@ -7,8 +7,6 @@
.. index:: object: module
-There are only a few functions special to module objects.
-
.. c:var:: PyTypeObject PyModule_Type
@@ -109,6 +107,14 @@
unencodable filenames, use :c:func:`PyModule_GetFilenameObject` instead.
+Per-interpreter module state
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Single-phase initialization creates singleton modules that can store additional
+information as part of the interpreter, allow that state to be retrieved later
+with only a reference to the module definition, rather than to the module
+itself.
+
.. c:function:: void* PyModule_GetState(PyObject *module)
Return the "state" of the module, that is, a pointer to the block of memory
@@ -146,27 +152,6 @@
Initializing C modules
^^^^^^^^^^^^^^^^^^^^^^
-These functions are usually used in the module initialization function.
-
-.. c:function:: PyObject* PyModule_Create(PyModuleDef *module)
-
- Create a new module object, given the definition in *module*. This behaves
- like :c:func:`PyModule_Create2` with *module_api_version* set to
- :const:`PYTHON_API_VERSION`.
-
-
-.. c:function:: PyObject* PyModule_Create2(PyModuleDef *module, int module_api_version)
-
- Create a new module object, given the definition in *module*, assuming the
- API version *module_api_version*. If that version does not match the version
- of the running interpreter, a :exc:`RuntimeWarning` is emitted.
-
- .. note::
-
- Most uses of this function should be using :c:func:`PyModule_Create`
- instead; only use this if you are sure you need it.
-
-
.. c:type:: PyModuleDef
This struct holds all information that is needed to create a module object.
@@ -210,9 +195,10 @@
A pointer to a table of module-level functions, described by
:c:type:`PyMethodDef` values. Can be *NULL* if no functions are present.
- .. c:member:: inquiry m_reload
+ .. c:member:: PyModuleDef_Slot* m_slots
- Currently unused, should be *NULL*.
+ An array of slot definitions for multi-phase initialization, terminated by
+ a *NULL* entry.
.. c:member:: traverseproc m_traverse
@@ -229,6 +215,61 @@
A function to call during deallocation of the module object, or *NULL* if
not needed.
+The module initialization function may create and return the module object
+directly. This is referred to as "single-phase initialization", and uses one
+of the following two module creation functions:
+
+.. c:function:: PyObject* PyModule_Create(PyModuleDef *module)
+
+ Create a new module object, given the definition in *module*. This behaves
+ like :c:func:`PyModule_Create2` with *module_api_version* set to
+ :const:`PYTHON_API_VERSION`.
+
+
+.. c:function:: PyObject* PyModule_Create2(PyModuleDef *module, int module_api_version)
+
+ Create a new module object, given the definition in *module*, assuming the
+ API version *module_api_version*. If that version does not match the version
+ of the running interpreter, a :exc:`RuntimeWarning` is emitted.
+
+ .. note::
+
+ Most uses of this function should be using :c:func:`PyModule_Create`
+ instead; only use this if you are sure you need it.
+
+
+Alternatively, the module initialization function may instead return a
+:c:type:`PyModuleDef` instance with a non-empty ``m_slots`` array. This is
+referred to as "multi-phase initialization", and ``PyModuleDef`` instance
+should be initialized with the following function:
+
+.. c:function:: PyObject* PyModuleDef_Init(PyModuleDef *module)
+
+ Ensures a module definition is a properly initialized Python object that
+ correctly reports its type and reference count.
+
+.. XXX (ncoghlan): It's not clear if it makes sense to document PyModule_ExecDef
+ PyModule_FromDefAndSpec or PyModule_FromDefAndSpec2 here, as end user code
+ generally shouldn't be calling those.
+
+The module initialization function (if using single phase initialization) or
+a function called from a module execution slot (if using multiphase
+initialization), can use the following functions to help initialize the module
+state:
+
+.. c:function:: int PyModule_SetDocString(PyObject *module, const char *docstring)
+
+ Set the docstring for *module* to *docstring*. Return ``-1`` on error, ``0``
+ on success.
+
+.. c:function:: int PyModule_AddFunctions(PyObject *module, PyMethodDef *functions)
+
+ Add the functions from the ``NULL`` terminated *functions* array to *module*.
+ Refer to the :c:type:`PyMethodDef` documentation for details on individual
+ entries (due to the lack of a shared module namespace, module level
+ "functions" implemented in C typically receive the module as their first
+ parameter, making them similar to instance methods on Python classes).
+
.. c:function:: int PyModule_AddObject(PyObject *module, const char *name, PyObject *value)
@@ -236,7 +277,6 @@
be used from the module's initialization function. This steals a reference to
*value*. Return ``-1`` on error, ``0`` on success.
-
.. c:function:: int PyModule_AddIntConstant(PyObject *module, const char *name, long value)
Add an integer constant to *module* as *name*. This convenience function can be
diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst
index 07d8ae1..771c4c5 100644
--- a/Doc/library/importlib.rst
+++ b/Doc/library/importlib.rst
@@ -55,6 +55,12 @@
:pep:`451`
A ModuleSpec Type for the Import System
+ :pep:`488`
+ Elimination of PYO files
+
+ :pep:`489`
+ Multi-phase extension module initialization
+
:pep:`3120`
Using UTF-8 as the Default Source Encoding
@@ -756,9 +762,9 @@
Only class methods are defined by this class to alleviate the need for
instantiation.
- .. note::
- Due to limitations in the extension module C-API, for now
- BuiltinImporter does not implement :meth:`Loader.exec_module`.
+ .. versionchanged:: 3.5
+ As part of :pep:`489`, the builtin importer now implements
+ :meth:`Loader.create_module` and :meth:`Loader.exec_module`
.. class:: FrozenImporter
@@ -973,14 +979,18 @@
Path to the extension module.
- .. method:: load_module(name=None)
+ .. method:: create_module(spec)
- Loads the extension module if and only if *fullname* is the same as
- :attr:`name` or is ``None``.
+ Creates the module object from the given specification in accordance
+ with :pep:`489`.
- .. note::
- Due to limitations in the extension module C-API, for now
- ExtensionFileLoader does not implement :meth:`Loader.exec_module`.
+ .. versionadded:: 3.5
+
+ .. method:: exec_module(module)
+
+ Initializes the given module object in accordance with :pep:`489`.
+
+ .. versionadded:: 3.5
.. method:: is_package(fullname)
diff --git a/Doc/whatsnew/3.5.rst b/Doc/whatsnew/3.5.rst
index c0f9346..75bc7fb 100644
--- a/Doc/whatsnew/3.5.rst
+++ b/Doc/whatsnew/3.5.rst
@@ -93,6 +93,7 @@
(:issue:`19977`).
* :pep:`488`, the elimination of ``.pyo`` files.
+* :pep:`489`, multi-phase initialization of extension modules.
Significantly Improved Library Modules:
@@ -265,6 +266,21 @@
:pep:`488` -- Elimination of PYO files
+PEP 489: Multi-phase extension module initialization
+----------------------------------------------------
+
+:pep:`489` updates extension module initialization to take advantage of the
+two step module loading mechanism introduced by :pep:`451` in Python 3.4.
+
+This change brings the import semantics of extension modules that opt-in to
+using the new mechanism much closer to those of Python source and bytecode
+modules, including the ability to any valid identifier as a module name,
+rather than being restricted to ASCII.
+
+.. seealso::
+
+ :pep:`488` -- Multi-phase extension module initialization
+
Other Language Changes
======================
@@ -667,7 +683,7 @@
tkinter
-------
-* The :module:`tkinter._fix` module used for setting up the Tcl/Tk environment
+* The :mod:`tkinter._fix` module used for setting up the Tcl/Tk environment
on Windows has been replaced by a private function in the :module:`_tkinter`
module which makes no permanent changes to environment variables.
(Contributed by Zachary Ware in :issue:`20035`.)
@@ -1012,7 +1028,6 @@
program depends on patching the module level variable to capture the debug
output, you will need to update it to capture sys.stderr instead.
-
Changes in the C API
--------------------