bpo-42236: Enhance init and encoding documentation (GH-23109)
Enhance the documentation of the Python startup, filesystem encoding
and error handling, locale encoding. Add a new "Python UTF-8 Mode"
section.
* Add "locale encoding" and "filesystem encoding and error handler"
to the glossary
* Remove documentation from Include/cpython/initconfig.h: move it to
Doc/c-api/init_config.rst.
* Doc/c-api/init_config.rst:
* Document command line options and environment variables
* Document default values.
* Add a new "Python UTF-8 Mode" section in Doc/library/os.rst.
* Add warnings to Py_DecodeLocale() and Py_EncodeLocale() docs.
* Document how Python selects the filesystem encoding and error
handler at a single place: PyConfig.filesystem_encoding and
PyConfig.filesystem_errors.
* PyConfig: move orig_argv member at the right place.
diff --git a/Doc/c-api/exceptions.rst b/Doc/c-api/exceptions.rst
index 247b6d6..4e99a01 100644
--- a/Doc/c-api/exceptions.rst
+++ b/Doc/c-api/exceptions.rst
@@ -182,8 +182,8 @@
.. c:function:: PyObject* PyErr_SetFromErrnoWithFilename(PyObject *type, const char *filename)
Similar to :c:func:`PyErr_SetFromErrnoWithFilenameObject`, but the filename
- is given as a C string. *filename* is decoded from the filesystem encoding
- (:func:`os.fsdecode`).
+ is given as a C string. *filename* is decoded from the :term:`filesystem
+ encoding and error handler`.
.. c:function:: PyObject* PyErr_SetFromWindowsErr(int ierr)
@@ -266,7 +266,7 @@
.. c:function:: void PyErr_SyntaxLocationEx(const char *filename, int lineno, int col_offset)
Like :c:func:`PyErr_SyntaxLocationObject`, but *filename* is a byte string
- decoded from the filesystem encoding (:func:`os.fsdecode`).
+ decoded from the :term:`filesystem encoding and error handler`.
.. versionadded:: 3.2
@@ -343,7 +343,7 @@
Similar to :c:func:`PyErr_WarnExplicitObject` except that *message* and
*module* are UTF-8 encoded strings, and *filename* is decoded from the
- filesystem encoding (:func:`os.fsdecode`).
+ :term:`filesystem encoding and error handler`.
.. c:function:: int PyErr_WarnFormat(PyObject *category, Py_ssize_t stack_level, const char *format, ...)
diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst
index 7f06648..3ce6892 100644
--- a/Doc/c-api/init.rst
+++ b/Doc/c-api/init.rst
@@ -151,8 +151,9 @@
.. c:var:: int Py_LegacyWindowsFSEncodingFlag
- If the flag is non-zero, use the ``mbcs`` encoding instead of the UTF-8
- encoding for the filesystem encoding.
+ If the flag is non-zero, use the ``mbcs`` encoding with ``replace`` error
+ handler, instead of the UTF-8 encoding with ``surrogatepass`` error handler,
+ for the :term:`filesystem encoding and error handler`.
Set to ``1`` if the :envvar:`PYTHONLEGACYWINDOWSFSENCODING` environment
variable is set to a non-empty string.
diff --git a/Doc/c-api/init_config.rst b/Doc/c-api/init_config.rst
index 92a6c3a..dad1f90 100644
--- a/Doc/c-api/init_config.rst
+++ b/Doc/c-api/init_config.rst
@@ -189,11 +189,7 @@
.. c:type:: PyPreConfig
- Structure used to preinitialize Python:
-
- * Set the Python memory allocator
- * Configure the LC_CTYPE locale
- * Set the UTF-8 mode
+ Structure used to preinitialize Python.
Function to initialize a preconfiguration:
@@ -211,7 +207,7 @@
.. c:member:: int allocator
- Name of the memory allocator:
+ Name of the Python memory allocators:
* ``PYMEM_ALLOCATOR_NOT_SET`` (``0``): don't change memory allocators
(use defaults)
@@ -231,27 +227,48 @@
See :ref:`Memory Management <memory>`.
+ Default: ``PYMEM_ALLOCATOR_NOT_SET``.
+
.. c:member:: int configure_locale
- Set the LC_CTYPE locale to the user preferred locale? If equals to 0, set
- :c:member:`coerce_c_locale` and :c:member:`coerce_c_locale_warn` to 0.
+ Set the LC_CTYPE locale to the user preferred locale?
+
+ If equals to 0, set :c:member:`~PyPreConfig.coerce_c_locale` and
+ :c:member:`~PyPreConfig.coerce_c_locale_warn` members to 0.
+
+ See the :term:`locale encoding`.
+
+ Default: ``1`` in Python config, ``0`` in isolated config.
.. c:member:: int coerce_c_locale
- If equals to 2, coerce the C locale; if equals to 1, read the LC_CTYPE
- locale to decide if it should be coerced.
+ If equals to 2, coerce the C locale.
+
+ If equals to 1, read the LC_CTYPE locale to decide if it should be
+ coerced.
+
+ See the :term:`locale encoding`.
+
+ Default: ``-1`` in Python config, ``0`` in isolated config.
.. c:member:: int coerce_c_locale_warn
If non-zero, emit a warning if the C locale is coerced.
+ Default: ``-1`` in Python config, ``0`` in isolated config.
+
.. c:member:: int dev_mode
- See :c:member:`PyConfig.dev_mode`.
+ If non-zero, enables the :ref:`Python Development Mode <devmode>`:
+ see :c:member:`PyConfig.dev_mode`.
+
+ Default: ``-1`` in Python mode, ``0`` in isolated mode.
.. c:member:: int isolated
- See :c:member:`PyConfig.isolated`.
+ Isolated mode: see :c:member:`PyConfig.isolated`.
+
+ Default: ``0`` in Python mode, ``1`` in isolated mode.
.. c:member:: int legacy_windows_fs_encoding
@@ -267,6 +284,8 @@
Only available on Windows. ``#ifdef MS_WINDOWS`` macro can be used for
Windows specific code.
+ Default: ``0``.
+
.. c:member:: int parse_argv
If non-zero, :c:func:`Py_PreInitializeFromArgs` and
@@ -274,16 +293,36 @@
same way the regular Python parses command line arguments: see
:ref:`Command Line Arguments <using-on-cmdline>`.
+ Default: ``1`` in Python config, ``0`` in isolated config.
+
.. c:member:: int use_environment
- See :c:member:`PyConfig.use_environment`.
+ Use :ref:`environment variables <using-on-envvars>`? See
+ :c:member:`PyConfig.use_environment`.
+
+ Default: ``1`` in Python config and ``0`` in isolated config.
.. c:member:: int utf8_mode
- If non-zero, enable the UTF-8 mode.
+ If non-zero, enable the :ref:`Python UTF-8 Mode <utf8-mode>`.
-Preinitialization with PyPreConfig
-----------------------------------
+ Set by the :option:`-X utf8 <-X>` command line option and the
+ :envvar:`PYTHONUTF8` environment variable.
+
+ Default: ``-1`` in Python config and ``0`` in isolated config.
+
+
+.. _c-preinit:
+
+Preinitialize Python with PyPreConfig
+-------------------------------------
+
+The preinitialization of Python:
+
+* Set the Python memory allocators (:c:member:`PyPreConfig.allocator`)
+* Configure the LC_CTYPE locale (:term:`locale encoding`)
+* Set the :ref:`Python UTF-8 Mode <utf8-mode>`
+ (:c:member:`PyPreConfig.utf8_mode`)
Functions to preinitialize Python:
@@ -293,13 +332,17 @@
.. c:function:: PyStatus Py_PreInitializeFromBytesArgs(const PyPreConfig *preconfig, int argc, char * const *argv)
- Preinitialize Python from *preconfig* preconfiguration and command line
- arguments (bytes strings).
+ Preinitialize Python from *preconfig* preconfiguration.
+
+ Parse *argv* command line arguments (bytes strings) if
+ :c:member:`~PyPreConfig.parse_argv` of *preconfig* is non-zero.
.. c:function:: PyStatus Py_PreInitializeFromArgs(const PyPreConfig *preconfig, int argc, wchar_t * const * argv)
- Preinitialize Python from *preconfig* preconfiguration and command line
- arguments (wide strings).
+ Preinitialize Python from *preconfig* preconfiguration.
+
+ Parse *argv* command line arguments (wide strings) if
+ :c:member:`~PyPreConfig.parse_argv` of *preconfig* is non-zero.
The caller is responsible to handle exceptions (error or exit) using
:c:func:`PyStatus_Exception` and :c:func:`Py_ExitStatusException`.
@@ -309,7 +352,7 @@
command line arguments, the command line arguments must also be passed to
preinitialize Python, since they have an effect on the pre-configuration
like encodings. For example, the :option:`-X utf8 <-X>` command line option
-enables the UTF-8 Mode.
+enables the :ref:`Python UTF-8 Mode <utf8-mode>`.
``PyMem_SetAllocator()`` can be called after :c:func:`Py_PreInitialize` and
before :c:func:`Py_InitializeFromConfig` to install a custom memory allocator.
@@ -317,11 +360,12 @@
:c:member:`PyPreConfig.allocator` is set to ``PYMEM_ALLOCATOR_NOT_SET``.
Python memory allocation functions like :c:func:`PyMem_RawMalloc` must not be
-used before Python preinitialization, whereas calling directly ``malloc()`` and
-``free()`` is always safe. :c:func:`Py_DecodeLocale` must not be called before
-the preinitialization.
+used before the Python preinitialization, whereas calling directly ``malloc()``
+and ``free()`` is always safe. :c:func:`Py_DecodeLocale` must not be called
+before the Python preinitialization.
-Example using the preinitialization to enable the UTF-8 Mode::
+Example using the preinitialization to enable
+the :ref:`Python UTF-8 Mode <utf8-mode>`::
PyStatus status;
PyPreConfig preconfig;
@@ -334,7 +378,7 @@
Py_ExitStatusException(status);
}
- /* at this point, Python will speak UTF-8 */
+ /* at this point, Python speaks UTF-8 */
Py_Initialize();
/* ... use Python API here ... */
@@ -348,47 +392,54 @@
Structure containing most parameters to configure Python.
+ When done, the :c:func:`PyConfig_Clear` function must be used to release the
+ configuration memory.
+
Structure methods:
.. c:function:: void PyConfig_InitPythonConfig(PyConfig *config)
- Initialize configuration with :ref:`Python Configuration
+ Initialize configuration with the :ref:`Python Configuration
<init-python-config>`.
.. c:function:: void PyConfig_InitIsolatedConfig(PyConfig *config)
- Initialize configuration with :ref:`Isolated Configuration
+ Initialize configuration with the :ref:`Isolated Configuration
<init-isolated-conf>`.
.. c:function:: PyStatus PyConfig_SetString(PyConfig *config, wchar_t * const *config_str, const wchar_t *str)
Copy the wide character string *str* into ``*config_str``.
- Preinitialize Python if needed.
+ :ref:`Preinitialize Python <c-preinit>` if needed.
.. c:function:: PyStatus PyConfig_SetBytesString(PyConfig *config, wchar_t * const *config_str, const char *str)
- Decode *str* using ``Py_DecodeLocale()`` and set the result into ``*config_str``.
+ Decode *str* using :c:func:`Py_DecodeLocale` and set the result into
+ ``*config_str``.
- Preinitialize Python if needed.
+ :ref:`Preinitialize Python <c-preinit>` if needed.
.. c:function:: PyStatus PyConfig_SetArgv(PyConfig *config, int argc, wchar_t * const *argv)
- Set command line arguments from wide character strings.
+ Set command line arguments (:c:member:`~PyConfig.argv` member of
+ *config*) from the *argv* list of wide character strings.
- Preinitialize Python if needed.
+ :ref:`Preinitialize Python <c-preinit>` if needed.
.. c:function:: PyStatus PyConfig_SetBytesArgv(PyConfig *config, int argc, char * const *argv)
- Set command line arguments: decode bytes using :c:func:`Py_DecodeLocale`.
+ Set command line arguments (:c:member:`~PyConfig.argv` member of
+ *config*) from the *argv* list of bytes strings. Decode bytes using
+ :c:func:`Py_DecodeLocale`.
- Preinitialize Python if needed.
+ :ref:`Preinitialize Python <c-preinit>` if needed.
.. c:function:: PyStatus PyConfig_SetWideStringList(PyConfig *config, PyWideStringList *list, Py_ssize_t length, wchar_t **items)
Set the list of wide strings *list* to *length* and *items*.
- Preinitialize Python if needed.
+ :ref:`Preinitialize Python <c-preinit>` if needed.
.. c:function:: PyStatus PyConfig_Read(PyConfig *config)
@@ -396,24 +447,25 @@
Fields which are already initialized are left unchanged.
- Preinitialize Python if needed.
+ :ref:`Preinitialize Python <c-preinit>` if needed.
.. c:function:: void PyConfig_Clear(PyConfig *config)
Release configuration memory.
- Most ``PyConfig`` methods preinitialize Python if needed. In that case, the
- Python preinitialization configuration in based on the :c:type:`PyConfig`.
- If configuration fields which are in common with :c:type:`PyPreConfig` are
- tuned, they must be set before calling a :c:type:`PyConfig` method:
+ Most ``PyConfig`` methods :ref:`preinitialize Python <c-preinit>` if needed.
+ In that case, the Python preinitialization configuration
+ (:c:type:`PyPreConfig`) in based on the :c:type:`PyConfig`. If configuration
+ fields which are in common with :c:type:`PyPreConfig` are tuned, they must
+ be set before calling a :c:type:`PyConfig` method:
- * :c:member:`~PyConfig.dev_mode`
- * :c:member:`~PyConfig.isolated`
- * :c:member:`~PyConfig.parse_argv`
- * :c:member:`~PyConfig.use_environment`
+ * :c:member:`PyConfig.dev_mode`
+ * :c:member:`PyConfig.isolated`
+ * :c:member:`PyConfig.parse_argv`
+ * :c:member:`PyConfig.use_environment`
Moreover, if :c:func:`PyConfig_SetArgv` or :c:func:`PyConfig_SetBytesArgv`
- is used, this method must be called first, before other methods, since the
+ is used, this method must be called before other methods, since the
preinitialization configuration depends on command line arguments (if
:c:member:`parse_argv` is non-zero).
@@ -424,11 +476,17 @@
.. c:member:: PyWideStringList argv
- Command line arguments, :data:`sys.argv`. See
- :c:member:`~PyConfig.parse_argv` to parse :c:member:`~PyConfig.argv` the
- same way the regular Python parses Python command line arguments. If
- :c:member:`~PyConfig.argv` is empty, an empty string is added to ensure
- that :data:`sys.argv` always exists and is never empty.
+ Command line arguments: :data:`sys.argv`.
+
+ Set :c:member:`~PyConfig.parse_argv` to ``1`` to parse
+ :c:member:`~PyConfig.argv` the same way the regular Python parses Python
+ command line arguments and then to strip Python arguments from
+ :c:member:`~PyConfig.argv`.
+
+ If :c:member:`~PyConfig.argv` is empty, an empty string is added to
+ ensure that :data:`sys.argv` always exists and is never empty.
+
+ Default: ``NULL``.
See also the :c:member:`~PyConfig.orig_argv` member.
@@ -436,76 +494,136 @@
:data:`sys.base_exec_prefix`.
+ Default: ``NULL``.
+
+ Part of the :ref:`Path Configuration <init-path-config>` output.
+
.. c:member:: wchar_t* base_executable
- :data:`sys._base_executable`: ``__PYVENV_LAUNCHER__`` environment
- variable value, or copy of :c:member:`PyConfig.executable`.
+ Python base executable: :data:`sys._base_executable`.
+
+ Set by the :envvar:`__PYVENV_LAUNCHER__` environment variable.
+
+ Set from :c:member:`PyConfig.executable` if ``NULL``.
+
+ Default: ``NULL``.
+
+ Part of the :ref:`Path Configuration <init-path-config>` output.
.. c:member:: wchar_t* base_prefix
:data:`sys.base_prefix`.
- .. c:member:: wchar_t* platlibdir
+ Default: ``NULL``.
- :data:`sys.platlibdir`: platform library directory name, set at configure time
- by ``--with-platlibdir``, overrideable by the ``PYTHONPLATLIBDIR``
- environment variable.
-
- .. versionadded:: 3.9
+ Part of the :ref:`Path Configuration <init-path-config>` output.
.. c:member:: int buffered_stdio
- If equals to 0, enable unbuffered mode, making the stdout and stderr
- streams unbuffered.
+ If equals to 0 and :c:member:`~PyConfig.configure_c_stdio` is non-zero,
+ disable buffering on the C streams stdout and stderr.
+
+ Set to 0 by the :option:`-u` command line option and the
+ :envvar:`PYTHONUNBUFFERED` environment variable.
stdin is always opened in buffered mode.
+ Default: ``1``.
+
.. c:member:: int bytes_warning
If equals to 1, issue a warning when comparing :class:`bytes` or
:class:`bytearray` with :class:`str`, or comparing :class:`bytes` with
- :class:`int`. If equal or greater to 2, raise a :exc:`BytesWarning`
- exception.
+ :class:`int`.
+
+ If equal or greater to 2, raise a :exc:`BytesWarning` exception in these
+ cases.
+
+ Incremented by the :option:`-b` command line option.
+
+ Default: ``0``.
.. c:member:: wchar_t* check_hash_pycs_mode
- Control the validation behavior of hash-based ``.pyc`` files (see
- :pep:`552`): :option:`--check-hash-based-pycs` command line option value.
+ Control the validation behavior of hash-based ``.pyc`` files:
+ value of the :option:`--check-hash-based-pycs` command line option.
- Valid values: ``always``, ``never`` and ``default``.
+ Valid values:
- The default value is: ``default``.
+ - ``L"always"``: Hash the source file for invalidation regardless of
+ value of the 'check_source' flag.
+ - ``L"never"``: Assume that hash-based pycs always are valid.
+ - ``L"default"``: The 'check_source' flag in hash-based pycs
+ determines invalidation.
+
+ Default: ``L"default"``.
+
+ See also :pep:`552` "Deterministic pycs".
.. c:member:: int configure_c_stdio
- If non-zero, configure C standard streams (``stdio``, ``stdout``,
- ``stdout``). For example, set their mode to ``O_BINARY`` on Windows.
+ If non-zero, configure C standard streams:
+
+ * On Windows, set the binary mode (``O_BINARY``) on stdin, stdout and
+ stderr.
+ * If :c:member:`~PyConfig.buffered_stdio` equals zero, disable buffering
+ of stdin, stdout and stderr streams.
+ * If :c:member:`~PyConfig.interactive` is non-zero, enable stream
+ buffering on stdin and stdout (only stdout on Windows).
+
+ Default: ``1`` in Python config, ``0`` in isolated config.
.. c:member:: int dev_mode
If non-zero, enable the :ref:`Python Development Mode <devmode>`.
+ Default: ``-1`` in Python mode, ``0`` in isolated mode.
+
.. c:member:: int dump_refs
+ Dump Python refererences?
+
If non-zero, dump all objects which are still alive at exit.
- ``Py_TRACE_REFS`` macro must be defined in build.
+ Set to ``1`` by the :envvar:`PYTHONDUMPREFS` environment variable.
+
+ Need a special build of Python with the ``Py_TRACE_REFS`` macro defined.
+
+ Default: ``0``.
.. c:member:: wchar_t* exec_prefix
- :data:`sys.exec_prefix`.
+ The site-specific directory prefix where the platform-dependent Python
+ files are installed: :data:`sys.exec_prefix`.
+
+ Default: ``NULL``.
+
+ Part of the :ref:`Path Configuration <init-path-config>` output.
.. c:member:: wchar_t* executable
+ The absolute path of the executable binary for the Python interpreter:
:data:`sys.executable`.
+ Default: ``NULL``.
+
+ Part of the :ref:`Path Configuration <init-path-config>` output.
+
.. c:member:: int faulthandler
+ Enable faulthandler?
+
If non-zero, call :func:`faulthandler.enable` at startup.
+ Set to ``1`` by :option:`-X faulthandler <-X>` and the
+ :envvar:`PYTHONFAULTHANDLER` environment variable.
+
+ Default: ``-1`` in Python mode, ``0`` in isolated mode.
+
.. c:member:: wchar_t* filesystem_encoding
- Filesystem encoding: :func:`sys.getfilesystemencoding`.
+ :term:`Filesystem encoding <filesystem encoding and error handler>`:
+ :func:`sys.getfilesystemencoding`.
On macOS, Android and VxWorks: use ``"utf-8"`` by default.
@@ -521,7 +639,7 @@
``mbstowcs()`` function decodes from a different encoding (usually
Latin1).
* ``"utf-8"`` if ``nl_langinfo(CODESET)`` returns an empty string.
- * Otherwise, use the LC_CTYPE locale encoding:
+ * Otherwise, use the :term:`locale encoding`:
``nl_langinfo(CODESET)`` result.
At Python statup, the encoding name is normalized to the Python codec
@@ -531,7 +649,8 @@
.. c:member:: wchar_t* filesystem_errors
- Filesystem error handler: :func:`sys.getfilesystemencodeerrors`.
+ :term:`Filesystem error handler <filesystem encoding and error handler>`:
+ :func:`sys.getfilesystemencodeerrors`.
On Windows: use ``"surrogatepass"`` by default, or ``"replace"`` if
:c:member:`~PyPreConfig.legacy_windows_fs_encoding` of
@@ -553,30 +672,62 @@
Randomized hash function seed.
If :c:member:`~PyConfig.use_hash_seed` is zero, a seed is chosen randomly
- at Pythonstartup, and :c:member:`~PyConfig.hash_seed` is ignored.
+ at Python startup, and :c:member:`~PyConfig.hash_seed` is ignored.
+
+ Set by the :envvar:`PYTHONHASHSEED` environment variable.
+
+ Default *use_hash_seed* value: ``-1`` in Python mode, ``0`` in isolated
+ mode.
.. c:member:: wchar_t* home
Python home directory.
- Initialized from :envvar:`PYTHONHOME` environment variable value by
- default.
+ If :c:func:`Py_SetPythonHome` has been called, use its argument if it is
+ not ``NULL``.
+
+ Set by the :envvar:`PYTHONHOME` environment variable.
+
+ Default: ``NULL``.
+
+ Part of the :ref:`Path Configuration <init-path-config>` input.
.. c:member:: int import_time
If non-zero, profile import time.
+ Set the ``1`` by the :option:`-X importtime <-X>` option and the
+ :envvar:`PYTHONPROFILEIMPORTTIME` environment variable.
+
+ Default: ``0``.
+
.. c:member:: int inspect
Enter interactive mode after executing a script or a command.
+ If greater than 0, enable inspect: when a script is passed as first
+ argument or the -c option is used, enter interactive mode after executing
+ the script or the command, even when :data:`sys.stdin` does not appear to
+ be a terminal.
+
+ Incremented by the :option:`-i` command line option. Set to ``1`` if the
+ :envvar:`PYTHONINSPECT` environment variable is non-empty.
+
+ Default: ``0``.
+
.. c:member:: int install_signal_handlers
- Install signal handlers?
+ Install Python signal handlers?
+
+ Default: ``1`` in Python mode, ``0`` in isolated mode.
.. c:member:: int interactive
- Interactive mode.
+ If greater than 0, enable the interactive mode (REPL).
+
+ Incremented by the :option:`-i` command line option.
+
+ Default: ``0``.
.. c:member:: int isolated
@@ -590,50 +741,93 @@
* Set :c:member:`~PyConfig.use_environment` and
:c:member:`~PyConfig.user_site_directory` to 0.
+ Default: ``0`` in Python mode, ``1`` in isolated mode.
+
+ See also :c:member:`PyPreConfig.isolated`.
+
.. c:member:: int legacy_windows_stdio
If non-zero, use :class:`io.FileIO` instead of
:class:`io.WindowsConsoleIO` for :data:`sys.stdin`, :data:`sys.stdout`
and :data:`sys.stderr`.
+ Set to ``1`` if the :envvar:`PYTHONLEGACYWINDOWSSTDIO` environment
+ variable is set to a non-empty string.
+
Only available on Windows. ``#ifdef MS_WINDOWS`` macro can be used for
Windows specific code.
+ Default: ``0``.
+
+ See also the :pep:`528` (Change Windows console encoding to UTF-8).
+
.. c:member:: int malloc_stats
If non-zero, dump statistics on :ref:`Python pymalloc memory allocator
<pymalloc>` at exit.
+ Set to ``1`` by the :envvar:`PYTHONMALLOCSTATS` environment variable.
+
The option is ignored if Python is built using ``--without-pymalloc``.
+ Default: ``0``.
+
+ .. c:member:: wchar_t* platlibdir
+
+ Platform library directory name: :data:`sys.platlibdir`.
+
+ Set by the :envvar:`PYTHONPLATLIBDIR` environment variable.
+
+ Default: value of the ``PLATLIBDIR`` macro which is set at configure time
+ by ``--with-platlibdir`` (default: ``"lib"``).
+
+ Part of the :ref:`Path Configuration <init-path-config>` input.
+
+ .. versionadded:: 3.9
+
.. c:member:: wchar_t* pythonpath_env
- Module search paths as a string separated by ``DELIM``
+ Module search paths (:data:`sys.path`) as a string separated by ``DELIM``
(:data:`os.path.pathsep`).
- Initialized from :envvar:`PYTHONPATH` environment variable value by
- default.
+ Set by the :envvar:`PYTHONPATH` environment variable.
+
+ Default: ``NULL``.
+
+ Part of the :ref:`Path Configuration <init-path-config>` input.
.. c:member:: PyWideStringList module_search_paths
.. c:member:: int module_search_paths_set
- :data:`sys.path`. If :c:member:`~PyConfig.module_search_paths_set` is
- equal to 0, the :c:member:`~PyConfig.module_search_paths` is overridden
- by the function calculating the :ref:`Path Configuration
- <init-path-config>`.
+ Module search paths: :data:`sys.path`.
+
+ If :c:member:`~PyConfig.module_search_paths_set` is equal to 0, the
+ function calculating the :ref:`Path Configuration <init-path-config>`
+ overrides the :c:member:`~PyConfig.module_search_paths` and sets
+ :c:member:`~PyConfig.module_search_paths_set` to ``1``.
+
+ Default: empty list (``module_search_paths``) and ``0``
+ (``module_search_paths_set``).
+
+ Part of the :ref:`Path Configuration <init-path-config>` output.
.. c:member:: int optimization_level
Compilation optimization level:
- * 0: Peephole optimizer (and ``__debug__`` is set to ``True``)
- * 1: Remove assertions, set ``__debug__`` to ``False``
- * 2: Strip docstrings
+ * ``0``: Peephole optimizer, set ``__debug__`` to ``True``.
+ * ``1``: Level 0, remove assertions, set ``__debug__`` to ``False``.
+ * ``2``: Level 1, strip docstrings.
+
+ Incremented by the :option:`-O` command line option. Set to the
+ :envvar:`PYTHONOPTIMIZE` environment variable value.
+
+ Default: ``0``.
.. c:member:: PyWideStringList orig_argv
The list of the original command line arguments passed to the Python
- executable.
+ executable: :data:`sys.orig_argv`.
If :c:member:`~PyConfig.orig_argv` list is empty and
:c:member:`~PyConfig.argv` is not a list only containing an empty
@@ -645,57 +839,117 @@
See also the :c:member:`~PyConfig.argv` member and the
:c:func:`Py_GetArgcArgv` function.
+ Default: empty list.
+
.. versionadded:: 3.10
.. c:member:: int parse_argv
+ Parse command line arguments?
+
If non-zero, parse :c:member:`~PyConfig.argv` the same way the regular
- Python command line arguments, and strip Python arguments from
- :c:member:`~PyConfig.argv`: see :ref:`Command Line Arguments
- <using-on-cmdline>`.
+ Python parses :ref:`command line arguments <using-on-cmdline>`, and strip
+ Python arguments from :c:member:`~PyConfig.argv`.
+
+ Default: ``1`` in Python mode, ``0`` in isolated mode.
.. c:member:: int parser_debug
- If non-zero, turn on parser debugging output (for expert only, depending
+ Parser debug mode. If greater than 0, turn on parser debugging output (for expert only, depending
on compilation options).
+ Incremented by the :option:`-d` command line option. Set to the
+ :envvar:`PYTHONDEBUG` environment variable value.
+
+ Default: ``0``.
+
.. c:member:: int pathconfig_warnings
- If equal to 0, suppress warnings when calculating the :ref:`Path
- Configuration <init-path-config>` (Unix only, Windows does not log any
- warning). Otherwise, warnings are written into ``stderr``.
+ On Unix, if non-zero, calculating the :ref:`Path Configuration
+ <init-path-config>` can log warnings into ``stderr``. If equals to 0,
+ suppress these warnings.
+
+ It has no effect on Windows.
+
+ Default: ``1`` in Python mode, ``0`` in isolated mode.
+
+ Part of the :ref:`Path Configuration <init-path-config>` input.
.. c:member:: wchar_t* prefix
- :data:`sys.prefix`.
+ The site-specific directory prefix where the platform independent Python
+ files are installed: :data:`sys.prefix`.
+
+ Default: ``NULL``.
+
+ Part of the :ref:`Path Configuration <init-path-config>` output.
.. c:member:: wchar_t* program_name
- Program name. Used to initialize :c:member:`~PyConfig.executable`, and in
- early error messages.
+ Program name used to initialize :c:member:`~PyConfig.executable` and in
+ early error messages during Python initialization.
+
+ * If :func:`Py_SetProgramName` has been called, use its argument.
+ * On macOS, use :envvar:`PYTHONEXECUTABLE` environment variable if set.
+ * If the ``WITH_NEXT_FRAMEWORK`` macro is defined, use
+ :envvar:`__PYVENV_LAUNCHER__` environment variable if set.
+ * Use ``argv[0]`` of :c:member:`~PyConfig.argv` if available and
+ non-empty.
+ * Otherwise, use ``L"python"`` on Windows, or ``L"python3"`` on other
+ platforms.
+
+ Default: ``NULL``.
+
+ Part of the :ref:`Path Configuration <init-path-config>` input.
.. c:member:: wchar_t* pycache_prefix
- :data:`sys.pycache_prefix`: ``.pyc`` cache prefix.
+ Directory where cached ``.pyc`` files are written:
+ :data:`sys.pycache_prefix`.
+
+ Set by the :option:`-X pycache_prefix=PATH <-X>` command line option and
+ the :envvar:`PYTHONPYCACHEPREFIX` environment variable.
If ``NULL``, :data:`sys.pycache_prefix` is set to ``None``.
+ Default: ``NULL``.
+
.. c:member:: int quiet
- Quiet mode. For example, don't display the copyright and version messages
- in interactive mode.
+ Quiet mode. If greater than 0, don't display the copyright and version at
+ Python startup in interactive mode.
+
+ Incremented by the :option:`-q` command line option.
+
+ Default: ``0``.
.. c:member:: wchar_t* run_command
- ``python3 -c COMMAND`` argument. Used by :c:func:`Py_RunMain`.
+ Value of the :option:`-c` command line option.
+
+ Used by :c:func:`Py_RunMain`.
+
+ Default: ``NULL``.
.. c:member:: wchar_t* run_filename
- ``python3 FILENAME`` argument. Used by :c:func:`Py_RunMain`.
+ Filename passed on the command line: trailing command line argument
+ without :option:`-c` or :option:`-m`.
+
+ For example, it is set to ``script.py`` by the ``python3 script.py arg``
+ command.
+
+ Used by :c:func:`Py_RunMain`.
+
+ Default: ``NULL``.
.. c:member:: wchar_t* run_module
- ``python3 -m MODULE`` argument. Used by :c:func:`Py_RunMain`.
+ Value of the :option:`-m` command line option.
+
+ Used by :c:func:`Py_RunMain`.
+
+ Default: ``NULL``.
.. c:member:: int show_ref_count
@@ -705,64 +959,146 @@
Need a debug build of Python (``Py_REF_DEBUG`` macro must be defined).
+ Default: ``0``.
+
.. c:member:: int site_import
Import the :mod:`site` module at startup?
+ If equal to zero, disable the import of the module site and the
+ site-dependent manipulations of :data:`sys.path` that it entails.
+
+ Also disable these manipulations if the :mod:`site` module is explicitly
+ imported later (call :func:`site.main` if you want them to be triggered).
+
+ Set to ``0`` by the :option:`-S` command line option.
+
+ :data:`sys.flags.no_site` is set to the inverted value of
+ :c:member:`~PyConfig.site_import`.
+
+ Default: ``1``.
+
.. c:member:: int skip_source_first_line
- Skip the first line of the source?
+ If non-zero, skip the first line of the :c:member:`PyConfig.run_filename`
+ source.
+
+ It allows the usage of non-Unix forms of ``#!cmd``. This is intended for
+ a DOS specific hack only.
+
+ Set to ``1`` by the :option:`-x` command line option.
+
+ Default: ``0``.
.. c:member:: wchar_t* stdio_encoding
.. c:member:: wchar_t* stdio_errors
Encoding and encoding errors of :data:`sys.stdin`, :data:`sys.stdout` and
- :data:`sys.stderr`.
+ :data:`sys.stderr` (but :data:`sys.stderr` always uses
+ ``"backslashreplace"`` error handler).
+
+ If :c:func:`Py_SetStandardStreamEncoding` has been called, use its
+ *error* and *errors* arguments if they are not ``NULL``.
+
+ Use the :envvar:`PYTHONIOENCODING` environment variable if it is
+ non-empty.
+
+ Default encoding:
+
+ * ``"UTF-8"`` if :c:member:`PyPreConfig.utf8_mode` is non-zero.
+ * Otherwise, use the :term:`locale encoding`.
+
+ Default error handler:
+
+ * On Windows: use ``"surrogateescape"``.
+ * ``"surrogateescape"`` if :c:member:`PyPreConfig.utf8_mode` is non-zero,
+ or if the LC_CTYPE locale is "C" or "POSIX".
+ * ``"strict"`` otherwise.
.. c:member:: int tracemalloc
+ Enable tracemalloc?
+
If non-zero, call :func:`tracemalloc.start` at startup.
+ Set by :option:`-X tracemalloc=N <-X>` command line option and by the
+ :envvar:`PYTHONTRACEMALLOC` environment variable.
+
+ Default: ``-1`` in Python mode, ``0`` in isolated mode.
+
.. c:member:: int use_environment
- If greater than 0, use :ref:`environment variables <using-on-envvars>`.
+ Use :ref:`environment variables <using-on-envvars>`?
+
+ If equals to zero, ignore the :ref:`environment variables
+ <using-on-envvars>`.
+
+ Default: ``1`` in Python config and ``0`` in isolated config.
.. c:member:: int user_site_directory
- If non-zero, add user site directory to :data:`sys.path`.
+ If non-zero, add the user site directory to :data:`sys.path`.
+
+ Set to ``0`` by the :option:`-s` and :option:`-I` command line options.
+
+ Set to ``0`` by the :envvar:`PYTHONNOUSERSITE` environment variable.
+
+ Default: ``1`` in Python mode, ``0`` in isolated mode.
.. c:member:: int verbose
- If non-zero, enable verbose mode.
+ Verbose mode. If greater than 0, print a message each time a module is
+ imported, showing the place (filename or built-in module) from which
+ it is loaded.
+
+ If greater or equal to 2, print a message for each file that is checked
+ for when searching for a module. Also provides information on module
+ cleanup at exit.
+
+ Incremented by the :option:`-v` command line option.
+
+ Set to the :envvar:`PYTHONVERBOSE` environment variable value.
+
+ Default: ``0``.
.. c:member:: PyWideStringList warnoptions
- :data:`sys.warnoptions`: options of the :mod:`warnings` module to build
- warnings filters: lowest to highest priority.
+ Options of the :mod:`warnings` module to build warnings filters, lowest
+ to highest priority: :data:`sys.warnoptions`.
The :mod:`warnings` module adds :data:`sys.warnoptions` in the reverse
order: the last :c:member:`PyConfig.warnoptions` item becomes the first
item of :data:`warnings.filters` which is checked first (highest
priority).
+ Default: empty list.
+
.. c:member:: int write_bytecode
- If non-zero, write ``.pyc`` files.
+ If equal to 0, Python won't try to write ``.pyc`` files on the import of
+ source modules.
+
+ Set to ``0`` by the :option:`-B` command line option and the
+ :envvar:`PYTHONDONTWRITEBYTECODE` environment variable.
:data:`sys.dont_write_bytecode` is initialized to the inverted value of
:c:member:`~PyConfig.write_bytecode`.
+ Default: ``1``.
+
.. c:member:: PyWideStringList xoptions
- :data:`sys._xoptions`.
+ Values of the :option:`-X` command line options: :data:`sys._xoptions`.
-If ``parse_argv`` is non-zero, ``argv`` arguments are parsed the same
-way the regular Python parses command line arguments, and Python
-arguments are stripped from ``argv``: see :ref:`Command Line Arguments
-<using-on-cmdline>`.
+ Default: empty list.
-The ``xoptions`` options are parsed to set other options: see :option:`-X`
-option.
+If :c:member:`~PyConfig.parse_argv` is non-zero, :c:member:`~PyConfig.argv`
+arguments are parsed the same way the regular Python parses :ref:`command line
+arguments <using-on-cmdline>`, and Python arguments are stripped from
+:c:member:`~PyConfig.argv`.
+
+The :c:member:`~PyConfig.xoptions` options are parsed to set other options: see
+the :option:`-X` command line option.
.. versionchanged:: 3.9
@@ -781,9 +1117,9 @@
The caller is responsible to handle exceptions (error or exit) using
:c:func:`PyStatus_Exception` and :c:func:`Py_ExitStatusException`.
-If ``PyImport_FrozenModules``, ``PyImport_AppendInittab()`` or
-``PyImport_ExtendInittab()`` are used, they must be set or called after Python
-preinitialization and before the Python initialization.
+If :c:func:`PyImport_FrozenModules`, :c:func:`PyImport_AppendInittab` or
+:c:func:`PyImport_ExtendInittab` are used, they must be set or called after
+Python preinitialization and before the Python initialization.
Example setting the program name::
@@ -828,7 +1164,7 @@
Implicitly preinitialize Python. */
status = PyConfig_SetBytesString(&config, &config.program_name,
- program_name);
+ program_name);
if (PyStatus_Exception(status)) {
goto done;
}
@@ -894,7 +1230,8 @@
Environments variables and command line arguments are used to configure
Python, whereas global configuration variables are ignored.
-This function enables C locale coercion (:pep:`538`) and UTF-8 Mode
+This function enables C locale coercion (:pep:`538`)
+and :ref:`Python UTF-8 Mode <utf8-mode>`
(:pep:`540`) depending on the LC_CTYPE locale, :envvar:`PYTHONUTF8` and
:envvar:`PYTHONCOERCECLOCALE` environment variables.
diff --git a/Doc/c-api/sys.rst b/Doc/c-api/sys.rst
index 9ac9179..97717f5 100644
--- a/Doc/c-api/sys.rst
+++ b/Doc/c-api/sys.rst
@@ -118,22 +118,21 @@
.. c:function:: wchar_t* Py_DecodeLocale(const char* arg, size_t *size)
- Decode a byte string from the locale encoding with the :ref:`surrogateescape
- error handler <surrogateescape>`: undecodable bytes are decoded as
- characters in range U+DC80..U+DCFF. If a byte sequence can be decoded as a
- surrogate character, escape the bytes using the surrogateescape error
- handler instead of decoding them.
+ .. warning::
+ This function should not be called directly: use the :c:type:`PyConfig`
+ API with the :c:func:`PyConfig_SetBytesString` function which ensures
+ that :ref:`Python is preinitialized <c-preinit>`.
- Encoding, highest priority to lowest priority:
+ This function must not be called before :ref:`Python is preinitialized
+ <c-preinit>` and so that the LC_CTYPE locale is properly configured: see
+ the :c:func:`Py_PreInitialize` function.
- * ``UTF-8`` on macOS, Android, and VxWorks;
- * ``UTF-8`` on Windows if :c:data:`Py_LegacyWindowsFSEncodingFlag` is zero;
- * ``UTF-8`` if the Python UTF-8 mode is enabled;
- * ``ASCII`` if the ``LC_CTYPE`` locale is ``"C"``,
- ``nl_langinfo(CODESET)`` returns the ``ASCII`` encoding (or an alias),
- and :c:func:`mbstowcs` and :c:func:`wcstombs` functions uses the
- ``ISO-8859-1`` encoding.
- * the current locale encoding.
+ Decode a byte string from the :term:`filesystem encoding and error handler`.
+ If the error handler is :ref:`surrogateescape error handler
+ <surrogateescape>`, undecodable bytes are decoded as characters in range
+ U+DC80..U+DCFF; and if a byte sequence can be decoded as a surrogate
+ character, the bytes are escaped using the surrogateescape error handler
+ instead of decoding them.
Return a pointer to a newly allocated wide character string, use
:c:func:`PyMem_RawFree` to free the memory. If size is not ``NULL``, write
@@ -143,6 +142,10 @@
not ``NULL``, ``*size`` is set to ``(size_t)-1`` on memory error or set to
``(size_t)-2`` on decoding error.
+ The :term:`filesystem encoding and error handler` are selected by
+ :c:func:`PyConfig_Read`: see :c:member:`~PyConfig.filesystem_encoding` and
+ :c:member:`~PyConfig.filesystem_errors` members of :c:type:`PyConfig`.
+
Decoding errors should never happen, unless there is a bug in the C
library.
@@ -157,7 +160,8 @@
.. versionadded:: 3.5
.. versionchanged:: 3.7
- The function now uses the UTF-8 encoding in the UTF-8 mode.
+ The function now uses the UTF-8 encoding in the :ref:`Python UTF-8 Mode
+ <utf8-mode>`.
.. versionchanged:: 3.8
The function now uses the UTF-8 encoding on Windows if
@@ -166,22 +170,10 @@
.. c:function:: char* Py_EncodeLocale(const wchar_t *text, size_t *error_pos)
- Encode a wide character string to the locale encoding with the
- :ref:`surrogateescape error handler <surrogateescape>`: surrogate characters
- in the range U+DC80..U+DCFF are converted to bytes 0x80..0xFF.
-
- Encoding, highest priority to lowest priority:
-
- * ``UTF-8`` on macOS, Android, and VxWorks;
- * ``UTF-8`` on Windows if :c:data:`Py_LegacyWindowsFSEncodingFlag` is zero;
- * ``UTF-8`` if the Python UTF-8 mode is enabled;
- * ``ASCII`` if the ``LC_CTYPE`` locale is ``"C"``,
- ``nl_langinfo(CODESET)`` returns the ``ASCII`` encoding (or an alias),
- and :c:func:`mbstowcs` and :c:func:`wcstombs` functions uses the
- ``ISO-8859-1`` encoding.
- * the current locale encoding.
-
- The function uses the UTF-8 encoding in the Python UTF-8 mode.
+ Encode a wide character string to the :term:`filesystem encoding and error
+ handler`. If the error handler is :ref:`surrogateescape error handler
+ <surrogateescape>`, surrogate characters in the range U+DC80..U+DCFF are
+ converted to bytes 0x80..0xFF.
Return a pointer to a newly allocated byte string, use :c:func:`PyMem_Free`
to free the memory. Return ``NULL`` on encoding error or memory allocation
@@ -190,9 +182,18 @@
If error_pos is not ``NULL``, ``*error_pos`` is set to ``(size_t)-1`` on
success, or set to the index of the invalid character on encoding error.
+ The :term:`filesystem encoding and error handler` are selected by
+ :c:func:`PyConfig_Read`: see :c:member:`~PyConfig.filesystem_encoding` and
+ :c:member:`~PyConfig.filesystem_errors` members of :c:type:`PyConfig`.
+
Use the :c:func:`Py_DecodeLocale` function to decode the bytes string back
to a wide character string.
+ .. warning::
+ This function must not be called before :ref:`Python is preinitialized
+ <c-preinit>` and so that the LC_CTYPE locale is properly configured: see
+ the :c:func:`Py_PreInitialize` function.
+
.. seealso::
The :c:func:`PyUnicode_EncodeFSDefault` and
@@ -201,7 +202,8 @@
.. versionadded:: 3.5
.. versionchanged:: 3.7
- The function now uses the UTF-8 encoding in the UTF-8 mode.
+ The function now uses the UTF-8 encoding in the :ref:`Python UTF-8 Mode
+ <utf8-mode>`.
.. versionchanged:: 3.8
The function now uses the UTF-8 encoding on Windows if
diff --git a/Doc/c-api/unicode.rst b/Doc/c-api/unicode.rst
index 54bd0a3..b7f99d3 100644
--- a/Doc/c-api/unicode.rst
+++ b/Doc/c-api/unicode.rst
@@ -783,7 +783,7 @@
:c:data:`Py_FileSystemDefaultEncoding` (the locale encoding read at
Python startup).
- This function ignores the Python UTF-8 mode.
+ This function ignores the :ref:`Python UTF-8 Mode <utf8-mode>`.
.. seealso::
@@ -819,7 +819,7 @@
:c:data:`Py_FileSystemDefaultEncoding` (the locale encoding read at
Python startup).
- This function ignores the Python UTF-8 mode.
+ This function ignores the :ref:`Python UTF-8 Mode <utf8-mode>`.
.. seealso::
@@ -878,8 +878,7 @@
.. c:function:: PyObject* PyUnicode_DecodeFSDefaultAndSize(const char *s, Py_ssize_t size)
- Decode a string using :c:data:`Py_FileSystemDefaultEncoding` and the
- :c:data:`Py_FileSystemDefaultEncodeErrors` error handler.
+ Decode a string from the :term:`filesystem encoding and error handler`.
If :c:data:`Py_FileSystemDefaultEncoding` is not set, fall back to the
locale encoding.
@@ -899,8 +898,8 @@
.. c:function:: PyObject* PyUnicode_DecodeFSDefault(const char *s)
- Decode a null-terminated string using :c:data:`Py_FileSystemDefaultEncoding`
- and the :c:data:`Py_FileSystemDefaultEncodeErrors` error handler.
+ Decode a null-terminated string from the :term:`filesystem encoding and
+ error handler`.
If :c:data:`Py_FileSystemDefaultEncoding` is not set, fall back to the
locale encoding.
diff --git a/Doc/c-api/veryhigh.rst b/Doc/c-api/veryhigh.rst
index b908cb8..0f760ea 100644
--- a/Doc/c-api/veryhigh.rst
+++ b/Doc/c-api/veryhigh.rst
@@ -112,9 +112,9 @@
Similar to :c:func:`PyRun_SimpleStringFlags`, but the Python source code is read
from *fp* instead of an in-memory string. *filename* should be the name of
- the file, it is decoded from the filesystem encoding
- (:func:`sys.getfilesystemencoding`). If *closeit* is true, the file is
- closed before PyRun_SimpleFileExFlags returns.
+ the file, it is decoded from :term:`filesystem encoding and error handler`.
+ If *closeit* is true, the file is closed before
+ ``PyRun_SimpleFileExFlags()`` returns.
.. note::
On Windows, *fp* should be opened as binary mode (e.g. ``fopen(filename, "rb")``).
@@ -132,7 +132,7 @@
Read and execute a single statement from a file associated with an
interactive device according to the *flags* argument. The user will be
prompted using ``sys.ps1`` and ``sys.ps2``. *filename* is decoded from the
- filesystem encoding (:func:`sys.getfilesystemencoding`).
+ :term:`filesystem encoding and error handler`.
Returns ``0`` when the input was
executed successfully, ``-1`` if there was an exception, or an error code
@@ -151,9 +151,8 @@
Read and execute statements from a file associated with an interactive device
until EOF is reached. The user will be prompted using ``sys.ps1`` and
- ``sys.ps2``. *filename* is decoded from the filesystem encoding
- (:func:`sys.getfilesystemencoding`). Returns ``0`` at EOF or a negative
- number upon failure.
+ ``sys.ps2``. *filename* is decoded from the :term:`filesystem encoding and
+ error handler`. Returns ``0`` at EOF or a negative number upon failure.
.. c:var:: int (*PyOS_InputHook)(void)
@@ -206,8 +205,8 @@
Parse Python source code from *str* using the start token *start* according to
the *flags* argument. The result can be used to create a code object which can
be evaluated efficiently. This is useful if a code fragment must be evaluated
- many times. *filename* is decoded from the filesystem encoding
- (:func:`sys.getfilesystemencoding`).
+ many times. *filename* is decoded from the :term:`filesystem encoding and
+ error handler`.
.. c:function:: struct _node* PyParser_SimpleParseFile(FILE *fp, const char *filename, int start)
@@ -262,7 +261,7 @@
Similar to :c:func:`PyRun_StringFlags`, but the Python source code is read from
*fp* instead of an in-memory string. *filename* should be the name of the file,
- it is decoded from the filesystem encoding (:func:`sys.getfilesystemencoding`).
+ it is decoded from the :term:`filesystem encoding and error handler`.
If *closeit* is true, the file is closed before :c:func:`PyRun_FileExFlags`
returns.
@@ -301,7 +300,7 @@
.. c:function:: PyObject* Py_CompileStringExFlags(const char *str, const char *filename, int start, PyCompilerFlags *flags, int optimize)
Like :c:func:`Py_CompileStringObject`, but *filename* is a byte string
- decoded from the filesystem encoding (:func:`os.fsdecode`).
+ decoded from the :term:`filesystem encoding and error handler`.
.. versionadded:: 3.2