bpo-42236: Enhance init and encoding documentation (GH-23109)
Enhance the documentation of the Python startup, filesystem encoding
and error handling, locale encoding. Add a new "Python UTF-8 Mode"
section.
* Add "locale encoding" and "filesystem encoding and error handler"
to the glossary
* Remove documentation from Include/cpython/initconfig.h: move it to
Doc/c-api/init_config.rst.
* Doc/c-api/init_config.rst:
* Document command line options and environment variables
* Document default values.
* Add a new "Python UTF-8 Mode" section in Doc/library/os.rst.
* Add warnings to Py_DecodeLocale() and Py_EncodeLocale() docs.
* Document how Python selects the filesystem encoding and error
handler at a single place: PyConfig.filesystem_encoding and
PyConfig.filesystem_errors.
* PyConfig: move orig_argv member at the right place.
diff --git a/Doc/c-api/sys.rst b/Doc/c-api/sys.rst
index 9ac9179..97717f5 100644
--- a/Doc/c-api/sys.rst
+++ b/Doc/c-api/sys.rst
@@ -118,22 +118,21 @@
.. c:function:: wchar_t* Py_DecodeLocale(const char* arg, size_t *size)
- Decode a byte string from the locale encoding with the :ref:`surrogateescape
- error handler <surrogateescape>`: undecodable bytes are decoded as
- characters in range U+DC80..U+DCFF. If a byte sequence can be decoded as a
- surrogate character, escape the bytes using the surrogateescape error
- handler instead of decoding them.
+ .. warning::
+ This function should not be called directly: use the :c:type:`PyConfig`
+ API with the :c:func:`PyConfig_SetBytesString` function which ensures
+ that :ref:`Python is preinitialized <c-preinit>`.
- Encoding, highest priority to lowest priority:
+ This function must not be called before :ref:`Python is preinitialized
+ <c-preinit>` and so that the LC_CTYPE locale is properly configured: see
+ the :c:func:`Py_PreInitialize` function.
- * ``UTF-8`` on macOS, Android, and VxWorks;
- * ``UTF-8`` on Windows if :c:data:`Py_LegacyWindowsFSEncodingFlag` is zero;
- * ``UTF-8`` if the Python UTF-8 mode is enabled;
- * ``ASCII`` if the ``LC_CTYPE`` locale is ``"C"``,
- ``nl_langinfo(CODESET)`` returns the ``ASCII`` encoding (or an alias),
- and :c:func:`mbstowcs` and :c:func:`wcstombs` functions uses the
- ``ISO-8859-1`` encoding.
- * the current locale encoding.
+ Decode a byte string from the :term:`filesystem encoding and error handler`.
+ If the error handler is :ref:`surrogateescape error handler
+ <surrogateescape>`, undecodable bytes are decoded as characters in range
+ U+DC80..U+DCFF; and if a byte sequence can be decoded as a surrogate
+ character, the bytes are escaped using the surrogateescape error handler
+ instead of decoding them.
Return a pointer to a newly allocated wide character string, use
:c:func:`PyMem_RawFree` to free the memory. If size is not ``NULL``, write
@@ -143,6 +142,10 @@
not ``NULL``, ``*size`` is set to ``(size_t)-1`` on memory error or set to
``(size_t)-2`` on decoding error.
+ The :term:`filesystem encoding and error handler` are selected by
+ :c:func:`PyConfig_Read`: see :c:member:`~PyConfig.filesystem_encoding` and
+ :c:member:`~PyConfig.filesystem_errors` members of :c:type:`PyConfig`.
+
Decoding errors should never happen, unless there is a bug in the C
library.
@@ -157,7 +160,8 @@
.. versionadded:: 3.5
.. versionchanged:: 3.7
- The function now uses the UTF-8 encoding in the UTF-8 mode.
+ The function now uses the UTF-8 encoding in the :ref:`Python UTF-8 Mode
+ <utf8-mode>`.
.. versionchanged:: 3.8
The function now uses the UTF-8 encoding on Windows if
@@ -166,22 +170,10 @@
.. c:function:: char* Py_EncodeLocale(const wchar_t *text, size_t *error_pos)
- Encode a wide character string to the locale encoding with the
- :ref:`surrogateescape error handler <surrogateescape>`: surrogate characters
- in the range U+DC80..U+DCFF are converted to bytes 0x80..0xFF.
-
- Encoding, highest priority to lowest priority:
-
- * ``UTF-8`` on macOS, Android, and VxWorks;
- * ``UTF-8`` on Windows if :c:data:`Py_LegacyWindowsFSEncodingFlag` is zero;
- * ``UTF-8`` if the Python UTF-8 mode is enabled;
- * ``ASCII`` if the ``LC_CTYPE`` locale is ``"C"``,
- ``nl_langinfo(CODESET)`` returns the ``ASCII`` encoding (or an alias),
- and :c:func:`mbstowcs` and :c:func:`wcstombs` functions uses the
- ``ISO-8859-1`` encoding.
- * the current locale encoding.
-
- The function uses the UTF-8 encoding in the Python UTF-8 mode.
+ Encode a wide character string to the :term:`filesystem encoding and error
+ handler`. If the error handler is :ref:`surrogateescape error handler
+ <surrogateescape>`, surrogate characters in the range U+DC80..U+DCFF are
+ converted to bytes 0x80..0xFF.
Return a pointer to a newly allocated byte string, use :c:func:`PyMem_Free`
to free the memory. Return ``NULL`` on encoding error or memory allocation
@@ -190,9 +182,18 @@
If error_pos is not ``NULL``, ``*error_pos`` is set to ``(size_t)-1`` on
success, or set to the index of the invalid character on encoding error.
+ The :term:`filesystem encoding and error handler` are selected by
+ :c:func:`PyConfig_Read`: see :c:member:`~PyConfig.filesystem_encoding` and
+ :c:member:`~PyConfig.filesystem_errors` members of :c:type:`PyConfig`.
+
Use the :c:func:`Py_DecodeLocale` function to decode the bytes string back
to a wide character string.
+ .. warning::
+ This function must not be called before :ref:`Python is preinitialized
+ <c-preinit>` and so that the LC_CTYPE locale is properly configured: see
+ the :c:func:`Py_PreInitialize` function.
+
.. seealso::
The :c:func:`PyUnicode_EncodeFSDefault` and
@@ -201,7 +202,8 @@
.. versionadded:: 3.5
.. versionchanged:: 3.7
- The function now uses the UTF-8 encoding in the UTF-8 mode.
+ The function now uses the UTF-8 encoding in the :ref:`Python UTF-8 Mode
+ <utf8-mode>`.
.. versionchanged:: 3.8
The function now uses the UTF-8 encoding on Windows if