Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
The utf-16* and utf-32* encoders no longer allow surrogate code points
(U+D800-U+DFFF) to be encoded.
The utf-32* decoders no longer decode byte sequences that correspond to
surrogate code points.
The surrogatepass error handler now works with the utf-16* and utf-32* codecs.
Based on patches by Victor Stinner and Kang-Hao (Kenny) Lu.
diff --git a/Doc/library/codecs.rst b/Doc/library/codecs.rst
index 0925e82..358fde7 100644
--- a/Doc/library/codecs.rst
+++ b/Doc/library/codecs.rst
@@ -365,18 +365,23 @@
| | in :pep:`383`. |
+-------------------------+-----------------------------------------------+
-In addition, the following error handlers are specific to a single codec:
+In addition, the following error handlers are specific to Unicode encoding
+schemes:
-+-------------------+---------+-------------------------------------------+
-| Value | Codec | Meaning |
-+===================+=========+===========================================+
-|``'surrogatepass'``| utf-8 | Allow encoding and decoding of surrogate |
-| | | codes in UTF-8. |
-+-------------------+---------+-------------------------------------------+
++-------------------+------------------------+-------------------------------------------+
+| Value | Codec | Meaning |
++===================+========================+===========================================+
+|``'surrogatepass'``| utf-8, utf-16, utf-32, | Allow encoding and decoding of surrogate |
+| | utf-16-be, utf-16-le, | codes in all the Unicode encoding schemes.|
+| | utf-32-be, utf-32-le | |
++-------------------+------------------------+-------------------------------------------+
.. versionadded:: 3.1
The ``'surrogateescape'`` and ``'surrogatepass'`` error handlers.
+.. versionchanged:: 3.4
+ The ``'surrogatepass'`` error handlers now works with utf-16\* and utf-32\* codecs.
+
The set of allowed values can be extended via :meth:`register_error`.
@@ -1167,6 +1172,12 @@
| utf_8_sig | | all languages |
+-----------------+--------------------------------+--------------------------------+
+.. versionchanged:: 3.4
+ The utf-16\* and utf-32\* encoders no longer allow surrogate code points
+ (U+D800--U+DFFF) to be encoded. The utf-32\* decoders no longer decode
+ byte sequences that correspond to surrogate code points.
+
+
Python Specific Encodings
-------------------------
diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst
index 197702a..f72f544 100644
--- a/Doc/whatsnew/3.4.rst
+++ b/Doc/whatsnew/3.4.rst
@@ -253,6 +253,13 @@
``__main__.__file__`` when a script has been executed directly using
a relative path (Contributed by Brett Cannon in :issue:`18416`).
+* Now all the UTF-\* codecs (except UTF-7) reject surrogates during both
+ encoding and decoding unless the ``surrogatepass`` error handler is used,
+ with the exception of the UTF-16 decoder that accepts valid surrogate pairs,
+ and the UTF-16 encoder that produces them while encoding non-BMP characters.
+ Contributed by Victor Stinner, Kang-Hao (Kenny) Lu and Serhiy Storchaka in
+ :issue:`12892`.
+
New Modules
===========