document TokenError and unclosed expression behavior (closes #12063)
Patch by Amandine Lee.
diff --git a/Doc/library/tokenize.rst b/Doc/library/tokenize.rst
index 0fc0d7e..3f25a2c 100644
--- a/Doc/library/tokenize.rst
+++ b/Doc/library/tokenize.rst
@@ -98,6 +98,24 @@
.. versionadded:: 2.5
+.. exception:: TokenError
+
+ Raised when either a docstring or expression that may be split over several
+ lines is not completed anywhere in the file, for example::
+
+ """Beginning of
+ docstring
+
+ or::
+
+ [1,
+ 2,
+ 3
+
+Note that unclosed single-quoted strings do not cause an error to be
+raised. They are tokenized as ``ERRORTOKEN``, followed by the tokenization of
+their contents.
+
Example of a script re-writer that transforms float literals into Decimal
objects::