bpo-35766: Merge typed_ast back into CPython (GH-11645)
diff --git a/Doc/library/ast.rst b/Doc/library/ast.rst
index 7715a28..3df7f9e 100644
--- a/Doc/library/ast.rst
+++ b/Doc/library/ast.rst
@@ -126,16 +126,33 @@
Apart from the node classes, the :mod:`ast` module defines these utility functions
and classes for traversing abstract syntax trees:
-.. function:: parse(source, filename='<unknown>', mode='exec')
+.. function:: parse(source, filename='<unknown>', mode='exec', *, type_comments=False)
Parse the source into an AST node. Equivalent to ``compile(source,
filename, mode, ast.PyCF_ONLY_AST)``.
+ If ``type_comments=True`` is given, the parser is modified to check
+ and return type comments as specified by :pep:`484` and :pep:`526`.
+ This is equivalent to adding :data:`ast.PyCF_TYPE_COMMENTS` to the
+ flags passed to :func:`compile()`. This will report syntax errors
+ for misplaced type comments. Without this flag, type comments will
+ be ignored, and the ``type_comment`` field on selected AST nodes
+ will always be ``None``. In addition, the locations of ``# type:
+ ignore`` comments will be returned as the ``type_ignores``
+ attribute of :class:`Module` (otherwise it is always an empty list).
+
+ In addition, if ``mode`` is ``'func_type'``, the input syntax is
+ modified to correspond to :pep:`484` "signature type comments",
+ e.g. ``(str, int) -> List[str]``.
+
.. warning::
It is possible to crash the Python interpreter with a
sufficiently large/complex string due to stack depth limitations
in Python's AST compiler.
+ .. versionchanged:: 3.8
+ Added ``type_comments=True`` and ``mode='func_type'``.
+
.. function:: literal_eval(node_or_string)
diff --git a/Doc/library/token-list.inc b/Doc/library/token-list.inc
index 3ea9439..cb9fcd7 100644
--- a/Doc/library/token-list.inc
+++ b/Doc/library/token-list.inc
@@ -203,6 +203,10 @@
.. data:: OP
+.. data:: TYPE_IGNORE
+
+.. data:: TYPE_COMMENT
+
.. data:: ERRORTOKEN
.. data:: N_TOKENS
diff --git a/Doc/library/token.rst b/Doc/library/token.rst
index 5358eb5..4936e9a 100644
--- a/Doc/library/token.rst
+++ b/Doc/library/token.rst
@@ -69,6 +69,13 @@
always be an ``ENCODING`` token.
+.. data:: TYPE_COMMENT
+
+ Token value indicating that a type comment was recognized. Such
+ tokens are only produced when :func:`ast.parse()` is invoked with
+ ``type_comments=True``.
+
+
.. versionchanged:: 3.5
Added :data:`AWAIT` and :data:`ASYNC` tokens.
@@ -78,3 +85,6 @@
.. versionchanged:: 3.7
Removed :data:`AWAIT` and :data:`ASYNC` tokens. "async" and "await" are
now tokenized as :data:`NAME` tokens.
+
+.. versionchanged:: 3.8
+ Added :data:`TYPE_COMMENT`.