bpo-30455: Generate all token related code and docs from Grammar/Tokens. (GH-10370)
"Include/token.h", "Lib/token.py" (containing now some data moved from
"Lib/tokenize.py") and new files "Parser/token.c" (containing the code
moved from "Parser/tokenizer.c") and "Doc/library/token-list.inc" (included
in "Doc/library/token.rst") are now generated from "Grammar/Tokens" by
"Tools/scripts/generate_token.py". The script overwrites files only if
needed and can be used on the read-only sources tree.
"Lib/symbol.py" is now generated by "Tools/scripts/generate_symbol_py.py"
instead of been executable itself.
Added new make targets "regen-token" and "regen-symbol" which are now
dependencies of "regen-all".
The documentation contains now strings for operators and punctuation tokens.
diff --git a/Doc/library/token.rst b/Doc/library/token.rst
index 3739910..5358eb5 100644
--- a/Doc/library/token.rst
+++ b/Doc/library/token.rst
@@ -44,64 +44,7 @@
The token constants are:
-.. data:: ENDMARKER
- NAME
- NUMBER
- STRING
- NEWLINE
- INDENT
- DEDENT
- LPAR
- RPAR
- LSQB
- RSQB
- COLON
- COMMA
- SEMI
- PLUS
- MINUS
- STAR
- SLASH
- VBAR
- AMPER
- LESS
- GREATER
- EQUAL
- DOT
- PERCENT
- LBRACE
- RBRACE
- EQEQUAL
- NOTEQUAL
- LESSEQUAL
- GREATEREQUAL
- TILDE
- CIRCUMFLEX
- LEFTSHIFT
- RIGHTSHIFT
- DOUBLESTAR
- PLUSEQUAL
- MINEQUAL
- STAREQUAL
- SLASHEQUAL
- PERCENTEQUAL
- AMPEREQUAL
- VBAREQUAL
- CIRCUMFLEXEQUAL
- LEFTSHIFTEQUAL
- RIGHTSHIFTEQUAL
- DOUBLESTAREQUAL
- DOUBLESLASH
- DOUBLESLASHEQUAL
- AT
- ATEQUAL
- RARROW
- ELLIPSIS
- OP
- ERRORTOKEN
- N_TOKENS
- NT_OFFSET
-
+.. include:: token-list.inc
The following token type values aren't used by the C tokenizer but are needed for
the :mod:`tokenize` module.