bpo-30455: Generate all token related code and docs from Grammar/Tokens. (GH-10370)
"Include/token.h", "Lib/token.py" (containing now some data moved from
"Lib/tokenize.py") and new files "Parser/token.c" (containing the code
moved from "Parser/tokenizer.c") and "Doc/library/token-list.inc" (included
in "Doc/library/token.rst") are now generated from "Grammar/Tokens" by
"Tools/scripts/generate_token.py". The script overwrites files only if
needed and can be used on the read-only sources tree.
"Lib/symbol.py" is now generated by "Tools/scripts/generate_symbol_py.py"
instead of been executable itself.
Added new make targets "regen-token" and "regen-symbol" which are now
dependencies of "regen-all".
The documentation contains now strings for operators and punctuation tokens.
diff --git a/Doc/library/token-list.inc b/Doc/library/token-list.inc
new file mode 100644
index 0000000..cd6e0f2
--- /dev/null
+++ b/Doc/library/token-list.inc
@@ -0,0 +1,206 @@
+.. Auto-generated by Tools/scripts/generate_token.py
+.. data:: ENDMARKER
+
+.. data:: NAME
+
+.. data:: NUMBER
+
+.. data:: STRING
+
+.. data:: NEWLINE
+
+.. data:: INDENT
+
+.. data:: DEDENT
+
+.. data:: LPAR
+
+ Token value for ``"("``.
+
+.. data:: RPAR
+
+ Token value for ``")"``.
+
+.. data:: LSQB
+
+ Token value for ``"["``.
+
+.. data:: RSQB
+
+ Token value for ``"]"``.
+
+.. data:: COLON
+
+ Token value for ``":"``.
+
+.. data:: COMMA
+
+ Token value for ``","``.
+
+.. data:: SEMI
+
+ Token value for ``";"``.
+
+.. data:: PLUS
+
+ Token value for ``"+"``.
+
+.. data:: MINUS
+
+ Token value for ``"-"``.
+
+.. data:: STAR
+
+ Token value for ``"*"``.
+
+.. data:: SLASH
+
+ Token value for ``"/"``.
+
+.. data:: VBAR
+
+ Token value for ``"|"``.
+
+.. data:: AMPER
+
+ Token value for ``"&"``.
+
+.. data:: LESS
+
+ Token value for ``"<"``.
+
+.. data:: GREATER
+
+ Token value for ``">"``.
+
+.. data:: EQUAL
+
+ Token value for ``"="``.
+
+.. data:: DOT
+
+ Token value for ``"."``.
+
+.. data:: PERCENT
+
+ Token value for ``"%"``.
+
+.. data:: LBRACE
+
+ Token value for ``"{"``.
+
+.. data:: RBRACE
+
+ Token value for ``"}"``.
+
+.. data:: EQEQUAL
+
+ Token value for ``"=="``.
+
+.. data:: NOTEQUAL
+
+ Token value for ``"!="``.
+
+.. data:: LESSEQUAL
+
+ Token value for ``"<="``.
+
+.. data:: GREATEREQUAL
+
+ Token value for ``">="``.
+
+.. data:: TILDE
+
+ Token value for ``"~"``.
+
+.. data:: CIRCUMFLEX
+
+ Token value for ``"^"``.
+
+.. data:: LEFTSHIFT
+
+ Token value for ``"<<"``.
+
+.. data:: RIGHTSHIFT
+
+ Token value for ``">>"``.
+
+.. data:: DOUBLESTAR
+
+ Token value for ``"**"``.
+
+.. data:: PLUSEQUAL
+
+ Token value for ``"+="``.
+
+.. data:: MINEQUAL
+
+ Token value for ``"-="``.
+
+.. data:: STAREQUAL
+
+ Token value for ``"*="``.
+
+.. data:: SLASHEQUAL
+
+ Token value for ``"/="``.
+
+.. data:: PERCENTEQUAL
+
+ Token value for ``"%="``.
+
+.. data:: AMPEREQUAL
+
+ Token value for ``"&="``.
+
+.. data:: VBAREQUAL
+
+ Token value for ``"|="``.
+
+.. data:: CIRCUMFLEXEQUAL
+
+ Token value for ``"^="``.
+
+.. data:: LEFTSHIFTEQUAL
+
+ Token value for ``"<<="``.
+
+.. data:: RIGHTSHIFTEQUAL
+
+ Token value for ``">>="``.
+
+.. data:: DOUBLESTAREQUAL
+
+ Token value for ``"**="``.
+
+.. data:: DOUBLESLASH
+
+ Token value for ``"//"``.
+
+.. data:: DOUBLESLASHEQUAL
+
+ Token value for ``"//="``.
+
+.. data:: AT
+
+ Token value for ``"@"``.
+
+.. data:: ATEQUAL
+
+ Token value for ``"@="``.
+
+.. data:: RARROW
+
+ Token value for ``"->"``.
+
+.. data:: ELLIPSIS
+
+ Token value for ``"..."``.
+
+.. data:: OP
+
+.. data:: ERRORTOKEN
+
+.. data:: N_TOKENS
+
+.. data:: NT_OFFSET
diff --git a/Doc/library/token.rst b/Doc/library/token.rst
index 3739910..5358eb5 100644
--- a/Doc/library/token.rst
+++ b/Doc/library/token.rst
@@ -44,64 +44,7 @@
The token constants are:
-.. data:: ENDMARKER
- NAME
- NUMBER
- STRING
- NEWLINE
- INDENT
- DEDENT
- LPAR
- RPAR
- LSQB
- RSQB
- COLON
- COMMA
- SEMI
- PLUS
- MINUS
- STAR
- SLASH
- VBAR
- AMPER
- LESS
- GREATER
- EQUAL
- DOT
- PERCENT
- LBRACE
- RBRACE
- EQEQUAL
- NOTEQUAL
- LESSEQUAL
- GREATEREQUAL
- TILDE
- CIRCUMFLEX
- LEFTSHIFT
- RIGHTSHIFT
- DOUBLESTAR
- PLUSEQUAL
- MINEQUAL
- STAREQUAL
- SLASHEQUAL
- PERCENTEQUAL
- AMPEREQUAL
- VBAREQUAL
- CIRCUMFLEXEQUAL
- LEFTSHIFTEQUAL
- RIGHTSHIFTEQUAL
- DOUBLESTAREQUAL
- DOUBLESLASH
- DOUBLESLASHEQUAL
- AT
- ATEQUAL
- RARROW
- ELLIPSIS
- OP
- ERRORTOKEN
- N_TOKENS
- NT_OFFSET
-
+.. include:: token-list.inc
The following token type values aren't used by the C tokenizer but are needed for
the :mod:`tokenize` module.