bpo-40267: Fix message when last input character produces a SyntaxError (GH-19521)
When there is a SyntaxError after reading the last input character from
the tokenizer and if no newline follows it, the error message used to be
`unexpected EOF while parsing`, which is wrong.
diff --git a/Tools/scripts/generate_token.py b/Tools/scripts/generate_token.py
index f2745e8..77bb5bd 100755
--- a/Tools/scripts/generate_token.py
+++ b/Tools/scripts/generate_token.py
@@ -69,6 +69,10 @@
#define ISTERMINAL(x) ((x) < NT_OFFSET)
#define ISNONTERMINAL(x) ((x) >= NT_OFFSET)
#define ISEOF(x) ((x) == ENDMARKER)
+#define ISWHITESPACE(x) ((x) == ENDMARKER || \\
+ (x) == NEWLINE || \\
+ (x) == INDENT || \\
+ (x) == DEDENT)
PyAPI_DATA(const char * const) _PyParser_TokenNames[]; /* Token names */