bpo-40334: Catch E_EOF error, when the tokenizer returns ERRORTOKEN (GH-19743)

An E_EOF error was only being caught after the parser exited before this commit. There are some cases though, where the tokenizer returns ERRORTOKEN *and* has set an E_EOF error (like when EOF directly follows a line continuation character) which weren't correctly handled before.
2 files changed