PyTokenizer_FindEncoding() always failed because it set the tokenizer state
with only a file pointer when it called fp_setreadl() which expected a file
path. Changed fp_setreadl() to use either a file path or file descriptor
(derived from the file pointer) to fix the issue.
Closes issue 3594.
Reviewed by Antoine Pitrou and Benjamin Peterson.
diff --git a/Parser/tokenizer.c b/Parser/tokenizer.c
index e2da3e5..e4cf8e4 100644
--- a/Parser/tokenizer.c
+++ b/Parser/tokenizer.c
@@ -448,8 +448,12 @@
if (io == NULL)
goto cleanup;
- stream = PyObject_CallMethod(io, "open", "ssis",
- tok->filename, "r", -1, enc);
+ if (tok->filename)
+ stream = PyObject_CallMethod(io, "open", "ssis",
+ tok->filename, "r", -1, enc);
+ else
+ stream = PyObject_CallMethod(io, "open", "isis",
+ fileno(tok->fp), "r", -1, enc);
if (stream == NULL)
goto cleanup;