Issue #10335: Add tokenize.open(), detect the file encoding using
tokenize.detect_encoding() and open it in read only mode.
diff --git a/Doc/library/tokenize.rst b/Doc/library/tokenize.rst
index dbd01c4..6a96609 100644
--- a/Doc/library/tokenize.rst
+++ b/Doc/library/tokenize.rst
@@ -101,14 +101,16 @@
If no encoding is specified, then the default of ``'utf-8'`` will be
returned.
- :func:`detect_encoding` is useful for robustly reading Python source files.
- A common pattern for this follows::
+ Use :func:`open` to open Python source files: it uses
+ :func:`detect_encoding` to detect the file encoding.
- def read_python_source(file_name):
- with open(file_name, "rb") as fp:
- encoding = tokenize.detect_encoding(fp.readline)[0]
- with open(file_name, "r", encoding=encoding) as fp:
- return fp.read()
+
+.. function:: open(filename)
+
+ Open a file in read only mode using the encoding detected by
+ :func:`detect_encoding`.
+
+ .. versionadded:: 3.2
Example of a script rewriter that transforms float literals into Decimal
@@ -153,4 +155,3 @@
result.append((toknum, tokval))
return untokenize(result).decode('utf-8')
-