Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 1 | :mod:`tokenize` --- Tokenizer for Python source |
| 2 | =============================================== |
| 3 | |
| 4 | .. module:: tokenize |
| 5 | :synopsis: Lexical scanner for Python source code. |
| 6 | .. moduleauthor:: Ka Ping Yee |
| 7 | .. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> |
| 8 | |
| 9 | |
| 10 | The :mod:`tokenize` module provides a lexical scanner for Python source code, |
Trent Nelson | 428de65 | 2008-03-18 22:41:35 +0000 | [diff] [blame] | 11 | implemented in Python. The scanner in this module returns comments as tokens |
| 12 | as well, making it useful for implementing "pretty-printers," including |
| 13 | colorizers for on-screen displays. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 14 | |
Georg Brandl | 9afde1c | 2007-11-01 20:32:30 +0000 | [diff] [blame] | 15 | The primary entry point is a :term:`generator`: |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 16 | |
Trent Nelson | 428de65 | 2008-03-18 22:41:35 +0000 | [diff] [blame] | 17 | .. function:: tokenize(readline) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 18 | |
Trent Nelson | 428de65 | 2008-03-18 22:41:35 +0000 | [diff] [blame] | 19 | The :func:`tokenize` generator requires one argument, *readline*, which |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 20 | must be a callable object which provides the same interface as the |
Antoine Pitrou | fa83395 | 2010-01-04 19:55:11 +0000 | [diff] [blame] | 21 | :meth:`io.IOBase.readline` method of file objects. Each call to the |
| 22 | function should return one line of input as bytes. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 23 | |
Georg Brandl | 48310cd | 2009-01-03 21:18:54 +0000 | [diff] [blame] | 24 | The generator produces 5-tuples with these members: the token type; the |
| 25 | token string; a 2-tuple ``(srow, scol)`` of ints specifying the row and |
| 26 | column where the token begins in the source; a 2-tuple ``(erow, ecol)`` of |
| 27 | ints specifying the row and column where the token ends in the source; and |
Georg Brandl | c28e1fa | 2008-06-10 19:20:26 +0000 | [diff] [blame] | 28 | the line on which the token was found. The line passed (the last tuple item) |
Raymond Hettinger | a48db39 | 2009-04-29 00:34:27 +0000 | [diff] [blame] | 29 | is the *logical* line; continuation lines are included. The 5 tuple is |
| 30 | returned as a :term:`named tuple` with the field names: |
| 31 | ``type string start end line``. |
| 32 | |
| 33 | .. versionchanged:: 3.1 |
| 34 | Added support for named tuples. |
Georg Brandl | 48310cd | 2009-01-03 21:18:54 +0000 | [diff] [blame] | 35 | |
Georg Brandl | c28e1fa | 2008-06-10 19:20:26 +0000 | [diff] [blame] | 36 | :func:`tokenize` determines the source encoding of the file by looking for a |
| 37 | UTF-8 BOM or encoding cookie, according to :pep:`263`. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 38 | |
Georg Brandl | 55ac8f0 | 2007-09-01 13:51:09 +0000 | [diff] [blame] | 39 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 40 | All constants from the :mod:`token` module are also exported from |
Trent Nelson | 428de65 | 2008-03-18 22:41:35 +0000 | [diff] [blame] | 41 | :mod:`tokenize`, as are three additional token type values: |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 42 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 43 | .. data:: COMMENT |
| 44 | |
| 45 | Token value used to indicate a comment. |
| 46 | |
| 47 | |
| 48 | .. data:: NL |
| 49 | |
| 50 | Token value used to indicate a non-terminating newline. The NEWLINE token |
Georg Brandl | 48310cd | 2009-01-03 21:18:54 +0000 | [diff] [blame] | 51 | indicates the end of a logical line of Python code; NL tokens are generated |
Trent Nelson | 428de65 | 2008-03-18 22:41:35 +0000 | [diff] [blame] | 52 | when a logical line of code is continued over multiple physical lines. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 53 | |
Trent Nelson | 428de65 | 2008-03-18 22:41:35 +0000 | [diff] [blame] | 54 | |
| 55 | .. data:: ENCODING |
| 56 | |
Georg Brandl | 48310cd | 2009-01-03 21:18:54 +0000 | [diff] [blame] | 57 | Token value that indicates the encoding used to decode the source bytes |
| 58 | into text. The first token returned by :func:`tokenize` will always be an |
Trent Nelson | 428de65 | 2008-03-18 22:41:35 +0000 | [diff] [blame] | 59 | ENCODING token. |
| 60 | |
| 61 | |
Georg Brandl | 48310cd | 2009-01-03 21:18:54 +0000 | [diff] [blame] | 62 | Another function is provided to reverse the tokenization process. This is |
| 63 | useful for creating tools that tokenize a script, modify the token stream, and |
Trent Nelson | 428de65 | 2008-03-18 22:41:35 +0000 | [diff] [blame] | 64 | write back the modified script. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 65 | |
| 66 | |
| 67 | .. function:: untokenize(iterable) |
| 68 | |
Trent Nelson | 428de65 | 2008-03-18 22:41:35 +0000 | [diff] [blame] | 69 | Converts tokens back into Python source code. The *iterable* must return |
Georg Brandl | 48310cd | 2009-01-03 21:18:54 +0000 | [diff] [blame] | 70 | sequences with at least two elements, the token type and the token string. |
Trent Nelson | 428de65 | 2008-03-18 22:41:35 +0000 | [diff] [blame] | 71 | Any additional sequence elements are ignored. |
Georg Brandl | 48310cd | 2009-01-03 21:18:54 +0000 | [diff] [blame] | 72 | |
Trent Nelson | 428de65 | 2008-03-18 22:41:35 +0000 | [diff] [blame] | 73 | The reconstructed script is returned as a single string. The result is |
| 74 | guaranteed to tokenize back to match the input so that the conversion is |
Georg Brandl | 48310cd | 2009-01-03 21:18:54 +0000 | [diff] [blame] | 75 | lossless and round-trips are assured. The guarantee applies only to the |
| 76 | token type and token string as the spacing between tokens (column |
Trent Nelson | 428de65 | 2008-03-18 22:41:35 +0000 | [diff] [blame] | 77 | positions) may change. |
Georg Brandl | 48310cd | 2009-01-03 21:18:54 +0000 | [diff] [blame] | 78 | |
| 79 | It returns bytes, encoded using the ENCODING token, which is the first |
Trent Nelson | 428de65 | 2008-03-18 22:41:35 +0000 | [diff] [blame] | 80 | token sequence output by :func:`tokenize`. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 81 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 82 | |
Trent Nelson | 428de65 | 2008-03-18 22:41:35 +0000 | [diff] [blame] | 83 | :func:`tokenize` needs to detect the encoding of source files it tokenizes. The |
| 84 | function it uses to do this is available: |
| 85 | |
| 86 | .. function:: detect_encoding(readline) |
| 87 | |
Georg Brandl | 48310cd | 2009-01-03 21:18:54 +0000 | [diff] [blame] | 88 | The :func:`detect_encoding` function is used to detect the encoding that |
Georg Brandl | ae2dbe2 | 2009-03-13 19:04:40 +0000 | [diff] [blame] | 89 | should be used to decode a Python source file. It requires one argument, |
Trent Nelson | 428de65 | 2008-03-18 22:41:35 +0000 | [diff] [blame] | 90 | readline, in the same way as the :func:`tokenize` generator. |
Georg Brandl | 48310cd | 2009-01-03 21:18:54 +0000 | [diff] [blame] | 91 | |
Trent Nelson | 428de65 | 2008-03-18 22:41:35 +0000 | [diff] [blame] | 92 | It will call readline a maximum of twice, and return the encoding used |
| 93 | (as a string) and a list of any lines (not decoded from bytes) it has read |
| 94 | in. |
Georg Brandl | 48310cd | 2009-01-03 21:18:54 +0000 | [diff] [blame] | 95 | |
Ezio Melotti | c5c8ff9 | 2009-12-20 12:26:45 +0000 | [diff] [blame] | 96 | It detects the encoding from the presence of a UTF-8 BOM or an encoding |
| 97 | cookie as specified in :pep:`263`. If both a BOM and a cookie are present, |
Trent Nelson | 428de65 | 2008-03-18 22:41:35 +0000 | [diff] [blame] | 98 | but disagree, a SyntaxError will be raised. |
Trent Nelson | 428de65 | 2008-03-18 22:41:35 +0000 | [diff] [blame] | 99 | |
Ezio Melotti | c5c8ff9 | 2009-12-20 12:26:45 +0000 | [diff] [blame] | 100 | If no encoding is specified, then the default of ``'utf-8'`` will be returned. |
Georg Brandl | 48310cd | 2009-01-03 21:18:54 +0000 | [diff] [blame] | 101 | |
| 102 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 103 | Example of a script re-writer that transforms float literals into Decimal |
| 104 | objects:: |
| 105 | |
Ezio Melotti | c5c8ff9 | 2009-12-20 12:26:45 +0000 | [diff] [blame] | 106 | from tokenize import tokenize, untokenize, NUMBER, STRING, NAME, OP |
| 107 | from io import BytesIO |
| 108 | |
Trent Nelson | 428de65 | 2008-03-18 22:41:35 +0000 | [diff] [blame] | 109 | def decistmt(s): |
| 110 | """Substitute Decimals for floats in a string of statements. |
Georg Brandl | 48310cd | 2009-01-03 21:18:54 +0000 | [diff] [blame] | 111 | |
Trent Nelson | 428de65 | 2008-03-18 22:41:35 +0000 | [diff] [blame] | 112 | >>> from decimal import Decimal |
| 113 | >>> s = 'print(+21.3e-5*-.1234/81.7)' |
| 114 | >>> decistmt(s) |
| 115 | "print (+Decimal ('21.3e-5')*-Decimal ('.1234')/Decimal ('81.7'))" |
Georg Brandl | 48310cd | 2009-01-03 21:18:54 +0000 | [diff] [blame] | 116 | |
Trent Nelson | 428de65 | 2008-03-18 22:41:35 +0000 | [diff] [blame] | 117 | The format of the exponent is inherited from the platform C library. |
| 118 | Known cases are "e-007" (Windows) and "e-07" (not Windows). Since |
| 119 | we're only showing 12 digits, and the 13th isn't close to 5, the |
| 120 | rest of the output should be platform-independent. |
Georg Brandl | 48310cd | 2009-01-03 21:18:54 +0000 | [diff] [blame] | 121 | |
Trent Nelson | 428de65 | 2008-03-18 22:41:35 +0000 | [diff] [blame] | 122 | >>> exec(s) #doctest: +ELLIPSIS |
| 123 | -3.21716034272e-0...7 |
Georg Brandl | 48310cd | 2009-01-03 21:18:54 +0000 | [diff] [blame] | 124 | |
Trent Nelson | 428de65 | 2008-03-18 22:41:35 +0000 | [diff] [blame] | 125 | Output from calculations with Decimal should be identical across all |
| 126 | platforms. |
Georg Brandl | 48310cd | 2009-01-03 21:18:54 +0000 | [diff] [blame] | 127 | |
Trent Nelson | 428de65 | 2008-03-18 22:41:35 +0000 | [diff] [blame] | 128 | >>> exec(decistmt(s)) |
| 129 | -3.217160342717258261933904529E-7 |
| 130 | """ |
| 131 | result = [] |
| 132 | g = tokenize(BytesIO(s.encode('utf-8')).readline) # tokenize the string |
| 133 | for toknum, tokval, _, _, _ in g: |
| 134 | if toknum == NUMBER and '.' in tokval: # replace NUMBER tokens |
| 135 | result.extend([ |
| 136 | (NAME, 'Decimal'), |
| 137 | (OP, '('), |
| 138 | (STRING, repr(tokval)), |
| 139 | (OP, ')') |
| 140 | ]) |
| 141 | else: |
| 142 | result.append((toknum, tokval)) |
| 143 | return untokenize(result).decode('utf-8') |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 144 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 145 | |