Fred Drake | 6b103f1 | 1999-02-18 21:06:50 +0000 | [diff] [blame] | 1 | \section{\module{tokenize} --- |
| 2 | Tokenizer for Python source} |
| 3 | |
| 4 | \declaremodule{standard}{tokenize} |
| 5 | \modulesynopsis{Lexical scanner for Python source code.} |
| 6 | \moduleauthor{Ka Ping Yee}{} |
| 7 | \sectionauthor{Fred L. Drake, Jr.}{fdrake@acm.org} |
| 8 | |
| 9 | |
| 10 | The \module{tokenize} module provides a lexical scanner for Python |
| 11 | source code, implemented in Python. The scanner in this module |
| 12 | returns comments as tokens as well, making it useful for implementing |
| 13 | ``pretty-printers,'' including colorizers for on-screen displays. |
| 14 | |
Tim Peters | 4efb6e9 | 2001-06-29 23:51:08 +0000 | [diff] [blame] | 15 | The primary entry point is a generator: |
Fred Drake | 6b103f1 | 1999-02-18 21:06:50 +0000 | [diff] [blame] | 16 | |
Tim Peters | 4efb6e9 | 2001-06-29 23:51:08 +0000 | [diff] [blame] | 17 | \begin{funcdesc}{generate_tokens}{readline} |
| 18 | The \function{generate_tokens()} generator requires one argment, |
| 19 | \var{readline}, which must be a callable object which |
| 20 | provides the same interface as the \method{readline()} method of |
| 21 | built-in file objects (see section~\ref{bltin-file-objects}). Each |
| 22 | call to the function should return one line of input as a string. |
| 23 | |
| 24 | The generator produces 5-tuples with these members: |
| 25 | the token type; |
| 26 | the token string; |
| 27 | a 2-tuple \code{(\var{srow}, \var{scol})} of ints specifying the |
| 28 | row and column where the token begins in the source; |
| 29 | a 2-tuple \code{(\var{erow}, \var{ecol})} of ints specifying the |
| 30 | row and column where the token ends in the source; |
| 31 | and the line on which the token was found. |
| 32 | The line passed is the \emph{logical} line; |
| 33 | continuation lines are included. |
| 34 | \versionadded{2.2} |
| 35 | \end{funcdesc} |
| 36 | |
| 37 | An older entry point is retained for backward compatibility: |
Fred Drake | 6b103f1 | 1999-02-18 21:06:50 +0000 | [diff] [blame] | 38 | |
| 39 | \begin{funcdesc}{tokenize}{readline\optional{, tokeneater}} |
| 40 | The \function{tokenize()} function accepts two parameters: one |
Tim Peters | 4efb6e9 | 2001-06-29 23:51:08 +0000 | [diff] [blame] | 41 | representing the input stream, and one providing an output mechanism |
Fred Drake | 6b103f1 | 1999-02-18 21:06:50 +0000 | [diff] [blame] | 42 | for \function{tokenize()}. |
| 43 | |
| 44 | The first parameter, \var{readline}, must be a callable object which |
Fred Drake | 16214fb | 1999-04-23 20:00:53 +0000 | [diff] [blame] | 45 | provides the same interface as the \method{readline()} method of |
Fred Drake | 6b103f1 | 1999-02-18 21:06:50 +0000 | [diff] [blame] | 46 | built-in file objects (see section~\ref{bltin-file-objects}). Each |
| 47 | call to the function should return one line of input as a string. |
| 48 | |
| 49 | The second parameter, \var{tokeneater}, must also be a callable |
Tim Peters | 4efb6e9 | 2001-06-29 23:51:08 +0000 | [diff] [blame] | 50 | object. It is called once for each token, with five arguments, |
| 51 | corresponding to the tuples generated by \function{generate_tokens()}. |
Fred Drake | 6b103f1 | 1999-02-18 21:06:50 +0000 | [diff] [blame] | 52 | \end{funcdesc} |
| 53 | |
| 54 | |
Tim Peters | 4efb6e9 | 2001-06-29 23:51:08 +0000 | [diff] [blame] | 55 | All constants from the \refmodule{token} module are also exported from |
| 56 | \module{tokenize}, as are two additional token type values that might be |
Fred Drake | 6b103f1 | 1999-02-18 21:06:50 +0000 | [diff] [blame] | 57 | passed to the \var{tokeneater} function by \function{tokenize()}: |
| 58 | |
| 59 | \begin{datadesc}{COMMENT} |
| 60 | Token value used to indicate a comment. |
| 61 | \end{datadesc} |
Skip Montanaro | 58177b9 | 2001-02-28 22:05:41 +0000 | [diff] [blame] | 62 | \begin{datadesc}{NL} |
Ka-Ping Yee | ce7298a | 2001-03-23 05:22:12 +0000 | [diff] [blame] | 63 | Token value used to indicate a non-terminating newline. The NEWLINE |
| 64 | token indicates the end of a logical line of Python code; NL tokens |
| 65 | are generated when a logical line of code is continued over multiple |
| 66 | physical lines. |
Skip Montanaro | 58177b9 | 2001-02-28 22:05:41 +0000 | [diff] [blame] | 67 | \end{datadesc} |