Guido van Rossum | 4b73a06 | 1995-10-11 17:30:04 +0000 | [diff] [blame] | 1 | % libparser.tex |
| 2 | % |
| 3 | % Introductory documentation for the new parser built-in module. |
| 4 | % |
| 5 | % Copyright 1995 Virginia Polytechnic Institute and State University |
| 6 | % and Fred L. Drake, Jr. This copyright notice must be distributed on |
| 7 | % all copies, but this document otherwise may be distributed as part |
| 8 | % of the Python distribution. No fee may be charged for this document |
| 9 | % in any representation, either on paper or electronically. This |
| 10 | % restriction does not affect other elements in a distributed package |
| 11 | % in any way. |
| 12 | % |
| 13 | |
| 14 | \section{Built-in Module \sectcode{parser}} |
| 15 | \bimodindex{parser} |
| 16 | |
Guido van Rossum | 4b73a06 | 1995-10-11 17:30:04 +0000 | [diff] [blame] | 17 | The \code{parser} module provides an interface to Python's internal |
| 18 | parser and byte-code compiler. The primary purpose for this interface |
| 19 | is to allow Python code to edit the parse tree of a Python expression |
Fred Drake | 4b7d5a4 | 1996-09-11 21:57:40 +0000 | [diff] [blame] | 20 | and create executable code from this. This is better than trying |
| 21 | to parse and modify an arbitrary Python code fragment as a string |
| 22 | because parsing is performed in a manner identical to the code |
| 23 | forming the application. It is also faster. |
Guido van Rossum | 4b73a06 | 1995-10-11 17:30:04 +0000 | [diff] [blame] | 24 | |
| 25 | There are a few things to note about this module which are important |
| 26 | to making use of the data structures created. This is not a tutorial |
Fred Drake | 4b7d5a4 | 1996-09-11 21:57:40 +0000 | [diff] [blame] | 27 | on editing the parse trees for Python code, but some examples of using |
| 28 | the \code{parser} module are presented. |
Guido van Rossum | 4b73a06 | 1995-10-11 17:30:04 +0000 | [diff] [blame] | 29 | |
| 30 | Most importantly, a good understanding of the Python grammar processed |
| 31 | by the internal parser is required. For full information on the |
| 32 | language syntax, refer to the Language Reference. The parser itself |
| 33 | is created from a grammar specification defined in the file |
Fred Drake | 4b7d5a4 | 1996-09-11 21:57:40 +0000 | [diff] [blame] | 34 | \file{Grammar/Grammar} in the standard Python distribution. The parse |
Guido van Rossum | 4b73a06 | 1995-10-11 17:30:04 +0000 | [diff] [blame] | 35 | trees stored in the ``AST objects'' created by this module are the |
| 36 | actual output from the internal parser when created by the |
| 37 | \code{expr()} or \code{suite()} functions, described below. The AST |
Guido van Rossum | 4747887 | 1996-08-21 14:32:37 +0000 | [diff] [blame] | 38 | objects created by \code{sequence2ast()} faithfully simulate those |
| 39 | structures. Be aware that the values of the sequences which are |
| 40 | considered ``correct'' will vary from one version of Python to another |
| 41 | as the formal grammar for the language is revised. However, |
| 42 | transporting code from one Python version to another as source text |
| 43 | will always allow correct parse trees to be created in the target |
| 44 | version, with the only restriction being that migrating to an older |
| 45 | version of the interpreter will not support more recent language |
| 46 | constructs. The parse trees are not typically compatible from one |
| 47 | version to another, whereas source code has always been |
| 48 | forward-compatible. |
Guido van Rossum | 4b73a06 | 1995-10-11 17:30:04 +0000 | [diff] [blame] | 49 | |
Guido van Rossum | 4747887 | 1996-08-21 14:32:37 +0000 | [diff] [blame] | 50 | Each element of the sequences returned by \code{ast2list} or |
| 51 | \code{ast2tuple()} has a simple form. Sequences representing |
| 52 | non-terminal elements in the grammar always have a length greater than |
| 53 | one. The first element is an integer which identifies a production in |
| 54 | the grammar. These integers are given symbolic names in the C header |
Fred Drake | 4b7d5a4 | 1996-09-11 21:57:40 +0000 | [diff] [blame] | 55 | file \file{Include/graminit.h} and the Python module |
Fred Drake | e061a51 | 1997-10-06 21:40:20 +0000 | [diff] [blame] | 56 | \code{symbol}. Each additional element of the sequence represents |
Guido van Rossum | 4747887 | 1996-08-21 14:32:37 +0000 | [diff] [blame] | 57 | a component of the production as recognized in the input string: these |
| 58 | are always sequences which have the same form as the parent. An |
| 59 | important aspect of this structure which should be noted is that |
| 60 | keywords used to identify the parent node type, such as the keyword |
Fred Drake | 4b7d5a4 | 1996-09-11 21:57:40 +0000 | [diff] [blame] | 61 | \code{if} in an \code{if_stmt}, are included in the node tree without |
Guido van Rossum | 4747887 | 1996-08-21 14:32:37 +0000 | [diff] [blame] | 62 | any special treatment. For example, the \code{if} keyword is |
Guido van Rossum | 4b73a06 | 1995-10-11 17:30:04 +0000 | [diff] [blame] | 63 | represented by the tuple \code{(1, 'if')}, where \code{1} is the |
Fred Drake | 4b7d5a4 | 1996-09-11 21:57:40 +0000 | [diff] [blame] | 64 | numeric value associated with all \code{NAME} tokens, including |
Guido van Rossum | 4747887 | 1996-08-21 14:32:37 +0000 | [diff] [blame] | 65 | variable and function names defined by the user. In an alternate form |
| 66 | returned when line number information is requested, the same token |
| 67 | might be represented as \code{(1, 'if', 12)}, where the \code{12} |
| 68 | represents the line number at which the terminal symbol was found. |
Guido van Rossum | 4b73a06 | 1995-10-11 17:30:04 +0000 | [diff] [blame] | 69 | |
| 70 | Terminal elements are represented in much the same way, but without |
| 71 | any child elements and the addition of the source text which was |
| 72 | identified. The example of the \code{if} keyword above is |
| 73 | representative. The various types of terminal symbols are defined in |
Fred Drake | 4b7d5a4 | 1996-09-11 21:57:40 +0000 | [diff] [blame] | 74 | the C header file \file{Include/token.h} and the Python module |
Fred Drake | e061a51 | 1997-10-06 21:40:20 +0000 | [diff] [blame] | 75 | \code{token}. |
Guido van Rossum | 4b73a06 | 1995-10-11 17:30:04 +0000 | [diff] [blame] | 76 | |
Fred Drake | 4b7d5a4 | 1996-09-11 21:57:40 +0000 | [diff] [blame] | 77 | The AST objects are not required to support the functionality of this |
| 78 | module, but are provided for three purposes: to allow an application |
| 79 | to amortize the cost of processing complex parse trees, to provide a |
| 80 | parse tree representation which conserves memory space when compared |
| 81 | to the Python list or tuple representation, and to ease the creation |
| 82 | of additional modules in C which manipulate parse trees. A simple |
| 83 | ``wrapper'' class may be created in Python to hide the use of AST |
| 84 | objects; the \code{AST} library module provides a variety of such |
| 85 | classes. |
Guido van Rossum | 4b73a06 | 1995-10-11 17:30:04 +0000 | [diff] [blame] | 86 | |
Fred Drake | 4b7d5a4 | 1996-09-11 21:57:40 +0000 | [diff] [blame] | 87 | The \code{parser} module defines functions for a few distinct |
| 88 | purposes. The most important purposes are to create AST objects and |
| 89 | to convert AST objects to other representations such as parse trees |
| 90 | and compiled code objects, but there are also functions which serve to |
| 91 | query the type of parse tree represented by an AST object. |
Guido van Rossum | 4b73a06 | 1995-10-11 17:30:04 +0000 | [diff] [blame] | 92 | |
Guido van Rossum | 4b73a06 | 1995-10-11 17:30:04 +0000 | [diff] [blame] | 93 | \renewcommand{\indexsubitem}{(in module parser)} |
| 94 | |
Fred Drake | 4b7d5a4 | 1996-09-11 21:57:40 +0000 | [diff] [blame] | 95 | |
| 96 | \subsection{Creating AST Objects} |
| 97 | |
| 98 | AST objects may be created from source code or from a parse tree. |
| 99 | When creating an AST object from source, different functions are used |
| 100 | to create the \code{'eval'} and \code{'exec'} forms. |
| 101 | |
| 102 | \begin{funcdesc}{expr}{string} |
| 103 | The \code{expr()} function parses the parameter \code{\var{string}} |
| 104 | as if it were an input to \code{compile(\var{string}, 'eval')}. If |
| 105 | the parse succeeds, an AST object is created to hold the internal |
| 106 | parse tree representation, otherwise an appropriate exception is |
| 107 | thrown. |
| 108 | \end{funcdesc} |
| 109 | |
| 110 | \begin{funcdesc}{suite}{string} |
| 111 | The \code{suite()} function parses the parameter \code{\var{string}} |
| 112 | as if it were an input to \code{compile(\var{string}, 'exec')}. If |
| 113 | the parse succeeds, an AST object is created to hold the internal |
| 114 | parse tree representation, otherwise an appropriate exception is |
| 115 | thrown. |
| 116 | \end{funcdesc} |
| 117 | |
| 118 | \begin{funcdesc}{sequence2ast}{sequence} |
| 119 | This function accepts a parse tree represented as a sequence and |
| 120 | builds an internal representation if possible. If it can validate |
| 121 | that the tree conforms to the Python grammar and all nodes are valid |
| 122 | node types in the host version of Python, an AST object is created |
| 123 | from the internal representation and returned to the called. If there |
| 124 | is a problem creating the internal representation, or if the tree |
| 125 | cannot be validated, a \code{ParserError} exception is thrown. An AST |
| 126 | object created this way should not be assumed to compile correctly; |
| 127 | normal exceptions thrown by compilation may still be initiated when |
| 128 | the AST object is passed to \code{compileast()}. This may indicate |
| 129 | problems not related to syntax (such as a \code{MemoryError} |
| 130 | exception), but may also be due to constructs such as the result of |
| 131 | parsing \code{del f(0)}, which escapes the Python parser but is |
| 132 | checked by the bytecode compiler. |
| 133 | |
| 134 | Sequences representing terminal tokens may be represented as either |
| 135 | two-element lists of the form \code{(1, 'name')} or as three-element |
| 136 | lists of the form \code{(1, 'name', 56)}. If the third element is |
| 137 | present, it is assumed to be a valid line number. The line number |
| 138 | may be specified for any subset of the terminal symbols in the input |
| 139 | tree. |
| 140 | \end{funcdesc} |
| 141 | |
| 142 | \begin{funcdesc}{tuple2ast}{sequence} |
| 143 | This is the same function as \code{sequence2ast()}. This entry point |
| 144 | is maintained for backward compatibility. |
| 145 | \end{funcdesc} |
| 146 | |
| 147 | |
| 148 | \subsection{Converting AST Objects} |
| 149 | |
| 150 | AST objects, regardless of the input used to create them, may be |
| 151 | converted to parse trees represented as list- or tuple- trees, or may |
| 152 | be compiled into executable code objects. Parse trees may be |
| 153 | extracted with or without line numbering information. |
| 154 | |
| 155 | \begin{funcdesc}{ast2list}{ast\optional{\, line_info\code{ = 0}}} |
Guido van Rossum | 4b73a06 | 1995-10-11 17:30:04 +0000 | [diff] [blame] | 156 | This function accepts an AST object from the caller in |
Guido van Rossum | 4747887 | 1996-08-21 14:32:37 +0000 | [diff] [blame] | 157 | \code{\var{ast}} and returns a Python list representing the |
| 158 | equivelent parse tree. The resulting list representation can be used |
Fred Drake | 4b7d5a4 | 1996-09-11 21:57:40 +0000 | [diff] [blame] | 159 | for inspection or the creation of a new parse tree in list form. This |
| 160 | function does not fail so long as memory is available to build the |
| 161 | list representation. If the parse tree will only be used for |
Guido van Rossum | 4747887 | 1996-08-21 14:32:37 +0000 | [diff] [blame] | 162 | inspection, \code{ast2tuple()} should be used instead to reduce memory |
Fred Drake | 4b7d5a4 | 1996-09-11 21:57:40 +0000 | [diff] [blame] | 163 | consumption and fragmentation. When the list representation is |
| 164 | required, this function is significantly faster than retrieving a |
| 165 | tuple representation and converting that to nested lists. |
Guido van Rossum | 4747887 | 1996-08-21 14:32:37 +0000 | [diff] [blame] | 166 | |
Fred Drake | 4b7d5a4 | 1996-09-11 21:57:40 +0000 | [diff] [blame] | 167 | If \code{\var{line_info}} is true, line number information will be |
| 168 | included for all terminal tokens as a third element of the list |
Fred Drake | 9abe64a | 1996-12-05 22:28:43 +0000 | [diff] [blame] | 169 | representing the token. Note that the line number provided specifies |
| 170 | the line on which the token \emph{ends\/}. This information is |
| 171 | omitted if the flag is false or omitted. |
Guido van Rossum | 4b73a06 | 1995-10-11 17:30:04 +0000 | [diff] [blame] | 172 | \end{funcdesc} |
| 173 | |
Fred Drake | 4b7d5a4 | 1996-09-11 21:57:40 +0000 | [diff] [blame] | 174 | \begin{funcdesc}{ast2tuple}{ast\optional{\, line_info\code{ = 0}}} |
Guido van Rossum | 4747887 | 1996-08-21 14:32:37 +0000 | [diff] [blame] | 175 | This function accepts an AST object from the caller in |
| 176 | \code{\var{ast}} and returns a Python tuple representing the |
| 177 | equivelent parse tree. Other than returning a tuple instead of a |
| 178 | list, this function is identical to \code{ast2list()}. |
Guido van Rossum | 4b73a06 | 1995-10-11 17:30:04 +0000 | [diff] [blame] | 179 | |
Fred Drake | 4b7d5a4 | 1996-09-11 21:57:40 +0000 | [diff] [blame] | 180 | If \code{\var{line_info}} is true, line number information will be |
| 181 | included for all terminal tokens as a third element of the list |
| 182 | representing the token. This information is omitted if the flag is |
| 183 | false or omitted. |
Guido van Rossum | 4747887 | 1996-08-21 14:32:37 +0000 | [diff] [blame] | 184 | \end{funcdesc} |
| 185 | |
| 186 | \begin{funcdesc}{compileast}{ast\optional{\, filename\code{ = '<ast>'}}} |
Guido van Rossum | 4b73a06 | 1995-10-11 17:30:04 +0000 | [diff] [blame] | 187 | The Python byte compiler can be invoked on an AST object to produce |
| 188 | code objects which can be used as part of an \code{exec} statement or |
| 189 | a call to the built-in \code{eval()} function. This function provides |
| 190 | the interface to the compiler, passing the internal parse tree from |
| 191 | \code{\var{ast}} to the parser, using the source file name specified |
| 192 | by the \code{\var{filename}} parameter. The default value supplied |
| 193 | for \code{\var{filename}} indicates that the source was an AST object. |
Guido van Rossum | 4747887 | 1996-08-21 14:32:37 +0000 | [diff] [blame] | 194 | |
| 195 | Compiling an AST object may result in exceptions related to |
| 196 | compilation; an example would be a \code{SyntaxError} caused by the |
Fred Drake | 4b7d5a4 | 1996-09-11 21:57:40 +0000 | [diff] [blame] | 197 | parse tree for \code{del f(0)}: this statement is considered legal |
Guido van Rossum | 4747887 | 1996-08-21 14:32:37 +0000 | [diff] [blame] | 198 | within the formal grammar for Python but is not a legal language |
| 199 | construct. The \code{SyntaxError} raised for this condition is |
| 200 | actually generated by the Python byte-compiler normally, which is why |
| 201 | it can be raised at this point by the \code{parser} module. Most |
| 202 | causes of compilation failure can be diagnosed programmatically by |
| 203 | inspection of the parse tree. |
Guido van Rossum | 4b73a06 | 1995-10-11 17:30:04 +0000 | [diff] [blame] | 204 | \end{funcdesc} |
| 205 | |
| 206 | |
Fred Drake | 4b7d5a4 | 1996-09-11 21:57:40 +0000 | [diff] [blame] | 207 | \subsection{Queries on AST Objects} |
Guido van Rossum | 4b73a06 | 1995-10-11 17:30:04 +0000 | [diff] [blame] | 208 | |
Fred Drake | 4b7d5a4 | 1996-09-11 21:57:40 +0000 | [diff] [blame] | 209 | Two functions are provided which allow an application to determine if |
| 210 | an AST was create as an expression or a suite. Neither of these |
| 211 | functions can be used to determine if an AST was created from source |
| 212 | code via \code{expr()} or \code{suite()} or from a parse tree via |
| 213 | \code{sequence2ast()}. |
Guido van Rossum | 4b73a06 | 1995-10-11 17:30:04 +0000 | [diff] [blame] | 214 | |
| 215 | \begin{funcdesc}{isexpr}{ast} |
| 216 | When \code{\var{ast}} represents an \code{'eval'} form, this function |
| 217 | returns a true value (\code{1}), otherwise it returns false |
| 218 | (\code{0}). This is useful, since code objects normally cannot be |
| 219 | queried for this information using existing built-in functions. Note |
| 220 | that the code objects created by \code{compileast()} cannot be queried |
| 221 | like this either, and are identical to those created by the built-in |
| 222 | \code{compile()} function. |
| 223 | \end{funcdesc} |
| 224 | |
| 225 | |
| 226 | \begin{funcdesc}{issuite}{ast} |
| 227 | This function mirrors \code{isexpr()} in that it reports whether an |
Fred Drake | 4b7d5a4 | 1996-09-11 21:57:40 +0000 | [diff] [blame] | 228 | AST object represents an \code{'exec'} form, commonly known as a |
| 229 | ``suite.'' It is not safe to assume that this function is equivelent |
| 230 | to \code{not isexpr(\var{ast})}, as additional syntactic fragments may |
| 231 | be supported in the future. |
Guido van Rossum | 4b73a06 | 1995-10-11 17:30:04 +0000 | [diff] [blame] | 232 | \end{funcdesc} |
| 233 | |
| 234 | |
Guido van Rossum | 4b73a06 | 1995-10-11 17:30:04 +0000 | [diff] [blame] | 235 | \subsection{Exceptions and Error Handling} |
| 236 | |
| 237 | The parser module defines a single exception, but may also pass other |
| 238 | built-in exceptions from other portions of the Python runtime |
| 239 | environment. See each function for information about the exceptions |
| 240 | it can raise. |
| 241 | |
| 242 | \begin{excdesc}{ParserError} |
| 243 | Exception raised when a failure occurs within the parser module. This |
| 244 | is generally produced for validation failures rather than the built in |
| 245 | \code{SyntaxError} thrown during normal parsing. |
| 246 | The exception argument is either a string describing the reason of the |
Guido van Rossum | 4747887 | 1996-08-21 14:32:37 +0000 | [diff] [blame] | 247 | failure or a tuple containing a sequence causing the failure from a parse |
| 248 | tree passed to \code{sequence2ast()} and an explanatory string. Calls to |
| 249 | \code{sequence2ast()} need to be able to handle either type of exception, |
Guido van Rossum | 4b73a06 | 1995-10-11 17:30:04 +0000 | [diff] [blame] | 250 | while calls to other functions in the module will only need to be |
| 251 | aware of the simple string values. |
| 252 | \end{excdesc} |
| 253 | |
| 254 | Note that the functions \code{compileast()}, \code{expr()}, and |
| 255 | \code{suite()} may throw exceptions which are normally thrown by the |
| 256 | parsing and compilation process. These include the built in |
| 257 | exceptions \code{MemoryError}, \code{OverflowError}, |
| 258 | \code{SyntaxError}, and \code{SystemError}. In these cases, these |
| 259 | exceptions carry all the meaning normally associated with them. Refer |
| 260 | to the descriptions of each function for detailed information. |
| 261 | |
Guido van Rossum | 4b73a06 | 1995-10-11 17:30:04 +0000 | [diff] [blame] | 262 | |
Guido van Rossum | 4747887 | 1996-08-21 14:32:37 +0000 | [diff] [blame] | 263 | \subsection{AST Objects} |
| 264 | |
Fred Drake | 4b7d5a4 | 1996-09-11 21:57:40 +0000 | [diff] [blame] | 265 | AST objects returned by \code{expr()}, \code{suite()}, and |
| 266 | \code{sequence2ast()} have no methods of their own. |
Guido van Rossum | 4747887 | 1996-08-21 14:32:37 +0000 | [diff] [blame] | 267 | Some of the functions defined which accept an AST object as their |
Fred Drake | 4b7d5a4 | 1996-09-11 21:57:40 +0000 | [diff] [blame] | 268 | first argument may change to object methods in the future. The type |
| 269 | of these objects is available as \code{ASTType} in the module. |
Guido van Rossum | 4747887 | 1996-08-21 14:32:37 +0000 | [diff] [blame] | 270 | |
| 271 | Ordered and equality comparisons are supported between AST objects. |
| 272 | |
| 273 | |
Guido van Rossum | 8206fb9 | 1996-08-26 00:33:29 +0000 | [diff] [blame] | 274 | \subsection{Examples} |
Fred Drake | 4b3f031 | 1996-12-13 22:04:31 +0000 | [diff] [blame] | 275 | \nodename{AST Examples} |
Guido van Rossum | 4b73a06 | 1995-10-11 17:30:04 +0000 | [diff] [blame] | 276 | |
Guido van Rossum | 4747887 | 1996-08-21 14:32:37 +0000 | [diff] [blame] | 277 | The parser modules allows operations to be performed on the parse tree |
| 278 | of Python source code before the bytecode is generated, and provides |
Fred Drake | 4b7d5a4 | 1996-09-11 21:57:40 +0000 | [diff] [blame] | 279 | for inspection of the parse tree for information gathering purposes. |
| 280 | Two examples are presented. The simple example demonstrates emulation |
| 281 | of the \code{compile()} built-in function and the complex example |
| 282 | shows the use of a parse tree for information discovery. |
Guido van Rossum | 8206fb9 | 1996-08-26 00:33:29 +0000 | [diff] [blame] | 283 | |
Fred Drake | 4b7d5a4 | 1996-09-11 21:57:40 +0000 | [diff] [blame] | 284 | \subsubsection{Emulation of \sectcode{compile()}} |
Guido van Rossum | 8206fb9 | 1996-08-26 00:33:29 +0000 | [diff] [blame] | 285 | |
| 286 | While many useful operations may take place between parsing and |
Guido van Rossum | 4747887 | 1996-08-21 14:32:37 +0000 | [diff] [blame] | 287 | bytecode generation, the simplest operation is to do nothing. For |
| 288 | this purpose, using the \code{parser} module to produce an |
| 289 | intermediate data structure is equivelent to the code |
| 290 | |
Guido van Rossum | e47da0a | 1997-07-17 16:34:52 +0000 | [diff] [blame] | 291 | \bcode\begin{verbatim} |
Guido van Rossum | 4747887 | 1996-08-21 14:32:37 +0000 | [diff] [blame] | 292 | >>> code = compile('a + 5', 'eval') |
| 293 | >>> a = 5 |
| 294 | >>> eval(code) |
| 295 | 10 |
Guido van Rossum | e47da0a | 1997-07-17 16:34:52 +0000 | [diff] [blame] | 296 | \end{verbatim}\ecode |
| 297 | % |
Guido van Rossum | 4747887 | 1996-08-21 14:32:37 +0000 | [diff] [blame] | 298 | The equivelent operation using the \code{parser} module is somewhat |
| 299 | longer, and allows the intermediate internal parse tree to be retained |
| 300 | as an AST object: |
Guido van Rossum | 4b73a06 | 1995-10-11 17:30:04 +0000 | [diff] [blame] | 301 | |
Guido van Rossum | e47da0a | 1997-07-17 16:34:52 +0000 | [diff] [blame] | 302 | \bcode\begin{verbatim} |
Guido van Rossum | 4b73a06 | 1995-10-11 17:30:04 +0000 | [diff] [blame] | 303 | >>> import parser |
| 304 | >>> ast = parser.expr('a + 5') |
| 305 | >>> code = parser.compileast(ast) |
| 306 | >>> a = 5 |
| 307 | >>> eval(code) |
| 308 | 10 |
Guido van Rossum | e47da0a | 1997-07-17 16:34:52 +0000 | [diff] [blame] | 309 | \end{verbatim}\ecode |
| 310 | % |
Guido van Rossum | 8206fb9 | 1996-08-26 00:33:29 +0000 | [diff] [blame] | 311 | An application which needs both AST and code objects can package this |
| 312 | code into readily available functions: |
| 313 | |
Guido van Rossum | e47da0a | 1997-07-17 16:34:52 +0000 | [diff] [blame] | 314 | \bcode\begin{verbatim} |
Guido van Rossum | 8206fb9 | 1996-08-26 00:33:29 +0000 | [diff] [blame] | 315 | import parser |
| 316 | |
| 317 | def load_suite(source_string): |
| 318 | ast = parser.suite(source_string) |
| 319 | code = parser.compileast(ast) |
| 320 | return ast, code |
| 321 | |
| 322 | def load_expression(source_string): |
| 323 | ast = parser.expr(source_string) |
| 324 | code = parser.compileast(ast) |
| 325 | return ast, code |
Guido van Rossum | e47da0a | 1997-07-17 16:34:52 +0000 | [diff] [blame] | 326 | \end{verbatim}\ecode |
| 327 | % |
Guido van Rossum | 8206fb9 | 1996-08-26 00:33:29 +0000 | [diff] [blame] | 328 | \subsubsection{Information Discovery} |
| 329 | |
Fred Drake | 4b7d5a4 | 1996-09-11 21:57:40 +0000 | [diff] [blame] | 330 | Some applications benefit from direct access to the parse tree. The |
| 331 | remainder of this section demonstrates how the parse tree provides |
| 332 | access to module documentation defined in docstrings without requiring |
| 333 | that the code being examined be loaded into a running interpreter via |
| 334 | \code{import}. This can be very useful for performing analyses of |
| 335 | untrusted code. |
Guido van Rossum | 4b73a06 | 1995-10-11 17:30:04 +0000 | [diff] [blame] | 336 | |
Guido van Rossum | 4747887 | 1996-08-21 14:32:37 +0000 | [diff] [blame] | 337 | Generally, the example will demonstrate how the parse tree may be |
| 338 | traversed to distill interesting information. Two functions and a set |
Fred Drake | 4b7d5a4 | 1996-09-11 21:57:40 +0000 | [diff] [blame] | 339 | of classes are developed which provide programmatic access to high |
Guido van Rossum | 4747887 | 1996-08-21 14:32:37 +0000 | [diff] [blame] | 340 | level function and class definitions provided by a module. The |
| 341 | classes extract information from the parse tree and provide access to |
| 342 | the information at a useful semantic level, one function provides a |
| 343 | simple low-level pattern matching capability, and the other function |
| 344 | defines a high-level interface to the classes by handling file |
| 345 | operations on behalf of the caller. All source files mentioned here |
| 346 | which are not part of the Python installation are located in the |
Fred Drake | 4b7d5a4 | 1996-09-11 21:57:40 +0000 | [diff] [blame] | 347 | \file{Demo/parser/} directory of the distribution. |
Guido van Rossum | 4b73a06 | 1995-10-11 17:30:04 +0000 | [diff] [blame] | 348 | |
Guido van Rossum | 8206fb9 | 1996-08-26 00:33:29 +0000 | [diff] [blame] | 349 | The dynamic nature of Python allows the programmer a great deal of |
| 350 | flexibility, but most modules need only a limited measure of this when |
| 351 | defining classes, functions, and methods. In this example, the only |
| 352 | definitions that will be considered are those which are defined in the |
| 353 | top level of their context, e.g., a function defined by a \code{def} |
| 354 | statement at column zero of a module, but not a function defined |
Fred Drake | 4b7d5a4 | 1996-09-11 21:57:40 +0000 | [diff] [blame] | 355 | within a branch of an \code{if} ... \code{else} construct, though |
Guido van Rossum | 8206fb9 | 1996-08-26 00:33:29 +0000 | [diff] [blame] | 356 | there are some good reasons for doing so in some situations. Nesting |
| 357 | of definitions will be handled by the code developed in the example. |
| 358 | |
Guido van Rossum | 4747887 | 1996-08-21 14:32:37 +0000 | [diff] [blame] | 359 | To construct the upper-level extraction methods, we need to know what |
| 360 | the parse tree structure looks like and how much of it we actually |
Fred Drake | 4b7d5a4 | 1996-09-11 21:57:40 +0000 | [diff] [blame] | 361 | need to be concerned about. Python uses a moderately deep parse tree |
Guido van Rossum | 4747887 | 1996-08-21 14:32:37 +0000 | [diff] [blame] | 362 | so there are a large number of intermediate nodes. It is important to |
| 363 | read and understand the formal grammar used by Python. This is |
| 364 | specified in the file \file{Grammar/Grammar} in the distribution. |
| 365 | Consider the simplest case of interest when searching for docstrings: |
Guido van Rossum | 8206fb9 | 1996-08-26 00:33:29 +0000 | [diff] [blame] | 366 | a module consisting of a docstring and nothing else. (See file |
| 367 | \file{docstring.py}.) |
Guido van Rossum | 4b73a06 | 1995-10-11 17:30:04 +0000 | [diff] [blame] | 368 | |
Guido van Rossum | e47da0a | 1997-07-17 16:34:52 +0000 | [diff] [blame] | 369 | \bcode\begin{verbatim} |
Guido van Rossum | 4747887 | 1996-08-21 14:32:37 +0000 | [diff] [blame] | 370 | """Some documentation. |
| 371 | """ |
Guido van Rossum | e47da0a | 1997-07-17 16:34:52 +0000 | [diff] [blame] | 372 | \end{verbatim}\ecode |
| 373 | % |
Guido van Rossum | 4747887 | 1996-08-21 14:32:37 +0000 | [diff] [blame] | 374 | Using the interpreter to take a look at the parse tree, we find a |
| 375 | bewildering mass of numbers and parentheses, with the documentation |
Fred Drake | 4b7d5a4 | 1996-09-11 21:57:40 +0000 | [diff] [blame] | 376 | buried deep in nested tuples. |
Guido van Rossum | 4b73a06 | 1995-10-11 17:30:04 +0000 | [diff] [blame] | 377 | |
Guido van Rossum | e47da0a | 1997-07-17 16:34:52 +0000 | [diff] [blame] | 378 | \bcode\begin{verbatim} |
Guido van Rossum | 4747887 | 1996-08-21 14:32:37 +0000 | [diff] [blame] | 379 | >>> import parser |
| 380 | >>> import pprint |
| 381 | >>> ast = parser.suite(open('docstring.py').read()) |
| 382 | >>> tup = parser.ast2tuple(ast) |
| 383 | >>> pprint.pprint(tup) |
| 384 | (257, |
| 385 | (264, |
| 386 | (265, |
| 387 | (266, |
| 388 | (267, |
| 389 | (307, |
| 390 | (287, |
| 391 | (288, |
| 392 | (289, |
| 393 | (290, |
| 394 | (292, |
| 395 | (293, |
| 396 | (294, |
| 397 | (295, |
| 398 | (296, |
| 399 | (297, |
| 400 | (298, |
| 401 | (299, |
| 402 | (300, (3, '"""Some documentation.\012"""'))))))))))))))))), |
| 403 | (4, ''))), |
| 404 | (4, ''), |
| 405 | (0, '')) |
Guido van Rossum | e47da0a | 1997-07-17 16:34:52 +0000 | [diff] [blame] | 406 | \end{verbatim}\ecode |
| 407 | % |
Guido van Rossum | 4747887 | 1996-08-21 14:32:37 +0000 | [diff] [blame] | 408 | The numbers at the first element of each node in the tree are the node |
| 409 | types; they map directly to terminal and non-terminal symbols in the |
| 410 | grammar. Unfortunately, they are represented as integers in the |
| 411 | internal representation, and the Python structures generated do not |
| 412 | change that. However, the \code{symbol} and \code{token} modules |
| 413 | provide symbolic names for the node types and dictionaries which map |
| 414 | from the integers to the symbolic names for the node types. |
| 415 | |
| 416 | In the output presented above, the outermost tuple contains four |
| 417 | elements: the integer \code{257} and three additional tuples. Node |
| 418 | type \code{257} has the symbolic name \code{file_input}. Each of |
| 419 | these inner tuples contains an integer as the first element; these |
| 420 | integers, \code{264}, \code{4}, and \code{0}, represent the node types |
| 421 | \code{stmt}, \code{NEWLINE}, and \code{ENDMARKER}, respectively. |
| 422 | Note that these values may change depending on the version of Python |
| 423 | you are using; consult \file{symbol.py} and \file{token.py} for |
| 424 | details of the mapping. It should be fairly clear that the outermost |
| 425 | node is related primarily to the input source rather than the contents |
| 426 | of the file, and may be disregarded for the moment. The \code{stmt} |
| 427 | node is much more interesting. In particular, all docstrings are |
| 428 | found in subtrees which are formed exactly as this node is formed, |
| 429 | with the only difference being the string itself. The association |
| 430 | between the docstring in a similar tree and the defined entity (class, |
| 431 | function, or module) which it describes is given by the position of |
| 432 | the docstring subtree within the tree defining the described |
| 433 | structure. |
| 434 | |
| 435 | By replacing the actual docstring with something to signify a variable |
Fred Drake | 4b7d5a4 | 1996-09-11 21:57:40 +0000 | [diff] [blame] | 436 | component of the tree, we allow a simple pattern matching approach to |
| 437 | check any given subtree for equivelence to the general pattern for |
| 438 | docstrings. Since the example demonstrates information extraction, we |
| 439 | can safely require that the tree be in tuple form rather than list |
| 440 | form, allowing a simple variable representation to be |
| 441 | \code{['variable_name']}. A simple recursive function can implement |
Guido van Rossum | 4747887 | 1996-08-21 14:32:37 +0000 | [diff] [blame] | 442 | the pattern matching, returning a boolean and a dictionary of variable |
Guido van Rossum | 8206fb9 | 1996-08-26 00:33:29 +0000 | [diff] [blame] | 443 | name to value mappings. (See file \file{example.py}.) |
Guido van Rossum | 4747887 | 1996-08-21 14:32:37 +0000 | [diff] [blame] | 444 | |
Guido van Rossum | e47da0a | 1997-07-17 16:34:52 +0000 | [diff] [blame] | 445 | \bcode\begin{verbatim} |
Guido van Rossum | 4747887 | 1996-08-21 14:32:37 +0000 | [diff] [blame] | 446 | from types import ListType, TupleType |
| 447 | |
| 448 | def match(pattern, data, vars=None): |
| 449 | if vars is None: |
| 450 | vars = {} |
| 451 | if type(pattern) is ListType: |
| 452 | vars[pattern[0]] = data |
| 453 | return 1, vars |
| 454 | if type(pattern) is not TupleType: |
| 455 | return (pattern == data), vars |
| 456 | if len(data) != len(pattern): |
| 457 | return 0, vars |
| 458 | for pattern, data in map(None, pattern, data): |
| 459 | same, vars = match(pattern, data, vars) |
| 460 | if not same: |
| 461 | break |
| 462 | return same, vars |
Guido van Rossum | e47da0a | 1997-07-17 16:34:52 +0000 | [diff] [blame] | 463 | \end{verbatim}\ecode |
| 464 | % |
Fred Drake | 4b7d5a4 | 1996-09-11 21:57:40 +0000 | [diff] [blame] | 465 | Using this simple representation for syntactic variables and the symbolic |
Guido van Rossum | 8206fb9 | 1996-08-26 00:33:29 +0000 | [diff] [blame] | 466 | node types, the pattern for the candidate docstring subtrees becomes |
| 467 | fairly readable. (See file \file{example.py}.) |
Guido van Rossum | 4747887 | 1996-08-21 14:32:37 +0000 | [diff] [blame] | 468 | |
Guido van Rossum | e47da0a | 1997-07-17 16:34:52 +0000 | [diff] [blame] | 469 | \bcode\begin{verbatim} |
Guido van Rossum | 8206fb9 | 1996-08-26 00:33:29 +0000 | [diff] [blame] | 470 | import symbol |
| 471 | import token |
| 472 | |
| 473 | DOCSTRING_STMT_PATTERN = ( |
| 474 | symbol.stmt, |
| 475 | (symbol.simple_stmt, |
| 476 | (symbol.small_stmt, |
| 477 | (symbol.expr_stmt, |
| 478 | (symbol.testlist, |
| 479 | (symbol.test, |
| 480 | (symbol.and_test, |
| 481 | (symbol.not_test, |
| 482 | (symbol.comparison, |
| 483 | (symbol.expr, |
| 484 | (symbol.xor_expr, |
| 485 | (symbol.and_expr, |
| 486 | (symbol.shift_expr, |
| 487 | (symbol.arith_expr, |
| 488 | (symbol.term, |
| 489 | (symbol.factor, |
| 490 | (symbol.power, |
| 491 | (symbol.atom, |
| 492 | (token.STRING, ['docstring']) |
| 493 | )))))))))))))))), |
| 494 | (token.NEWLINE, '') |
| 495 | )) |
Guido van Rossum | e47da0a | 1997-07-17 16:34:52 +0000 | [diff] [blame] | 496 | \end{verbatim}\ecode |
| 497 | % |
Guido van Rossum | 4747887 | 1996-08-21 14:32:37 +0000 | [diff] [blame] | 498 | Using the \code{match()} function with this pattern, extracting the |
| 499 | module docstring from the parse tree created previously is easy: |
| 500 | |
Guido van Rossum | e47da0a | 1997-07-17 16:34:52 +0000 | [diff] [blame] | 501 | \bcode\begin{verbatim} |
Guido van Rossum | 4747887 | 1996-08-21 14:32:37 +0000 | [diff] [blame] | 502 | >>> found, vars = match(DOCSTRING_STMT_PATTERN, tup[1]) |
| 503 | >>> found |
| 504 | 1 |
| 505 | >>> vars |
| 506 | {'docstring': '"""Some documentation.\012"""'} |
Guido van Rossum | e47da0a | 1997-07-17 16:34:52 +0000 | [diff] [blame] | 507 | \end{verbatim}\ecode |
| 508 | % |
Guido van Rossum | 4747887 | 1996-08-21 14:32:37 +0000 | [diff] [blame] | 509 | Once specific data can be extracted from a location where it is |
| 510 | expected, the question of where information can be expected |
| 511 | needs to be answered. When dealing with docstrings, the answer is |
| 512 | fairly simple: the docstring is the first \code{stmt} node in a code |
| 513 | block (\code{file_input} or \code{suite} node types). A module |
| 514 | consists of a single \code{file_input} node, and class and function |
| 515 | definitions each contain exactly one \code{suite} node. Classes and |
| 516 | functions are readily identified as subtrees of code block nodes which |
| 517 | start with \code{(stmt, (compound_stmt, (classdef, ...} or |
| 518 | \code{(stmt, (compound_stmt, (funcdef, ...}. Note that these subtrees |
| 519 | cannot be matched by \code{match()} since it does not support multiple |
| 520 | sibling nodes to match without regard to number. A more elaborate |
| 521 | matching function could be used to overcome this limitation, but this |
| 522 | is sufficient for the example. |
| 523 | |
Guido van Rossum | 8206fb9 | 1996-08-26 00:33:29 +0000 | [diff] [blame] | 524 | Given the ability to determine whether a statement might be a |
| 525 | docstring and extract the actual string from the statement, some work |
| 526 | needs to be performed to walk the parse tree for an entire module and |
| 527 | extract information about the names defined in each context of the |
| 528 | module and associate any docstrings with the names. The code to |
| 529 | perform this work is not complicated, but bears some explanation. |
| 530 | |
| 531 | The public interface to the classes is straightforward and should |
| 532 | probably be somewhat more flexible. Each ``major'' block of the |
| 533 | module is described by an object providing several methods for inquiry |
| 534 | and a constructor which accepts at least the subtree of the complete |
| 535 | parse tree which it represents. The \code{ModuleInfo} constructor |
| 536 | accepts an optional \code{\var{name}} parameter since it cannot |
| 537 | otherwise determine the name of the module. |
| 538 | |
| 539 | The public classes include \code{ClassInfo}, \code{FunctionInfo}, |
| 540 | and \code{ModuleInfo}. All objects provide the |
| 541 | methods \code{get_name()}, \code{get_docstring()}, |
| 542 | \code{get_class_names()}, and \code{get_class_info()}. The |
| 543 | \code{ClassInfo} objects support \code{get_method_names()} and |
| 544 | \code{get_method_info()} while the other classes provide |
| 545 | \code{get_function_names()} and \code{get_function_info()}. |
| 546 | |
| 547 | Within each of the forms of code block that the public classes |
| 548 | represent, most of the required information is in the same form and is |
Fred Drake | 4b7d5a4 | 1996-09-11 21:57:40 +0000 | [diff] [blame] | 549 | accessed in the same way, with classes having the distinction that |
Guido van Rossum | 8206fb9 | 1996-08-26 00:33:29 +0000 | [diff] [blame] | 550 | functions defined at the top level are referred to as ``methods.'' |
| 551 | Since the difference in nomenclature reflects a real semantic |
Fred Drake | 4b7d5a4 | 1996-09-11 21:57:40 +0000 | [diff] [blame] | 552 | distinction from functions defined outside of a class, the |
| 553 | implementation needs to maintain the distinction. |
Guido van Rossum | 8206fb9 | 1996-08-26 00:33:29 +0000 | [diff] [blame] | 554 | Hence, most of the functionality of the public classes can be |
| 555 | implemented in a common base class, \code{SuiteInfoBase}, with the |
| 556 | accessors for function and method information provided elsewhere. |
| 557 | Note that there is only one class which represents function and method |
Fred Drake | 43d287a | 1997-01-22 14:25:21 +0000 | [diff] [blame] | 558 | information; this parallels the use of the \code{def} statement to |
Fred Drake | 4b7d5a4 | 1996-09-11 21:57:40 +0000 | [diff] [blame] | 559 | define both types of elements. |
Guido van Rossum | 8206fb9 | 1996-08-26 00:33:29 +0000 | [diff] [blame] | 560 | |
| 561 | Most of the accessor functions are declared in \code{SuiteInfoBase} |
| 562 | and do not need to be overriden by subclasses. More importantly, the |
| 563 | extraction of most information from a parse tree is handled through a |
| 564 | method called by the \code{SuiteInfoBase} constructor. The example |
| 565 | code for most of the classes is clear when read alongside the formal |
| 566 | grammar, but the method which recursively creates new information |
| 567 | objects requires further examination. Here is the relevant part of |
| 568 | the \code{SuiteInfoBase} definition from \file{example.py}: |
| 569 | |
Guido van Rossum | e47da0a | 1997-07-17 16:34:52 +0000 | [diff] [blame] | 570 | \bcode\begin{verbatim} |
Guido van Rossum | 8206fb9 | 1996-08-26 00:33:29 +0000 | [diff] [blame] | 571 | class SuiteInfoBase: |
| 572 | _docstring = '' |
| 573 | _name = '' |
| 574 | |
| 575 | def __init__(self, tree = None): |
| 576 | self._class_info = {} |
| 577 | self._function_info = {} |
| 578 | if tree: |
| 579 | self._extract_info(tree) |
| 580 | |
| 581 | def _extract_info(self, tree): |
| 582 | # extract docstring |
| 583 | if len(tree) == 2: |
| 584 | found, vars = match(DOCSTRING_STMT_PATTERN[1], tree[1]) |
| 585 | else: |
| 586 | found, vars = match(DOCSTRING_STMT_PATTERN, tree[3]) |
| 587 | if found: |
| 588 | self._docstring = eval(vars['docstring']) |
| 589 | # discover inner definitions |
| 590 | for node in tree[1:]: |
| 591 | found, vars = match(COMPOUND_STMT_PATTERN, node) |
| 592 | if found: |
| 593 | cstmt = vars['compound'] |
| 594 | if cstmt[0] == symbol.funcdef: |
| 595 | name = cstmt[2][1] |
| 596 | self._function_info[name] = FunctionInfo(cstmt) |
| 597 | elif cstmt[0] == symbol.classdef: |
| 598 | name = cstmt[2][1] |
| 599 | self._class_info[name] = ClassInfo(cstmt) |
Guido van Rossum | e47da0a | 1997-07-17 16:34:52 +0000 | [diff] [blame] | 600 | \end{verbatim}\ecode |
| 601 | % |
Guido van Rossum | 8206fb9 | 1996-08-26 00:33:29 +0000 | [diff] [blame] | 602 | After initializing some internal state, the constructor calls the |
| 603 | \code{_extract_info()} method. This method performs the bulk of the |
| 604 | information extraction which takes place in the entire example. The |
| 605 | extraction has two distinct phases: the location of the docstring for |
| 606 | the parse tree passed in, and the discovery of additional definitions |
| 607 | within the code block represented by the parse tree. |
| 608 | |
| 609 | The initial \code{if} test determines whether the nested suite is of |
| 610 | the ``short form'' or the ``long form.'' The short form is used when |
| 611 | the code block is on the same line as the definition of the code |
| 612 | block, as in |
| 613 | |
Guido van Rossum | e47da0a | 1997-07-17 16:34:52 +0000 | [diff] [blame] | 614 | \bcode\begin{verbatim} |
Guido van Rossum | 8206fb9 | 1996-08-26 00:33:29 +0000 | [diff] [blame] | 615 | def square(x): "Square an argument."; return x ** 2 |
Guido van Rossum | e47da0a | 1997-07-17 16:34:52 +0000 | [diff] [blame] | 616 | \end{verbatim}\ecode |
| 617 | % |
Guido van Rossum | 8206fb9 | 1996-08-26 00:33:29 +0000 | [diff] [blame] | 618 | while the long form uses an indented block and allows nested |
| 619 | definitions: |
| 620 | |
Guido van Rossum | e47da0a | 1997-07-17 16:34:52 +0000 | [diff] [blame] | 621 | \bcode\begin{verbatim} |
Guido van Rossum | 8206fb9 | 1996-08-26 00:33:29 +0000 | [diff] [blame] | 622 | def make_power(exp): |
| 623 | "Make a function that raises an argument to the exponent `exp'." |
| 624 | def raiser(x, y=exp): |
| 625 | return x ** y |
| 626 | return raiser |
Guido van Rossum | e47da0a | 1997-07-17 16:34:52 +0000 | [diff] [blame] | 627 | \end{verbatim}\ecode |
| 628 | % |
Guido van Rossum | 8206fb9 | 1996-08-26 00:33:29 +0000 | [diff] [blame] | 629 | When the short form is used, the code block may contain a docstring as |
| 630 | the first, and possibly only, \code{small_stmt} element. The |
| 631 | extraction of such a docstring is slightly different and requires only |
| 632 | a portion of the complete pattern used in the more common case. As |
Fred Drake | 4b7d5a4 | 1996-09-11 21:57:40 +0000 | [diff] [blame] | 633 | implemented, the docstring will only be found if there is only |
Guido van Rossum | 8206fb9 | 1996-08-26 00:33:29 +0000 | [diff] [blame] | 634 | one \code{small_stmt} node in the \code{simple_stmt} node. Since most |
Fred Drake | 4b7d5a4 | 1996-09-11 21:57:40 +0000 | [diff] [blame] | 635 | functions and methods which use the short form do not provide a |
Guido van Rossum | 8206fb9 | 1996-08-26 00:33:29 +0000 | [diff] [blame] | 636 | docstring, this may be considered sufficient. The extraction of the |
| 637 | docstring proceeds using the \code{match()} function as described |
| 638 | above, and the value of the docstring is stored as an attribute of the |
| 639 | \code{SuiteInfoBase} object. |
| 640 | |
Fred Drake | 4b7d5a4 | 1996-09-11 21:57:40 +0000 | [diff] [blame] | 641 | After docstring extraction, a simple definition discovery |
| 642 | algorithm operates on the \code{stmt} nodes of the \code{suite} node. The |
Guido van Rossum | 8206fb9 | 1996-08-26 00:33:29 +0000 | [diff] [blame] | 643 | special case of the short form is not tested; since there are no |
| 644 | \code{stmt} nodes in the short form, the algorithm will silently skip |
| 645 | the single \code{simple_stmt} node and correctly not discover any |
| 646 | nested definitions. |
| 647 | |
Fred Drake | 4b7d5a4 | 1996-09-11 21:57:40 +0000 | [diff] [blame] | 648 | Each statement in the code block is categorized as |
| 649 | a class definition, function or method definition, or |
Guido van Rossum | 8206fb9 | 1996-08-26 00:33:29 +0000 | [diff] [blame] | 650 | something else. For the definition statements, the name of the |
Fred Drake | 4b7d5a4 | 1996-09-11 21:57:40 +0000 | [diff] [blame] | 651 | element defined is extracted and a representation object |
Guido van Rossum | 8206fb9 | 1996-08-26 00:33:29 +0000 | [diff] [blame] | 652 | appropriate to the definition is created with the defining subtree |
| 653 | passed as an argument to the constructor. The repesentation objects |
| 654 | are stored in instance variables and may be retrieved by name using |
| 655 | the appropriate accessor methods. |
| 656 | |
| 657 | The public classes provide any accessors required which are more |
| 658 | specific than those provided by the \code{SuiteInfoBase} class, but |
| 659 | the real extraction algorithm remains common to all forms of code |
| 660 | blocks. A high-level function can be used to extract the complete set |
Fred Drake | 4b7d5a4 | 1996-09-11 21:57:40 +0000 | [diff] [blame] | 661 | of information from a source file. (See file \file{example.py}.) |
Guido van Rossum | 8206fb9 | 1996-08-26 00:33:29 +0000 | [diff] [blame] | 662 | |
Guido van Rossum | e47da0a | 1997-07-17 16:34:52 +0000 | [diff] [blame] | 663 | \bcode\begin{verbatim} |
Guido van Rossum | 8206fb9 | 1996-08-26 00:33:29 +0000 | [diff] [blame] | 664 | def get_docs(fileName): |
| 665 | source = open(fileName).read() |
| 666 | import os |
| 667 | basename = os.path.basename(os.path.splitext(fileName)[0]) |
| 668 | import parser |
| 669 | ast = parser.suite(source) |
| 670 | tup = parser.ast2tuple(ast) |
| 671 | return ModuleInfo(tup, basename) |
Guido van Rossum | e47da0a | 1997-07-17 16:34:52 +0000 | [diff] [blame] | 672 | \end{verbatim}\ecode |
| 673 | % |
Guido van Rossum | 8206fb9 | 1996-08-26 00:33:29 +0000 | [diff] [blame] | 674 | This provides an easy-to-use interface to the documentation of a |
| 675 | module. If information is required which is not extracted by the code |
| 676 | of this example, the code may be extended at clearly defined points to |
| 677 | provide additional capabilities. |
Guido van Rossum | 4747887 | 1996-08-21 14:32:37 +0000 | [diff] [blame] | 678 | |
| 679 | |
Fred Drake | e061a51 | 1997-10-06 21:40:20 +0000 | [diff] [blame] | 680 | \section{Standard Module \sectcode{symbol}} |
| 681 | \stmodindex{symbol} |
| 682 | |
| 683 | This module provides constants which represent the numeric values of |
| 684 | internal nodes of the parse tree. Unlike most Python constants, these |
| 685 | use lower-case names. Refer to the file \file{Grammar/Grammar} in the |
| 686 | Python distribution for the defintions of the names in the context of |
| 687 | the language grammar. The specific numeric values which the names map |
| 688 | to may change between Python versions. |
| 689 | |
| 690 | This module also provides one additional data object: |
| 691 | |
| 692 | \begin{datadesc}{sym_name} |
| 693 | Dictionary mapping the numeric values of the constants defined in this |
| 694 | module back to name strings, allowing more human-readable |
| 695 | representation of parse trees to be generated. |
| 696 | \end{datadesc} |
| 697 | |
| 698 | |
| 699 | \section{Standard Module \sectcode{token}} |
| 700 | \stmodindex{token} |
| 701 | |
| 702 | This module provides constants which represent the numeric values of |
| 703 | leaf nodes of the parse tree (terminal tokens). Refer to the file |
| 704 | \file{Grammar/Grammar} in the Python distribution for the defintions |
| 705 | of the names in the context of the language grammar. The specific |
| 706 | numeric values which the names map to may change between Python |
| 707 | versions. |
| 708 | |
| 709 | This module also provides one data object and some functions. The |
| 710 | functions mirror definitions in the Python C header files. |
| 711 | |
| 712 | \begin{datadesc}{tok_name} |
| 713 | Dictionary mapping the numeric values of the constants defined in this |
| 714 | module back to name strings, allowing more human-readable |
| 715 | representation of parse trees to be generated. |
| 716 | \end{datadesc} |
| 717 | |
| 718 | \begin{funcdesc}{ISTERMINAL}{x} |
| 719 | Return true for terminal token values. |
| 720 | \end{funcdesc} |
| 721 | |
| 722 | \begin{funcdesc}{ISNONTERMINAL}{x} |
| 723 | Return true for non-terminal token values. |
| 724 | \end{funcdesc} |
| 725 | |
| 726 | \begin{funcdesc}{ISEOF}{x} |
| 727 | Return true if \var{x} is the marker indicating the end of input. |
| 728 | \end{funcdesc} |
| 729 | |
Guido van Rossum | 4747887 | 1996-08-21 14:32:37 +0000 | [diff] [blame] | 730 | %% |
| 731 | %% end of file |