Guido van Rossum | 4b73a06 | 1995-10-11 17:30:04 +0000 | [diff] [blame] | 1 | % libparser.tex |
| 2 | % |
| 3 | % Introductory documentation for the new parser built-in module. |
| 4 | % |
| 5 | % Copyright 1995 Virginia Polytechnic Institute and State University |
| 6 | % and Fred L. Drake, Jr. This copyright notice must be distributed on |
| 7 | % all copies, but this document otherwise may be distributed as part |
| 8 | % of the Python distribution. No fee may be charged for this document |
| 9 | % in any representation, either on paper or electronically. This |
| 10 | % restriction does not affect other elements in a distributed package |
| 11 | % in any way. |
| 12 | % |
| 13 | |
| 14 | \section{Built-in Module \sectcode{parser}} |
| 15 | \bimodindex{parser} |
| 16 | |
Guido van Rossum | 4b73a06 | 1995-10-11 17:30:04 +0000 | [diff] [blame] | 17 | The \code{parser} module provides an interface to Python's internal |
| 18 | parser and byte-code compiler. The primary purpose for this interface |
| 19 | is to allow Python code to edit the parse tree of a Python expression |
| 20 | and create executable code from this. This can be better than trying |
| 21 | to parse and modify an arbitrary Python code fragment as a string, and |
| 22 | ensures that parsing is performed in a manner identical to the code |
| 23 | forming the application. It's also faster. |
| 24 | |
| 25 | There are a few things to note about this module which are important |
| 26 | to making use of the data structures created. This is not a tutorial |
| 27 | on editing the parse trees for Python code. |
| 28 | |
| 29 | Most importantly, a good understanding of the Python grammar processed |
| 30 | by the internal parser is required. For full information on the |
| 31 | language syntax, refer to the Language Reference. The parser itself |
| 32 | is created from a grammar specification defined in the file |
| 33 | \code{Grammar/Grammar} in the standard Python distribution. The parse |
| 34 | trees stored in the ``AST objects'' created by this module are the |
| 35 | actual output from the internal parser when created by the |
| 36 | \code{expr()} or \code{suite()} functions, described below. The AST |
Guido van Rossum | 4747887 | 1996-08-21 14:32:37 +0000 | [diff] [blame^] | 37 | objects created by \code{sequence2ast()} faithfully simulate those |
| 38 | structures. Be aware that the values of the sequences which are |
| 39 | considered ``correct'' will vary from one version of Python to another |
| 40 | as the formal grammar for the language is revised. However, |
| 41 | transporting code from one Python version to another as source text |
| 42 | will always allow correct parse trees to be created in the target |
| 43 | version, with the only restriction being that migrating to an older |
| 44 | version of the interpreter will not support more recent language |
| 45 | constructs. The parse trees are not typically compatible from one |
| 46 | version to another, whereas source code has always been |
| 47 | forward-compatible. |
Guido van Rossum | 4b73a06 | 1995-10-11 17:30:04 +0000 | [diff] [blame] | 48 | |
Guido van Rossum | 4747887 | 1996-08-21 14:32:37 +0000 | [diff] [blame^] | 49 | Each element of the sequences returned by \code{ast2list} or |
| 50 | \code{ast2tuple()} has a simple form. Sequences representing |
| 51 | non-terminal elements in the grammar always have a length greater than |
| 52 | one. The first element is an integer which identifies a production in |
| 53 | the grammar. These integers are given symbolic names in the C header |
| 54 | file \code{Include/graminit.h} and the Python module |
| 55 | \code{Lib/symbol.py}. Each additional element of the sequence represents |
| 56 | a component of the production as recognized in the input string: these |
| 57 | are always sequences which have the same form as the parent. An |
| 58 | important aspect of this structure which should be noted is that |
| 59 | keywords used to identify the parent node type, such as the keyword |
| 60 | \code{if} in an \emph{if\_stmt}, are included in the node tree without |
| 61 | any special treatment. For example, the \code{if} keyword is |
Guido van Rossum | 4b73a06 | 1995-10-11 17:30:04 +0000 | [diff] [blame] | 62 | represented by the tuple \code{(1, 'if')}, where \code{1} is the |
| 63 | numeric value associated with all \code{NAME} elements, including |
Guido van Rossum | 4747887 | 1996-08-21 14:32:37 +0000 | [diff] [blame^] | 64 | variable and function names defined by the user. In an alternate form |
| 65 | returned when line number information is requested, the same token |
| 66 | might be represented as \code{(1, 'if', 12)}, where the \code{12} |
| 67 | represents the line number at which the terminal symbol was found. |
Guido van Rossum | 4b73a06 | 1995-10-11 17:30:04 +0000 | [diff] [blame] | 68 | |
| 69 | Terminal elements are represented in much the same way, but without |
| 70 | any child elements and the addition of the source text which was |
| 71 | identified. The example of the \code{if} keyword above is |
| 72 | representative. The various types of terminal symbols are defined in |
| 73 | the C header file \code{Include/token.h} and the Python module |
| 74 | \code{Lib/token.py}. |
| 75 | |
| 76 | The AST objects are not actually required to support the functionality |
| 77 | of this module, but are provided for three purposes: to allow an |
| 78 | application to amortize the cost of processing complex parse trees, to |
| 79 | provide a parse tree representation which conserves memory space when |
Guido van Rossum | 4747887 | 1996-08-21 14:32:37 +0000 | [diff] [blame^] | 80 | compared to the Python list or tuple representation, and to ease the |
| 81 | creation of additional modules in C which manipulate parse trees. A |
| 82 | simple ``wrapper'' module may be created in Python to hide the use of |
| 83 | AST objects. |
Guido van Rossum | 4b73a06 | 1995-10-11 17:30:04 +0000 | [diff] [blame] | 84 | |
| 85 | |
Guido van Rossum | 4b73a06 | 1995-10-11 17:30:04 +0000 | [diff] [blame] | 86 | The \code{parser} module defines the following functions: |
| 87 | |
Guido van Rossum | 4b73a06 | 1995-10-11 17:30:04 +0000 | [diff] [blame] | 88 | \renewcommand{\indexsubitem}{(in module parser)} |
| 89 | |
Guido van Rossum | 4747887 | 1996-08-21 14:32:37 +0000 | [diff] [blame^] | 90 | \begin{funcdesc}{ast2list}{ast\optional{\, line\_info\code{ = 0}}} |
Guido van Rossum | 4b73a06 | 1995-10-11 17:30:04 +0000 | [diff] [blame] | 91 | This function accepts an AST object from the caller in |
Guido van Rossum | 4747887 | 1996-08-21 14:32:37 +0000 | [diff] [blame^] | 92 | \code{\var{ast}} and returns a Python list representing the |
| 93 | equivelent parse tree. The resulting list representation can be used |
| 94 | for inspection or the creation of a new parse tree in list form. |
Guido van Rossum | 4b73a06 | 1995-10-11 17:30:04 +0000 | [diff] [blame] | 95 | This function does not fail so long as memory is available to build |
Guido van Rossum | 4747887 | 1996-08-21 14:32:37 +0000 | [diff] [blame^] | 96 | the list representation. If a parse tree will only be used for |
| 97 | inspection, \code{ast2tuple()} should be used instead to reduce memory |
| 98 | consumption and fragmentation. When modifications are to be made to |
| 99 | the parse tree, this function is significantly faster than retrieving |
| 100 | a tuple representation and converting that to nested lists. |
| 101 | |
| 102 | If the \code{line\_info} flag is given true value, line number |
| 103 | information will be included for all terminal tokens as a third |
| 104 | element of the list representing the token. This information is |
| 105 | omitted if the flag is false or omitted. |
Guido van Rossum | 4b73a06 | 1995-10-11 17:30:04 +0000 | [diff] [blame] | 106 | \end{funcdesc} |
| 107 | |
Guido van Rossum | 4747887 | 1996-08-21 14:32:37 +0000 | [diff] [blame^] | 108 | \begin{funcdesc}{ast2tuple}{ast\optional{\, line\_info\code{ = 0}}} |
| 109 | This function accepts an AST object from the caller in |
| 110 | \code{\var{ast}} and returns a Python tuple representing the |
| 111 | equivelent parse tree. Other than returning a tuple instead of a |
| 112 | list, this function is identical to \code{ast2list()}. |
Guido van Rossum | 4b73a06 | 1995-10-11 17:30:04 +0000 | [diff] [blame] | 113 | |
Guido van Rossum | 4747887 | 1996-08-21 14:32:37 +0000 | [diff] [blame^] | 114 | If the \code{line\_info} flag is given true value, line number |
| 115 | information will be included for all terminal tokens as a third |
| 116 | element of the list representing the token. This information is |
| 117 | omitted if the flag is false or omitted. |
| 118 | \end{funcdesc} |
| 119 | |
| 120 | \begin{funcdesc}{compileast}{ast\optional{\, filename\code{ = '<ast>'}}} |
Guido van Rossum | 4b73a06 | 1995-10-11 17:30:04 +0000 | [diff] [blame] | 121 | The Python byte compiler can be invoked on an AST object to produce |
| 122 | code objects which can be used as part of an \code{exec} statement or |
| 123 | a call to the built-in \code{eval()} function. This function provides |
| 124 | the interface to the compiler, passing the internal parse tree from |
| 125 | \code{\var{ast}} to the parser, using the source file name specified |
| 126 | by the \code{\var{filename}} parameter. The default value supplied |
| 127 | for \code{\var{filename}} indicates that the source was an AST object. |
Guido van Rossum | 4747887 | 1996-08-21 14:32:37 +0000 | [diff] [blame^] | 128 | |
| 129 | Compiling an AST object may result in exceptions related to |
| 130 | compilation; an example would be a \code{SyntaxError} caused by the |
| 131 | parse tree for \code{del f(0)}; this statement is considered legal |
| 132 | within the formal grammar for Python but is not a legal language |
| 133 | construct. The \code{SyntaxError} raised for this condition is |
| 134 | actually generated by the Python byte-compiler normally, which is why |
| 135 | it can be raised at this point by the \code{parser} module. Most |
| 136 | causes of compilation failure can be diagnosed programmatically by |
| 137 | inspection of the parse tree. |
Guido van Rossum | 4b73a06 | 1995-10-11 17:30:04 +0000 | [diff] [blame] | 138 | \end{funcdesc} |
| 139 | |
| 140 | |
| 141 | \begin{funcdesc}{expr}{string} |
| 142 | The \code{expr()} function parses the parameter \code{\var{string}} |
| 143 | as if it were an input to \code{compile(\var{string}, 'eval')}. If |
| 144 | the parse succeeds, an AST object is created to hold the internal |
| 145 | parse tree representation, otherwise an appropriate exception is |
| 146 | thrown. |
| 147 | \end{funcdesc} |
| 148 | |
| 149 | |
| 150 | \begin{funcdesc}{isexpr}{ast} |
| 151 | When \code{\var{ast}} represents an \code{'eval'} form, this function |
| 152 | returns a true value (\code{1}), otherwise it returns false |
| 153 | (\code{0}). This is useful, since code objects normally cannot be |
| 154 | queried for this information using existing built-in functions. Note |
| 155 | that the code objects created by \code{compileast()} cannot be queried |
| 156 | like this either, and are identical to those created by the built-in |
| 157 | \code{compile()} function. |
| 158 | \end{funcdesc} |
| 159 | |
| 160 | |
| 161 | \begin{funcdesc}{issuite}{ast} |
| 162 | This function mirrors \code{isexpr()} in that it reports whether an |
| 163 | AST object represents a suite of statements. It is not safe to assume |
| 164 | that this function is equivelent to \code{not isexpr(\var{ast})}, as |
| 165 | additional syntactic fragments may be supported in the future. |
| 166 | \end{funcdesc} |
| 167 | |
| 168 | |
| 169 | \begin{funcdesc}{suite}{string} |
| 170 | The \code{suite()} function parses the parameter \code{\var{string}} |
| 171 | as if it were an input to \code{compile(\var{string}, 'exec')}. If |
| 172 | the parse succeeds, an AST object is created to hold the internal |
| 173 | parse tree representation, otherwise an appropriate exception is |
| 174 | thrown. |
| 175 | \end{funcdesc} |
| 176 | |
| 177 | |
Guido van Rossum | 4747887 | 1996-08-21 14:32:37 +0000 | [diff] [blame^] | 178 | \begin{funcdesc}{sequence2ast}{sequence} |
| 179 | This function accepts a parse tree represented as a sequence and |
| 180 | builds an internal representation if possible. If it can validate |
| 181 | that the tree conforms to the Python grammar and all nodes are valid |
| 182 | node types in the host version of Python, an AST object is created |
| 183 | from the internal representation and returned to the called. If there |
| 184 | is a problem creating the internal representation, or if the tree |
| 185 | cannot be validated, a \code{ParserError} exception is thrown. An AST |
| 186 | object created this way should not be assumed to compile correctly; |
| 187 | normal exceptions thrown by compilation may still be initiated when |
| 188 | the AST object is passed to \code{compileast()}. This will normally |
| 189 | indicate problems not related to syntax (such as a \code{MemoryError} |
| 190 | exception), but may also be due to constructs such as the result of |
| 191 | parsing \code{del f(0)}, which escapes the Python parser but is |
| 192 | checked by the bytecode compiler. |
| 193 | |
| 194 | Sequences representing terminal tokens may be represented as either |
| 195 | two-element lists of the form \code{(1, 'name')} or as three-element |
| 196 | lists of the form \code{(1, 'name', 56)}. If the third element is |
| 197 | present, it is assumed to be a valid line number. The line number |
| 198 | may be specified for any subset of the terminal symbols in the input |
| 199 | tree. |
| 200 | \end{funcdesc} |
| 201 | |
| 202 | \begin{funcdesc}{tuple2ast}{sequence} |
| 203 | This is the same function as \code{sequence2ast}. This entry point is |
| 204 | maintained for backward compatibility. |
Guido van Rossum | 4b73a06 | 1995-10-11 17:30:04 +0000 | [diff] [blame] | 205 | \end{funcdesc} |
| 206 | |
| 207 | |
Guido van Rossum | 4b73a06 | 1995-10-11 17:30:04 +0000 | [diff] [blame] | 208 | \subsection{Exceptions and Error Handling} |
| 209 | |
| 210 | The parser module defines a single exception, but may also pass other |
| 211 | built-in exceptions from other portions of the Python runtime |
| 212 | environment. See each function for information about the exceptions |
| 213 | it can raise. |
| 214 | |
| 215 | \begin{excdesc}{ParserError} |
| 216 | Exception raised when a failure occurs within the parser module. This |
| 217 | is generally produced for validation failures rather than the built in |
| 218 | \code{SyntaxError} thrown during normal parsing. |
| 219 | The exception argument is either a string describing the reason of the |
Guido van Rossum | 4747887 | 1996-08-21 14:32:37 +0000 | [diff] [blame^] | 220 | failure or a tuple containing a sequence causing the failure from a parse |
| 221 | tree passed to \code{sequence2ast()} and an explanatory string. Calls to |
| 222 | \code{sequence2ast()} need to be able to handle either type of exception, |
Guido van Rossum | 4b73a06 | 1995-10-11 17:30:04 +0000 | [diff] [blame] | 223 | while calls to other functions in the module will only need to be |
| 224 | aware of the simple string values. |
| 225 | \end{excdesc} |
| 226 | |
| 227 | Note that the functions \code{compileast()}, \code{expr()}, and |
| 228 | \code{suite()} may throw exceptions which are normally thrown by the |
| 229 | parsing and compilation process. These include the built in |
| 230 | exceptions \code{MemoryError}, \code{OverflowError}, |
| 231 | \code{SyntaxError}, and \code{SystemError}. In these cases, these |
| 232 | exceptions carry all the meaning normally associated with them. Refer |
| 233 | to the descriptions of each function for detailed information. |
| 234 | |
Guido van Rossum | 4b73a06 | 1995-10-11 17:30:04 +0000 | [diff] [blame] | 235 | |
Guido van Rossum | 4747887 | 1996-08-21 14:32:37 +0000 | [diff] [blame^] | 236 | \subsection{AST Objects} |
| 237 | |
| 238 | AST objects (returned by \code{expr()}, \code{suite()}, and |
| 239 | \code{tuple2ast()}, described above) have no methods of their own. |
| 240 | Some of the functions defined which accept an AST object as their |
| 241 | first argument may change to object methods in the future. |
| 242 | |
| 243 | Ordered and equality comparisons are supported between AST objects. |
| 244 | |
| 245 | |
Guido van Rossum | 4b73a06 | 1995-10-11 17:30:04 +0000 | [diff] [blame] | 246 | \subsection{Example} |
| 247 | |
Guido van Rossum | 4747887 | 1996-08-21 14:32:37 +0000 | [diff] [blame^] | 248 | The parser modules allows operations to be performed on the parse tree |
| 249 | of Python source code before the bytecode is generated, and provides |
| 250 | for inspection of the parse tree for information gathering purposes as |
| 251 | well. While many useful operations may take place between parsing and |
| 252 | bytecode generation, the simplest operation is to do nothing. For |
| 253 | this purpose, using the \code{parser} module to produce an |
| 254 | intermediate data structure is equivelent to the code |
| 255 | |
| 256 | \begin{verbatim} |
| 257 | >>> code = compile('a + 5', 'eval') |
| 258 | >>> a = 5 |
| 259 | >>> eval(code) |
| 260 | 10 |
| 261 | \end{verbatim} |
| 262 | |
| 263 | The equivelent operation using the \code{parser} module is somewhat |
| 264 | longer, and allows the intermediate internal parse tree to be retained |
| 265 | as an AST object: |
Guido van Rossum | 4b73a06 | 1995-10-11 17:30:04 +0000 | [diff] [blame] | 266 | |
| 267 | \begin{verbatim} |
| 268 | >>> import parser |
| 269 | >>> ast = parser.expr('a + 5') |
| 270 | >>> code = parser.compileast(ast) |
| 271 | >>> a = 5 |
| 272 | >>> eval(code) |
| 273 | 10 |
| 274 | \end{verbatim} |
| 275 | |
Guido van Rossum | 4747887 | 1996-08-21 14:32:37 +0000 | [diff] [blame^] | 276 | Some applications can benfit from access to the parse tree itself, and |
| 277 | can take advantage of the intermediate data structure provided by the |
| 278 | \code{parser} module. The remainder of this section of examples will |
| 279 | demonstrate how the intermediate data structure can provide access to |
| 280 | module documentation defined in docstrings without requiring that the |
| 281 | code being examined be imported into a running interpreter. This can |
| 282 | be very useful for performing analyses of untrusted code. |
Guido van Rossum | 4b73a06 | 1995-10-11 17:30:04 +0000 | [diff] [blame] | 283 | |
Guido van Rossum | 4747887 | 1996-08-21 14:32:37 +0000 | [diff] [blame^] | 284 | Generally, the example will demonstrate how the parse tree may be |
| 285 | traversed to distill interesting information. Two functions and a set |
| 286 | of classes is developed which provide programmatic access to high |
| 287 | level function and class definitions provided by a module. The |
| 288 | classes extract information from the parse tree and provide access to |
| 289 | the information at a useful semantic level, one function provides a |
| 290 | simple low-level pattern matching capability, and the other function |
| 291 | defines a high-level interface to the classes by handling file |
| 292 | operations on behalf of the caller. All source files mentioned here |
| 293 | which are not part of the Python installation are located in the |
| 294 | \file{Demo/parser} directory of the distribution. |
Guido van Rossum | 4b73a06 | 1995-10-11 17:30:04 +0000 | [diff] [blame] | 295 | |
Guido van Rossum | 4747887 | 1996-08-21 14:32:37 +0000 | [diff] [blame^] | 296 | To construct the upper-level extraction methods, we need to know what |
| 297 | the parse tree structure looks like and how much of it we actually |
| 298 | need to be concerned about. Python uses a moderately deep parse tree, |
| 299 | so there are a large number of intermediate nodes. It is important to |
| 300 | read and understand the formal grammar used by Python. This is |
| 301 | specified in the file \file{Grammar/Grammar} in the distribution. |
| 302 | Consider the simplest case of interest when searching for docstrings: |
| 303 | a module consisting of a docstring and nothing else: |
Guido van Rossum | 4b73a06 | 1995-10-11 17:30:04 +0000 | [diff] [blame] | 304 | |
Guido van Rossum | 4747887 | 1996-08-21 14:32:37 +0000 | [diff] [blame^] | 305 | \begin{verbatim} |
| 306 | """Some documentation. |
| 307 | """ |
| 308 | \end{verbatim} |
Guido van Rossum | 4b73a06 | 1995-10-11 17:30:04 +0000 | [diff] [blame] | 309 | |
Guido van Rossum | 4747887 | 1996-08-21 14:32:37 +0000 | [diff] [blame^] | 310 | Using the interpreter to take a look at the parse tree, we find a |
| 311 | bewildering mass of numbers and parentheses, with the documentation |
| 312 | buried deep in the nested tuples: |
Guido van Rossum | 4b73a06 | 1995-10-11 17:30:04 +0000 | [diff] [blame] | 313 | |
Guido van Rossum | 4747887 | 1996-08-21 14:32:37 +0000 | [diff] [blame^] | 314 | \begin{verbatim} |
| 315 | >>> import parser |
| 316 | >>> import pprint |
| 317 | >>> ast = parser.suite(open('docstring.py').read()) |
| 318 | >>> tup = parser.ast2tuple(ast) |
| 319 | >>> pprint.pprint(tup) |
| 320 | (257, |
| 321 | (264, |
| 322 | (265, |
| 323 | (266, |
| 324 | (267, |
| 325 | (307, |
| 326 | (287, |
| 327 | (288, |
| 328 | (289, |
| 329 | (290, |
| 330 | (292, |
| 331 | (293, |
| 332 | (294, |
| 333 | (295, |
| 334 | (296, |
| 335 | (297, |
| 336 | (298, |
| 337 | (299, |
| 338 | (300, (3, '"""Some documentation.\012"""'))))))))))))))))), |
| 339 | (4, ''))), |
| 340 | (4, ''), |
| 341 | (0, '')) |
| 342 | \end{verbatim} |
| 343 | |
| 344 | The numbers at the first element of each node in the tree are the node |
| 345 | types; they map directly to terminal and non-terminal symbols in the |
| 346 | grammar. Unfortunately, they are represented as integers in the |
| 347 | internal representation, and the Python structures generated do not |
| 348 | change that. However, the \code{symbol} and \code{token} modules |
| 349 | provide symbolic names for the node types and dictionaries which map |
| 350 | from the integers to the symbolic names for the node types. |
| 351 | |
| 352 | In the output presented above, the outermost tuple contains four |
| 353 | elements: the integer \code{257} and three additional tuples. Node |
| 354 | type \code{257} has the symbolic name \code{file_input}. Each of |
| 355 | these inner tuples contains an integer as the first element; these |
| 356 | integers, \code{264}, \code{4}, and \code{0}, represent the node types |
| 357 | \code{stmt}, \code{NEWLINE}, and \code{ENDMARKER}, respectively. |
| 358 | Note that these values may change depending on the version of Python |
| 359 | you are using; consult \file{symbol.py} and \file{token.py} for |
| 360 | details of the mapping. It should be fairly clear that the outermost |
| 361 | node is related primarily to the input source rather than the contents |
| 362 | of the file, and may be disregarded for the moment. The \code{stmt} |
| 363 | node is much more interesting. In particular, all docstrings are |
| 364 | found in subtrees which are formed exactly as this node is formed, |
| 365 | with the only difference being the string itself. The association |
| 366 | between the docstring in a similar tree and the defined entity (class, |
| 367 | function, or module) which it describes is given by the position of |
| 368 | the docstring subtree within the tree defining the described |
| 369 | structure. |
| 370 | |
| 371 | By replacing the actual docstring with something to signify a variable |
| 372 | component of the tree, we allow a simple pattern matching approach may |
| 373 | be taken to checking any given subtree for equivelence to the general |
| 374 | pattern for docstrings. Since the example demonstrates information |
| 375 | extraction, we can safely require that the tree be in tuple form |
| 376 | rather than list form, allowing a simple variable representation to be |
| 377 | \code{['variable\_name']}. A simple recursive function can implement |
| 378 | the pattern matching, returning a boolean and a dictionary of variable |
| 379 | name to value mappings. |
| 380 | |
| 381 | \begin{verbatim} |
| 382 | from types import ListType, TupleType |
| 383 | |
| 384 | def match(pattern, data, vars=None): |
| 385 | if vars is None: |
| 386 | vars = {} |
| 387 | if type(pattern) is ListType: |
| 388 | vars[pattern[0]] = data |
| 389 | return 1, vars |
| 390 | if type(pattern) is not TupleType: |
| 391 | return (pattern == data), vars |
| 392 | if len(data) != len(pattern): |
| 393 | return 0, vars |
| 394 | for pattern, data in map(None, pattern, data): |
| 395 | same, vars = match(pattern, data, vars) |
| 396 | if not same: |
| 397 | break |
| 398 | return same, vars |
| 399 | \end{verbatim} |
| 400 | |
| 401 | Using this simple recursive pattern matching function and the symbolic |
| 402 | node types, the pattern for the candidate docstring subtrees becomes: |
| 403 | |
| 404 | \begin{verbatim} |
| 405 | >>> DOCSTRING_STMT_PATTERN = ( |
| 406 | ... symbol.stmt, |
| 407 | ... (symbol.simple_stmt, |
| 408 | ... (symbol.small_stmt, |
| 409 | ... (symbol.expr_stmt, |
| 410 | ... (symbol.testlist, |
| 411 | ... (symbol.test, |
| 412 | ... (symbol.and_test, |
| 413 | ... (symbol.not_test, |
| 414 | ... (symbol.comparison, |
| 415 | ... (symbol.expr, |
| 416 | ... (symbol.xor_expr, |
| 417 | ... (symbol.and_expr, |
| 418 | ... (symbol.shift_expr, |
| 419 | ... (symbol.arith_expr, |
| 420 | ... (symbol.term, |
| 421 | ... (symbol.factor, |
| 422 | ... (symbol.power, |
| 423 | ... (symbol.atom, |
| 424 | ... (token.STRING, ['docstring']) |
| 425 | ... )))))))))))))))), |
| 426 | ... (token.NEWLINE, '') |
| 427 | ... )) |
| 428 | \end{verbatim} |
| 429 | |
| 430 | Using the \code{match()} function with this pattern, extracting the |
| 431 | module docstring from the parse tree created previously is easy: |
| 432 | |
| 433 | \begin{verbatim} |
| 434 | >>> found, vars = match(DOCSTRING_STMT_PATTERN, tup[1]) |
| 435 | >>> found |
| 436 | 1 |
| 437 | >>> vars |
| 438 | {'docstring': '"""Some documentation.\012"""'} |
| 439 | \end{verbatim} |
| 440 | |
| 441 | Once specific data can be extracted from a location where it is |
| 442 | expected, the question of where information can be expected |
| 443 | needs to be answered. When dealing with docstrings, the answer is |
| 444 | fairly simple: the docstring is the first \code{stmt} node in a code |
| 445 | block (\code{file_input} or \code{suite} node types). A module |
| 446 | consists of a single \code{file_input} node, and class and function |
| 447 | definitions each contain exactly one \code{suite} node. Classes and |
| 448 | functions are readily identified as subtrees of code block nodes which |
| 449 | start with \code{(stmt, (compound_stmt, (classdef, ...} or |
| 450 | \code{(stmt, (compound_stmt, (funcdef, ...}. Note that these subtrees |
| 451 | cannot be matched by \code{match()} since it does not support multiple |
| 452 | sibling nodes to match without regard to number. A more elaborate |
| 453 | matching function could be used to overcome this limitation, but this |
| 454 | is sufficient for the example. |
| 455 | |
| 456 | |
| 457 | |
| 458 | %% |
| 459 | %% end of file |