Georg Brandl | 0c77a82 | 2008-06-10 16:37:50 +0000 | [diff] [blame] | 1 | .. _ast: |
| 2 | |
| 3 | Abstract Syntax Trees |
| 4 | ===================== |
| 5 | |
| 6 | .. module:: ast |
| 7 | :synopsis: Abstract Syntax Tree classes and manipulation. |
| 8 | |
| 9 | .. sectionauthor:: Martin v. Lรถwis <martin@v.loewis.de> |
| 10 | .. sectionauthor:: Georg Brandl <georg@python.org> |
| 11 | |
Georg Brandl | 0c77a82 | 2008-06-10 16:37:50 +0000 | [diff] [blame] | 12 | |
| 13 | The :mod:`ast` module helps Python applications to process trees of the Python |
| 14 | abstract syntax grammar. The abstract syntax itself might change with each |
| 15 | Python release; this module helps to find out programmatically what the current |
| 16 | grammar looks like. |
| 17 | |
Benjamin Peterson | ec9199b | 2008-11-08 17:05:00 +0000 | [diff] [blame] | 18 | An abstract syntax tree can be generated by passing :data:`ast.PyCF_ONLY_AST` as |
| 19 | a flag to the :func:`compile` builtin function, or using the :func:`parse` |
Georg Brandl | 0c77a82 | 2008-06-10 16:37:50 +0000 | [diff] [blame] | 20 | helper provided in this module. The result will be a tree of objects whose |
Benjamin Peterson | ec9199b | 2008-11-08 17:05:00 +0000 | [diff] [blame] | 21 | classes all inherit from :class:`ast.AST`. An abstract syntax tree can be |
| 22 | compiled into a Python code object using the built-in :func:`compile` function. |
Georg Brandl | 0c77a82 | 2008-06-10 16:37:50 +0000 | [diff] [blame] | 23 | |
Georg Brandl | 0c77a82 | 2008-06-10 16:37:50 +0000 | [diff] [blame] | 24 | |
| 25 | Node classes |
| 26 | ------------ |
| 27 | |
| 28 | .. class:: AST |
| 29 | |
| 30 | This is the base of all AST node classes. The actual node classes are |
| 31 | derived from the :file:`Parser/Python.asdl` file, which is reproduced |
| 32 | :ref:`below <abstract-grammar>`. They are defined in the :mod:`_ast` C |
| 33 | module and re-exported in :mod:`ast`. |
| 34 | |
| 35 | There is one class defined for each left-hand side symbol in the abstract |
| 36 | grammar (for example, :class:`ast.stmt` or :class:`ast.expr`). In addition, |
| 37 | there is one class defined for each constructor on the right-hand side; these |
| 38 | classes inherit from the classes for the left-hand side trees. For example, |
| 39 | :class:`ast.BinOp` inherits from :class:`ast.expr`. For production rules |
| 40 | with alternatives (aka "sums"), the left-hand side class is abstract: only |
| 41 | instances of specific constructor nodes are ever created. |
| 42 | |
| 43 | .. attribute:: _fields |
| 44 | |
| 45 | Each concrete class has an attribute :attr:`_fields` which gives the names |
| 46 | of all child nodes. |
| 47 | |
| 48 | Each instance of a concrete class has one attribute for each child node, |
| 49 | of the type as defined in the grammar. For example, :class:`ast.BinOp` |
| 50 | instances have an attribute :attr:`left` of type :class:`ast.expr`. |
| 51 | |
| 52 | If these attributes are marked as optional in the grammar (using a |
| 53 | question mark), the value might be ``None``. If the attributes can have |
| 54 | zero-or-more values (marked with an asterisk), the values are represented |
| 55 | as Python lists. All possible attributes must be present and have valid |
| 56 | values when compiling an AST with :func:`compile`. |
| 57 | |
| 58 | .. attribute:: lineno |
| 59 | col_offset |
| 60 | |
| 61 | Instances of :class:`ast.expr` and :class:`ast.stmt` subclasses have |
| 62 | :attr:`lineno` and :attr:`col_offset` attributes. The :attr:`lineno` is |
| 63 | the line number of source text (1-indexed so the first line is line 1) and |
| 64 | the :attr:`col_offset` is the UTF-8 byte offset of the first token that |
| 65 | generated the node. The UTF-8 offset is recorded because the parser uses |
| 66 | UTF-8 internally. |
| 67 | |
| 68 | The constructor of a class :class:`ast.T` parses its arguments as follows: |
| 69 | |
| 70 | * If there are positional arguments, there must be as many as there are items |
| 71 | in :attr:`T._fields`; they will be assigned as attributes of these names. |
| 72 | * If there are keyword arguments, they will set the attributes of the same |
| 73 | names to the given values. |
| 74 | |
| 75 | For example, to create and populate an :class:`ast.UnaryOp` node, you could |
| 76 | use :: |
| 77 | |
| 78 | node = ast.UnaryOp() |
| 79 | node.op = ast.USub() |
| 80 | node.operand = ast.Num() |
| 81 | node.operand.n = 5 |
| 82 | node.operand.lineno = 0 |
| 83 | node.operand.col_offset = 0 |
| 84 | node.lineno = 0 |
| 85 | node.col_offset = 0 |
| 86 | |
| 87 | or the more compact :: |
| 88 | |
| 89 | node = ast.UnaryOp(ast.USub(), ast.Num(5, lineno=0, col_offset=0), |
| 90 | lineno=0, col_offset=0) |
| 91 | |
| 92 | |
| 93 | .. _abstract-grammar: |
| 94 | |
| 95 | Abstract Grammar |
| 96 | ---------------- |
| 97 | |
| 98 | The module defines a string constant ``__version__`` which is the decimal |
| 99 | Subversion revision number of the file shown below. |
| 100 | |
| 101 | The abstract grammar is currently defined as follows: |
| 102 | |
| 103 | .. literalinclude:: ../../Parser/Python.asdl |
| 104 | |
| 105 | |
| 106 | :mod:`ast` Helpers |
| 107 | ------------------ |
| 108 | |
Georg Brandl | 0c77a82 | 2008-06-10 16:37:50 +0000 | [diff] [blame] | 109 | Apart from the node classes, :mod:`ast` module defines these utility functions |
| 110 | and classes for traversing abstract syntax trees: |
| 111 | |
| 112 | .. function:: parse(expr, filename='<unknown>', mode='exec') |
| 113 | |
| 114 | Parse an expression into an AST node. Equivalent to ``compile(expr, |
Benjamin Peterson | ec9199b | 2008-11-08 17:05:00 +0000 | [diff] [blame] | 115 | filename, mode, ast.PyCF_ONLY_AST)``. |
Georg Brandl | 0c77a82 | 2008-06-10 16:37:50 +0000 | [diff] [blame] | 116 | |
Georg Brandl | 48310cd | 2009-01-03 21:18:54 +0000 | [diff] [blame] | 117 | |
Georg Brandl | 0c77a82 | 2008-06-10 16:37:50 +0000 | [diff] [blame] | 118 | .. function:: literal_eval(node_or_string) |
| 119 | |
| 120 | Safely evaluate an expression node or a string containing a Python |
| 121 | expression. The string or node provided may only consist of the following |
| 122 | Python literal structures: strings, numbers, tuples, lists, dicts, booleans, |
| 123 | and ``None``. |
| 124 | |
| 125 | This can be used for safely evaluating strings containing Python expressions |
| 126 | from untrusted sources without the need to parse the values oneself. |
| 127 | |
| 128 | |
Amaury Forgeot d'Arc | fdfe62d | 2008-06-17 20:36:03 +0000 | [diff] [blame] | 129 | .. function:: get_docstring(node, clean=True) |
Georg Brandl | 0c77a82 | 2008-06-10 16:37:50 +0000 | [diff] [blame] | 130 | |
| 131 | Return the docstring of the given *node* (which must be a |
| 132 | :class:`FunctionDef`, :class:`ClassDef` or :class:`Module` node), or ``None`` |
| 133 | if it has no docstring. If *clean* is true, clean up the docstring's |
| 134 | indentation with :func:`inspect.cleandoc`. |
| 135 | |
| 136 | |
| 137 | .. function:: fix_missing_locations(node) |
| 138 | |
| 139 | When you compile a node tree with :func:`compile`, the compiler expects |
| 140 | :attr:`lineno` and :attr:`col_offset` attributes for every node that supports |
| 141 | them. This is rather tedious to fill in for generated nodes, so this helper |
| 142 | adds these attributes recursively where not already set, by setting them to |
| 143 | the values of the parent node. It works recursively starting at *node*. |
| 144 | |
| 145 | |
| 146 | .. function:: increment_lineno(node, n=1) |
| 147 | |
| 148 | Increment the line number of each node in the tree starting at *node* by *n*. |
| 149 | This is useful to "move code" to a different location in a file. |
| 150 | |
| 151 | |
| 152 | .. function:: copy_location(new_node, old_node) |
| 153 | |
| 154 | Copy source location (:attr:`lineno` and :attr:`col_offset`) from *old_node* |
| 155 | to *new_node* if possible, and return *new_node*. |
| 156 | |
| 157 | |
| 158 | .. function:: iter_fields(node) |
| 159 | |
| 160 | Yield a tuple of ``(fieldname, value)`` for each field in ``node._fields`` |
| 161 | that is present on *node*. |
| 162 | |
| 163 | |
| 164 | .. function:: iter_child_nodes(node) |
| 165 | |
| 166 | Yield all direct child nodes of *node*, that is, all fields that are nodes |
| 167 | and all items of fields that are lists of nodes. |
| 168 | |
| 169 | |
| 170 | .. function:: walk(node) |
| 171 | |
| 172 | Recursively yield all child nodes of *node*, in no specified order. This is |
| 173 | useful if you only want to modify nodes in place and don't care about the |
| 174 | context. |
| 175 | |
| 176 | |
| 177 | .. class:: NodeVisitor() |
| 178 | |
| 179 | A node visitor base class that walks the abstract syntax tree and calls a |
| 180 | visitor function for every node found. This function may return a value |
Georg Brandl | 36ab1ef | 2009-01-03 21:17:04 +0000 | [diff] [blame] | 181 | which is forwarded by the :meth:`visit` method. |
Georg Brandl | 0c77a82 | 2008-06-10 16:37:50 +0000 | [diff] [blame] | 182 | |
| 183 | This class is meant to be subclassed, with the subclass adding visitor |
| 184 | methods. |
| 185 | |
| 186 | .. method:: visit(node) |
| 187 | |
| 188 | Visit a node. The default implementation calls the method called |
| 189 | :samp:`self.visit_{classname}` where *classname* is the name of the node |
| 190 | class, or :meth:`generic_visit` if that method doesn't exist. |
| 191 | |
| 192 | .. method:: generic_visit(node) |
| 193 | |
| 194 | This visitor calls :meth:`visit` on all children of the node. |
Georg Brandl | 48310cd | 2009-01-03 21:18:54 +0000 | [diff] [blame] | 195 | |
Georg Brandl | 0c77a82 | 2008-06-10 16:37:50 +0000 | [diff] [blame] | 196 | Note that child nodes of nodes that have a custom visitor method won't be |
| 197 | visited unless the visitor calls :meth:`generic_visit` or visits them |
| 198 | itself. |
| 199 | |
| 200 | Don't use the :class:`NodeVisitor` if you want to apply changes to nodes |
| 201 | during traversal. For this a special visitor exists |
| 202 | (:class:`NodeTransformer`) that allows modifications. |
| 203 | |
| 204 | |
| 205 | .. class:: NodeTransformer() |
| 206 | |
| 207 | A :class:`NodeVisitor` subclass that walks the abstract syntax tree and |
| 208 | allows modification of nodes. |
| 209 | |
Georg Brandl | 36ab1ef | 2009-01-03 21:17:04 +0000 | [diff] [blame] | 210 | The :class:`NodeTransformer` will walk the AST and use the return value of |
| 211 | the visitor methods to replace or remove the old node. If the return value |
| 212 | of the visitor method is ``None``, the node will be removed from its |
| 213 | location, otherwise it is replaced with the return value. The return value |
| 214 | may be the original node in which case no replacement takes place. |
Georg Brandl | 0c77a82 | 2008-06-10 16:37:50 +0000 | [diff] [blame] | 215 | |
| 216 | Here is an example transformer that rewrites all occurrences of name lookups |
| 217 | (``foo``) to ``data['foo']``:: |
| 218 | |
| 219 | class RewriteName(NodeTransformer): |
| 220 | |
| 221 | def visit_Name(self, node): |
| 222 | return copy_location(Subscript( |
| 223 | value=Name(id='data', ctx=Load()), |
| 224 | slice=Index(value=Str(s=node.id)), |
| 225 | ctx=node.ctx |
| 226 | ), node) |
| 227 | |
| 228 | Keep in mind that if the node you're operating on has child nodes you must |
| 229 | either transform the child nodes yourself or call the :meth:`generic_visit` |
| 230 | method for the node first. |
| 231 | |
| 232 | For nodes that were part of a collection of statements (that applies to all |
| 233 | statement nodes), the visitor may also return a list of nodes rather than |
| 234 | just a single node. |
| 235 | |
| 236 | Usually you use the transformer like this:: |
| 237 | |
| 238 | node = YourTransformer().visit(node) |
| 239 | |
| 240 | |
| 241 | .. function:: dump(node, annotate_fields=True, include_attributes=False) |
| 242 | |
| 243 | Return a formatted dump of the tree in *node*. This is mainly useful for |
| 244 | debugging purposes. The returned string will show the names and the values |
| 245 | for fields. This makes the code impossible to evaluate, so if evaluation is |
| 246 | wanted *annotate_fields* must be set to False. Attributes such as line |
Benjamin Peterson | dcf97b9 | 2008-07-02 17:30:14 +0000 | [diff] [blame] | 247 | numbers and column offsets are not dumped by default. If this is wanted, |
Georg Brandl | 0c77a82 | 2008-06-10 16:37:50 +0000 | [diff] [blame] | 248 | *include_attributes* can be set to ``True``. |