Alexander Belopolsky | f0a0d14 | 2010-10-27 03:06:43 +0000 | [diff] [blame] | 1 | :mod:`ast` --- Abstract Syntax Trees |
| 2 | ==================================== |
Georg Brandl | 0c77a82 | 2008-06-10 16:37:50 +0000 | [diff] [blame] | 3 | |
| 4 | .. module:: ast |
| 5 | :synopsis: Abstract Syntax Tree classes and manipulation. |
| 6 | |
| 7 | .. sectionauthor:: Martin v. Lรถwis <martin@v.loewis.de> |
| 8 | .. sectionauthor:: Georg Brandl <georg@python.org> |
| 9 | |
Raymond Hettinger | 1048094 | 2011-01-10 03:26:08 +0000 | [diff] [blame] | 10 | **Source code:** :source:`Lib/ast.py` |
Georg Brandl | 0c77a82 | 2008-06-10 16:37:50 +0000 | [diff] [blame] | 11 | |
Raymond Hettinger | 4f707fd | 2011-01-10 19:54:11 +0000 | [diff] [blame] | 12 | -------------- |
| 13 | |
Georg Brandl | 0c77a82 | 2008-06-10 16:37:50 +0000 | [diff] [blame] | 14 | The :mod:`ast` module helps Python applications to process trees of the Python |
| 15 | abstract syntax grammar. The abstract syntax itself might change with each |
| 16 | Python release; this module helps to find out programmatically what the current |
| 17 | grammar looks like. |
| 18 | |
Benjamin Peterson | ec9199b | 2008-11-08 17:05:00 +0000 | [diff] [blame] | 19 | An abstract syntax tree can be generated by passing :data:`ast.PyCF_ONLY_AST` as |
Georg Brandl | 22b3431 | 2009-07-26 14:54:51 +0000 | [diff] [blame] | 20 | a flag to the :func:`compile` built-in function, or using the :func:`parse` |
Georg Brandl | 0c77a82 | 2008-06-10 16:37:50 +0000 | [diff] [blame] | 21 | helper provided in this module. The result will be a tree of objects whose |
Benjamin Peterson | ec9199b | 2008-11-08 17:05:00 +0000 | [diff] [blame] | 22 | classes all inherit from :class:`ast.AST`. An abstract syntax tree can be |
| 23 | compiled into a Python code object using the built-in :func:`compile` function. |
Georg Brandl | 0c77a82 | 2008-06-10 16:37:50 +0000 | [diff] [blame] | 24 | |
Georg Brandl | 0c77a82 | 2008-06-10 16:37:50 +0000 | [diff] [blame] | 25 | |
| 26 | Node classes |
| 27 | ------------ |
| 28 | |
| 29 | .. class:: AST |
| 30 | |
| 31 | This is the base of all AST node classes. The actual node classes are |
| 32 | derived from the :file:`Parser/Python.asdl` file, which is reproduced |
| 33 | :ref:`below <abstract-grammar>`. They are defined in the :mod:`_ast` C |
| 34 | module and re-exported in :mod:`ast`. |
| 35 | |
| 36 | There is one class defined for each left-hand side symbol in the abstract |
| 37 | grammar (for example, :class:`ast.stmt` or :class:`ast.expr`). In addition, |
| 38 | there is one class defined for each constructor on the right-hand side; these |
| 39 | classes inherit from the classes for the left-hand side trees. For example, |
| 40 | :class:`ast.BinOp` inherits from :class:`ast.expr`. For production rules |
| 41 | with alternatives (aka "sums"), the left-hand side class is abstract: only |
| 42 | instances of specific constructor nodes are ever created. |
| 43 | |
| 44 | .. attribute:: _fields |
| 45 | |
| 46 | Each concrete class has an attribute :attr:`_fields` which gives the names |
| 47 | of all child nodes. |
| 48 | |
| 49 | Each instance of a concrete class has one attribute for each child node, |
| 50 | of the type as defined in the grammar. For example, :class:`ast.BinOp` |
| 51 | instances have an attribute :attr:`left` of type :class:`ast.expr`. |
| 52 | |
| 53 | If these attributes are marked as optional in the grammar (using a |
| 54 | question mark), the value might be ``None``. If the attributes can have |
| 55 | zero-or-more values (marked with an asterisk), the values are represented |
| 56 | as Python lists. All possible attributes must be present and have valid |
| 57 | values when compiling an AST with :func:`compile`. |
| 58 | |
| 59 | .. attribute:: lineno |
| 60 | col_offset |
| 61 | |
| 62 | Instances of :class:`ast.expr` and :class:`ast.stmt` subclasses have |
| 63 | :attr:`lineno` and :attr:`col_offset` attributes. The :attr:`lineno` is |
| 64 | the line number of source text (1-indexed so the first line is line 1) and |
| 65 | the :attr:`col_offset` is the UTF-8 byte offset of the first token that |
| 66 | generated the node. The UTF-8 offset is recorded because the parser uses |
| 67 | UTF-8 internally. |
| 68 | |
| 69 | The constructor of a class :class:`ast.T` parses its arguments as follows: |
| 70 | |
| 71 | * If there are positional arguments, there must be as many as there are items |
| 72 | in :attr:`T._fields`; they will be assigned as attributes of these names. |
| 73 | * If there are keyword arguments, they will set the attributes of the same |
| 74 | names to the given values. |
| 75 | |
| 76 | For example, to create and populate an :class:`ast.UnaryOp` node, you could |
| 77 | use :: |
| 78 | |
| 79 | node = ast.UnaryOp() |
| 80 | node.op = ast.USub() |
| 81 | node.operand = ast.Num() |
| 82 | node.operand.n = 5 |
| 83 | node.operand.lineno = 0 |
| 84 | node.operand.col_offset = 0 |
| 85 | node.lineno = 0 |
| 86 | node.col_offset = 0 |
| 87 | |
| 88 | or the more compact :: |
| 89 | |
| 90 | node = ast.UnaryOp(ast.USub(), ast.Num(5, lineno=0, col_offset=0), |
| 91 | lineno=0, col_offset=0) |
| 92 | |
| 93 | |
| 94 | .. _abstract-grammar: |
| 95 | |
| 96 | Abstract Grammar |
| 97 | ---------------- |
| 98 | |
Georg Brandl | 0c77a82 | 2008-06-10 16:37:50 +0000 | [diff] [blame] | 99 | The abstract grammar is currently defined as follows: |
| 100 | |
| 101 | .. literalinclude:: ../../Parser/Python.asdl |
| 102 | |
| 103 | |
| 104 | :mod:`ast` Helpers |
| 105 | ------------------ |
| 106 | |
Georg Brandl | 0c77a82 | 2008-06-10 16:37:50 +0000 | [diff] [blame] | 107 | Apart from the node classes, :mod:`ast` module defines these utility functions |
| 108 | and classes for traversing abstract syntax trees: |
| 109 | |
Terry Reedy | feac624 | 2011-01-24 21:36:03 +0000 | [diff] [blame] | 110 | .. function:: parse(source, filename='<unknown>', mode='exec') |
Georg Brandl | 0c77a82 | 2008-06-10 16:37:50 +0000 | [diff] [blame] | 111 | |
Terry Reedy | feac624 | 2011-01-24 21:36:03 +0000 | [diff] [blame] | 112 | Parse the source into an AST node. Equivalent to ``compile(source, |
Benjamin Peterson | ec9199b | 2008-11-08 17:05:00 +0000 | [diff] [blame] | 113 | filename, mode, ast.PyCF_ONLY_AST)``. |
Georg Brandl | 0c77a82 | 2008-06-10 16:37:50 +0000 | [diff] [blame] | 114 | |
Georg Brandl | 48310cd | 2009-01-03 21:18:54 +0000 | [diff] [blame] | 115 | |
Georg Brandl | 0c77a82 | 2008-06-10 16:37:50 +0000 | [diff] [blame] | 116 | .. function:: literal_eval(node_or_string) |
| 117 | |
| 118 | Safely evaluate an expression node or a string containing a Python |
| 119 | expression. The string or node provided may only consist of the following |
Georg Brandl | 85f2177 | 2010-07-13 06:38:10 +0000 | [diff] [blame] | 120 | Python literal structures: strings, bytes, numbers, tuples, lists, dicts, |
| 121 | sets, booleans, and ``None``. |
Georg Brandl | 0c77a82 | 2008-06-10 16:37:50 +0000 | [diff] [blame] | 122 | |
| 123 | This can be used for safely evaluating strings containing Python expressions |
| 124 | from untrusted sources without the need to parse the values oneself. |
| 125 | |
Georg Brandl | 492f3fc | 2010-07-11 09:41:21 +0000 | [diff] [blame] | 126 | .. versionchanged:: 3.2 |
Georg Brandl | 85f2177 | 2010-07-13 06:38:10 +0000 | [diff] [blame] | 127 | Now allows bytes and set literals. |
Georg Brandl | 492f3fc | 2010-07-11 09:41:21 +0000 | [diff] [blame] | 128 | |
Georg Brandl | 0c77a82 | 2008-06-10 16:37:50 +0000 | [diff] [blame] | 129 | |
Amaury Forgeot d'Arc | fdfe62d | 2008-06-17 20:36:03 +0000 | [diff] [blame] | 130 | .. function:: get_docstring(node, clean=True) |
Georg Brandl | 0c77a82 | 2008-06-10 16:37:50 +0000 | [diff] [blame] | 131 | |
| 132 | Return the docstring of the given *node* (which must be a |
| 133 | :class:`FunctionDef`, :class:`ClassDef` or :class:`Module` node), or ``None`` |
| 134 | if it has no docstring. If *clean* is true, clean up the docstring's |
| 135 | indentation with :func:`inspect.cleandoc`. |
| 136 | |
| 137 | |
| 138 | .. function:: fix_missing_locations(node) |
| 139 | |
| 140 | When you compile a node tree with :func:`compile`, the compiler expects |
| 141 | :attr:`lineno` and :attr:`col_offset` attributes for every node that supports |
| 142 | them. This is rather tedious to fill in for generated nodes, so this helper |
| 143 | adds these attributes recursively where not already set, by setting them to |
| 144 | the values of the parent node. It works recursively starting at *node*. |
| 145 | |
| 146 | |
| 147 | .. function:: increment_lineno(node, n=1) |
| 148 | |
| 149 | Increment the line number of each node in the tree starting at *node* by *n*. |
| 150 | This is useful to "move code" to a different location in a file. |
| 151 | |
| 152 | |
| 153 | .. function:: copy_location(new_node, old_node) |
| 154 | |
| 155 | Copy source location (:attr:`lineno` and :attr:`col_offset`) from *old_node* |
| 156 | to *new_node* if possible, and return *new_node*. |
| 157 | |
| 158 | |
| 159 | .. function:: iter_fields(node) |
| 160 | |
| 161 | Yield a tuple of ``(fieldname, value)`` for each field in ``node._fields`` |
| 162 | that is present on *node*. |
| 163 | |
| 164 | |
| 165 | .. function:: iter_child_nodes(node) |
| 166 | |
| 167 | Yield all direct child nodes of *node*, that is, all fields that are nodes |
| 168 | and all items of fields that are lists of nodes. |
| 169 | |
| 170 | |
| 171 | .. function:: walk(node) |
| 172 | |
Georg Brandl | 619e7ba | 2011-01-09 07:38:51 +0000 | [diff] [blame] | 173 | Recursively yield all descendant nodes in the tree starting at *node* |
| 174 | (including *node* itself), in no specified order. This is useful if you only |
| 175 | want to modify nodes in place and don't care about the context. |
Georg Brandl | 0c77a82 | 2008-06-10 16:37:50 +0000 | [diff] [blame] | 176 | |
| 177 | |
| 178 | .. class:: NodeVisitor() |
| 179 | |
| 180 | A node visitor base class that walks the abstract syntax tree and calls a |
| 181 | visitor function for every node found. This function may return a value |
Georg Brandl | 36ab1ef | 2009-01-03 21:17:04 +0000 | [diff] [blame] | 182 | which is forwarded by the :meth:`visit` method. |
Georg Brandl | 0c77a82 | 2008-06-10 16:37:50 +0000 | [diff] [blame] | 183 | |
| 184 | This class is meant to be subclassed, with the subclass adding visitor |
| 185 | methods. |
| 186 | |
| 187 | .. method:: visit(node) |
| 188 | |
| 189 | Visit a node. The default implementation calls the method called |
| 190 | :samp:`self.visit_{classname}` where *classname* is the name of the node |
| 191 | class, or :meth:`generic_visit` if that method doesn't exist. |
| 192 | |
| 193 | .. method:: generic_visit(node) |
| 194 | |
| 195 | This visitor calls :meth:`visit` on all children of the node. |
Georg Brandl | 48310cd | 2009-01-03 21:18:54 +0000 | [diff] [blame] | 196 | |
Georg Brandl | 0c77a82 | 2008-06-10 16:37:50 +0000 | [diff] [blame] | 197 | Note that child nodes of nodes that have a custom visitor method won't be |
| 198 | visited unless the visitor calls :meth:`generic_visit` or visits them |
| 199 | itself. |
| 200 | |
| 201 | Don't use the :class:`NodeVisitor` if you want to apply changes to nodes |
| 202 | during traversal. For this a special visitor exists |
| 203 | (:class:`NodeTransformer`) that allows modifications. |
| 204 | |
| 205 | |
| 206 | .. class:: NodeTransformer() |
| 207 | |
| 208 | A :class:`NodeVisitor` subclass that walks the abstract syntax tree and |
| 209 | allows modification of nodes. |
| 210 | |
Georg Brandl | 36ab1ef | 2009-01-03 21:17:04 +0000 | [diff] [blame] | 211 | The :class:`NodeTransformer` will walk the AST and use the return value of |
| 212 | the visitor methods to replace or remove the old node. If the return value |
| 213 | of the visitor method is ``None``, the node will be removed from its |
| 214 | location, otherwise it is replaced with the return value. The return value |
| 215 | may be the original node in which case no replacement takes place. |
Georg Brandl | 0c77a82 | 2008-06-10 16:37:50 +0000 | [diff] [blame] | 216 | |
| 217 | Here is an example transformer that rewrites all occurrences of name lookups |
| 218 | (``foo``) to ``data['foo']``:: |
| 219 | |
| 220 | class RewriteName(NodeTransformer): |
| 221 | |
| 222 | def visit_Name(self, node): |
| 223 | return copy_location(Subscript( |
| 224 | value=Name(id='data', ctx=Load()), |
| 225 | slice=Index(value=Str(s=node.id)), |
| 226 | ctx=node.ctx |
| 227 | ), node) |
| 228 | |
| 229 | Keep in mind that if the node you're operating on has child nodes you must |
| 230 | either transform the child nodes yourself or call the :meth:`generic_visit` |
| 231 | method for the node first. |
| 232 | |
| 233 | For nodes that were part of a collection of statements (that applies to all |
| 234 | statement nodes), the visitor may also return a list of nodes rather than |
| 235 | just a single node. |
| 236 | |
| 237 | Usually you use the transformer like this:: |
| 238 | |
| 239 | node = YourTransformer().visit(node) |
| 240 | |
| 241 | |
| 242 | .. function:: dump(node, annotate_fields=True, include_attributes=False) |
| 243 | |
| 244 | Return a formatted dump of the tree in *node*. This is mainly useful for |
| 245 | debugging purposes. The returned string will show the names and the values |
| 246 | for fields. This makes the code impossible to evaluate, so if evaluation is |
Serhiy Storchaka | fbc1c26 | 2013-11-29 12:17:13 +0200 | [diff] [blame] | 247 | wanted *annotate_fields* must be set to ``False``. Attributes such as line |
Benjamin Peterson | dcf97b9 | 2008-07-02 17:30:14 +0000 | [diff] [blame] | 248 | numbers and column offsets are not dumped by default. If this is wanted, |
Georg Brandl | 0c77a82 | 2008-06-10 16:37:50 +0000 | [diff] [blame] | 249 | *include_attributes* can be set to ``True``. |