blob: 16de3ca09723c713153b8bf0bc23e992cf19d1f1 [file] [log] [blame]
Alexander Belopolskyf0a0d142010-10-27 03:06:43 +00001:mod:`ast` --- Abstract Syntax Trees
2====================================
Georg Brandl0c77a822008-06-10 16:37:50 +00003
4.. module:: ast
5 :synopsis: Abstract Syntax Tree classes and manipulation.
6
7.. sectionauthor:: Martin v. Lรถwis <martin@v.loewis.de>
8.. sectionauthor:: Georg Brandl <georg@python.org>
9
Raymond Hettinger10480942011-01-10 03:26:08 +000010**Source code:** :source:`Lib/ast.py`
Georg Brandl0c77a822008-06-10 16:37:50 +000011
Raymond Hettinger4f707fd2011-01-10 19:54:11 +000012--------------
13
Georg Brandl0c77a822008-06-10 16:37:50 +000014The :mod:`ast` module helps Python applications to process trees of the Python
15abstract syntax grammar. The abstract syntax itself might change with each
16Python release; this module helps to find out programmatically what the current
17grammar looks like.
18
Benjamin Petersonec9199b2008-11-08 17:05:00 +000019An abstract syntax tree can be generated by passing :data:`ast.PyCF_ONLY_AST` as
Georg Brandl22b34312009-07-26 14:54:51 +000020a flag to the :func:`compile` built-in function, or using the :func:`parse`
Georg Brandl0c77a822008-06-10 16:37:50 +000021helper provided in this module. The result will be a tree of objects whose
Benjamin Petersonec9199b2008-11-08 17:05:00 +000022classes all inherit from :class:`ast.AST`. An abstract syntax tree can be
23compiled into a Python code object using the built-in :func:`compile` function.
Georg Brandl0c77a822008-06-10 16:37:50 +000024
Georg Brandl0c77a822008-06-10 16:37:50 +000025
26Node classes
27------------
28
29.. class:: AST
30
31 This is the base of all AST node classes. The actual node classes are
32 derived from the :file:`Parser/Python.asdl` file, which is reproduced
33 :ref:`below <abstract-grammar>`. They are defined in the :mod:`_ast` C
34 module and re-exported in :mod:`ast`.
35
36 There is one class defined for each left-hand side symbol in the abstract
37 grammar (for example, :class:`ast.stmt` or :class:`ast.expr`). In addition,
38 there is one class defined for each constructor on the right-hand side; these
39 classes inherit from the classes for the left-hand side trees. For example,
40 :class:`ast.BinOp` inherits from :class:`ast.expr`. For production rules
41 with alternatives (aka "sums"), the left-hand side class is abstract: only
42 instances of specific constructor nodes are ever created.
43
44 .. attribute:: _fields
45
46 Each concrete class has an attribute :attr:`_fields` which gives the names
47 of all child nodes.
48
49 Each instance of a concrete class has one attribute for each child node,
50 of the type as defined in the grammar. For example, :class:`ast.BinOp`
51 instances have an attribute :attr:`left` of type :class:`ast.expr`.
52
53 If these attributes are marked as optional in the grammar (using a
54 question mark), the value might be ``None``. If the attributes can have
55 zero-or-more values (marked with an asterisk), the values are represented
56 as Python lists. All possible attributes must be present and have valid
57 values when compiling an AST with :func:`compile`.
58
59 .. attribute:: lineno
60 col_offset
61
62 Instances of :class:`ast.expr` and :class:`ast.stmt` subclasses have
63 :attr:`lineno` and :attr:`col_offset` attributes. The :attr:`lineno` is
64 the line number of source text (1-indexed so the first line is line 1) and
65 the :attr:`col_offset` is the UTF-8 byte offset of the first token that
66 generated the node. The UTF-8 offset is recorded because the parser uses
67 UTF-8 internally.
68
69 The constructor of a class :class:`ast.T` parses its arguments as follows:
70
71 * If there are positional arguments, there must be as many as there are items
72 in :attr:`T._fields`; they will be assigned as attributes of these names.
73 * If there are keyword arguments, they will set the attributes of the same
74 names to the given values.
75
76 For example, to create and populate an :class:`ast.UnaryOp` node, you could
77 use ::
78
79 node = ast.UnaryOp()
80 node.op = ast.USub()
81 node.operand = ast.Num()
82 node.operand.n = 5
83 node.operand.lineno = 0
84 node.operand.col_offset = 0
85 node.lineno = 0
86 node.col_offset = 0
87
88 or the more compact ::
89
90 node = ast.UnaryOp(ast.USub(), ast.Num(5, lineno=0, col_offset=0),
91 lineno=0, col_offset=0)
92
93
94.. _abstract-grammar:
95
96Abstract Grammar
97----------------
98
Georg Brandl0c77a822008-06-10 16:37:50 +000099The abstract grammar is currently defined as follows:
100
101.. literalinclude:: ../../Parser/Python.asdl
102
103
104:mod:`ast` Helpers
105------------------
106
Georg Brandl0c77a822008-06-10 16:37:50 +0000107Apart from the node classes, :mod:`ast` module defines these utility functions
108and classes for traversing abstract syntax trees:
109
Terry Reedyfeac6242011-01-24 21:36:03 +0000110.. function:: parse(source, filename='<unknown>', mode='exec')
Georg Brandl0c77a822008-06-10 16:37:50 +0000111
Terry Reedyfeac6242011-01-24 21:36:03 +0000112 Parse the source into an AST node. Equivalent to ``compile(source,
Benjamin Petersonec9199b2008-11-08 17:05:00 +0000113 filename, mode, ast.PyCF_ONLY_AST)``.
Georg Brandl0c77a822008-06-10 16:37:50 +0000114
Georg Brandl48310cd2009-01-03 21:18:54 +0000115
Georg Brandl0c77a822008-06-10 16:37:50 +0000116.. function:: literal_eval(node_or_string)
117
118 Safely evaluate an expression node or a string containing a Python
119 expression. The string or node provided may only consist of the following
Georg Brandl85f21772010-07-13 06:38:10 +0000120 Python literal structures: strings, bytes, numbers, tuples, lists, dicts,
121 sets, booleans, and ``None``.
Georg Brandl0c77a822008-06-10 16:37:50 +0000122
123 This can be used for safely evaluating strings containing Python expressions
124 from untrusted sources without the need to parse the values oneself.
125
Georg Brandl492f3fc2010-07-11 09:41:21 +0000126 .. versionchanged:: 3.2
Georg Brandl85f21772010-07-13 06:38:10 +0000127 Now allows bytes and set literals.
Georg Brandl492f3fc2010-07-11 09:41:21 +0000128
Georg Brandl0c77a822008-06-10 16:37:50 +0000129
Amaury Forgeot d'Arcfdfe62d2008-06-17 20:36:03 +0000130.. function:: get_docstring(node, clean=True)
Georg Brandl0c77a822008-06-10 16:37:50 +0000131
132 Return the docstring of the given *node* (which must be a
133 :class:`FunctionDef`, :class:`ClassDef` or :class:`Module` node), or ``None``
134 if it has no docstring. If *clean* is true, clean up the docstring's
135 indentation with :func:`inspect.cleandoc`.
136
137
138.. function:: fix_missing_locations(node)
139
140 When you compile a node tree with :func:`compile`, the compiler expects
141 :attr:`lineno` and :attr:`col_offset` attributes for every node that supports
142 them. This is rather tedious to fill in for generated nodes, so this helper
143 adds these attributes recursively where not already set, by setting them to
144 the values of the parent node. It works recursively starting at *node*.
145
146
147.. function:: increment_lineno(node, n=1)
148
149 Increment the line number of each node in the tree starting at *node* by *n*.
150 This is useful to "move code" to a different location in a file.
151
152
153.. function:: copy_location(new_node, old_node)
154
155 Copy source location (:attr:`lineno` and :attr:`col_offset`) from *old_node*
156 to *new_node* if possible, and return *new_node*.
157
158
159.. function:: iter_fields(node)
160
161 Yield a tuple of ``(fieldname, value)`` for each field in ``node._fields``
162 that is present on *node*.
163
164
165.. function:: iter_child_nodes(node)
166
167 Yield all direct child nodes of *node*, that is, all fields that are nodes
168 and all items of fields that are lists of nodes.
169
170
171.. function:: walk(node)
172
Georg Brandl619e7ba2011-01-09 07:38:51 +0000173 Recursively yield all descendant nodes in the tree starting at *node*
174 (including *node* itself), in no specified order. This is useful if you only
175 want to modify nodes in place and don't care about the context.
Georg Brandl0c77a822008-06-10 16:37:50 +0000176
177
178.. class:: NodeVisitor()
179
180 A node visitor base class that walks the abstract syntax tree and calls a
181 visitor function for every node found. This function may return a value
Georg Brandl36ab1ef2009-01-03 21:17:04 +0000182 which is forwarded by the :meth:`visit` method.
Georg Brandl0c77a822008-06-10 16:37:50 +0000183
184 This class is meant to be subclassed, with the subclass adding visitor
185 methods.
186
187 .. method:: visit(node)
188
189 Visit a node. The default implementation calls the method called
190 :samp:`self.visit_{classname}` where *classname* is the name of the node
191 class, or :meth:`generic_visit` if that method doesn't exist.
192
193 .. method:: generic_visit(node)
194
195 This visitor calls :meth:`visit` on all children of the node.
Georg Brandl48310cd2009-01-03 21:18:54 +0000196
Georg Brandl0c77a822008-06-10 16:37:50 +0000197 Note that child nodes of nodes that have a custom visitor method won't be
198 visited unless the visitor calls :meth:`generic_visit` or visits them
199 itself.
200
201 Don't use the :class:`NodeVisitor` if you want to apply changes to nodes
202 during traversal. For this a special visitor exists
203 (:class:`NodeTransformer`) that allows modifications.
204
205
206.. class:: NodeTransformer()
207
208 A :class:`NodeVisitor` subclass that walks the abstract syntax tree and
209 allows modification of nodes.
210
Georg Brandl36ab1ef2009-01-03 21:17:04 +0000211 The :class:`NodeTransformer` will walk the AST and use the return value of
212 the visitor methods to replace or remove the old node. If the return value
213 of the visitor method is ``None``, the node will be removed from its
214 location, otherwise it is replaced with the return value. The return value
215 may be the original node in which case no replacement takes place.
Georg Brandl0c77a822008-06-10 16:37:50 +0000216
217 Here is an example transformer that rewrites all occurrences of name lookups
218 (``foo``) to ``data['foo']``::
219
220 class RewriteName(NodeTransformer):
221
222 def visit_Name(self, node):
223 return copy_location(Subscript(
224 value=Name(id='data', ctx=Load()),
225 slice=Index(value=Str(s=node.id)),
226 ctx=node.ctx
227 ), node)
228
229 Keep in mind that if the node you're operating on has child nodes you must
230 either transform the child nodes yourself or call the :meth:`generic_visit`
231 method for the node first.
232
233 For nodes that were part of a collection of statements (that applies to all
234 statement nodes), the visitor may also return a list of nodes rather than
235 just a single node.
236
237 Usually you use the transformer like this::
238
239 node = YourTransformer().visit(node)
240
241
242.. function:: dump(node, annotate_fields=True, include_attributes=False)
243
244 Return a formatted dump of the tree in *node*. This is mainly useful for
245 debugging purposes. The returned string will show the names and the values
246 for fields. This makes the code impossible to evaluate, so if evaluation is
247 wanted *annotate_fields* must be set to False. Attributes such as line
Benjamin Petersondcf97b92008-07-02 17:30:14 +0000248 numbers and column offsets are not dumped by default. If this is wanted,
Georg Brandl0c77a822008-06-10 16:37:50 +0000249 *include_attributes* can be set to ``True``.