1 :mod:`ast` --- Abstract Syntax Trees 2 ==================================== 3 4 .. module:: ast 5 :synopsis: Abstract Syntax Tree classes and manipulation. 6 7 .. sectionauthor:: Martin v. Lwis <martin (a] v.loewis.de> 8 .. sectionauthor:: Georg Brandl <georg (a] python.org> 9 10 .. versionadded:: 2.5 11 The low-level ``_ast`` module containing only the node classes. 12 13 .. versionadded:: 2.6 14 The high-level ``ast`` module containing all helpers. 15 16 **Source code:** :source:`Lib/ast.py` 17 18 -------------- 19 20 The :mod:`ast` module helps Python applications to process trees of the Python 21 abstract syntax grammar. The abstract syntax itself might change with each 22 Python release; this module helps to find out programmatically what the current 23 grammar looks like. 24 25 An abstract syntax tree can be generated by passing :data:`ast.PyCF_ONLY_AST` as 26 a flag to the :func:`compile` built-in function, or using the :func:`parse` 27 helper provided in this module. The result will be a tree of objects whose 28 classes all inherit from :class:`ast.AST`. An abstract syntax tree can be 29 compiled into a Python code object using the built-in :func:`compile` function. 30 31 32 Node classes 33 ------------ 34 35 .. class:: AST 36 37 This is the base of all AST node classes. The actual node classes are 38 derived from the :file:`Parser/Python.asdl` file, which is reproduced 39 :ref:`below <abstract-grammar>`. They are defined in the :mod:`_ast` C 40 module and re-exported in :mod:`ast`. 41 42 There is one class defined for each left-hand side symbol in the abstract 43 grammar (for example, :class:`ast.stmt` or :class:`ast.expr`). In addition, 44 there is one class defined for each constructor on the right-hand side; these 45 classes inherit from the classes for the left-hand side trees. For example, 46 :class:`ast.BinOp` inherits from :class:`ast.expr`. For production rules 47 with alternatives (aka "sums"), the left-hand side class is abstract: only 48 instances of specific constructor nodes are ever created. 49 50 .. attribute:: _fields 51 52 Each concrete class has an attribute :attr:`_fields` which gives the names 53 of all child nodes. 54 55 Each instance of a concrete class has one attribute for each child node, 56 of the type as defined in the grammar. For example, :class:`ast.BinOp` 57 instances have an attribute :attr:`left` of type :class:`ast.expr`. 58 59 If these attributes are marked as optional in the grammar (using a 60 question mark), the value might be ``None``. If the attributes can have 61 zero-or-more values (marked with an asterisk), the values are represented 62 as Python lists. All possible attributes must be present and have valid 63 values when compiling an AST with :func:`compile`. 64 65 .. attribute:: lineno 66 col_offset 67 68 Instances of :class:`ast.expr` and :class:`ast.stmt` subclasses have 69 :attr:`lineno` and :attr:`col_offset` attributes. The :attr:`lineno` is 70 the line number of source text (1-indexed so the first line is line 1) and 71 the :attr:`col_offset` is the UTF-8 byte offset of the first token that 72 generated the node. The UTF-8 offset is recorded because the parser uses 73 UTF-8 internally. 74 75 The constructor of a class :class:`ast.T` parses its arguments as follows: 76 77 * If there are positional arguments, there must be as many as there are items 78 in :attr:`T._fields`; they will be assigned as attributes of these names. 79 * If there are keyword arguments, they will set the attributes of the same 80 names to the given values. 81 82 For example, to create and populate an :class:`ast.UnaryOp` node, you could 83 use :: 84 85 node = ast.UnaryOp() 86 node.op = ast.USub() 87 node.operand = ast.Num() 88 node.operand.n = 5 89 node.operand.lineno = 0 90 node.operand.col_offset = 0 91 node.lineno = 0 92 node.col_offset = 0 93 94 or the more compact :: 95 96 node = ast.UnaryOp(ast.USub(), ast.Num(5, lineno=0, col_offset=0), 97 lineno=0, col_offset=0) 98 99 .. versionadded:: 2.6 100 The constructor as explained above was added. In Python 2.5 nodes had 101 to be created by calling the class constructor without arguments and 102 setting the attributes afterwards. 103 104 105 .. _abstract-grammar: 106 107 Abstract Grammar 108 ---------------- 109 110 The module defines a string constant ``__version__`` which is the decimal 111 Subversion revision number of the file shown below. 112 113 The abstract grammar is currently defined as follows: 114 115 .. literalinclude:: ../../Parser/Python.asdl 116 :language: none 117 118 119 :mod:`ast` Helpers 120 ------------------ 121 122 .. versionadded:: 2.6 123 124 Apart from the node classes, :mod:`ast` module defines these utility functions 125 and classes for traversing abstract syntax trees: 126 127 .. function:: parse(source, filename='<unknown>', mode='exec') 128 129 Parse the source into an AST node. Equivalent to ``compile(source, 130 filename, mode, ast.PyCF_ONLY_AST)``. 131 132 133 .. function:: literal_eval(node_or_string) 134 135 Safely evaluate an expression node or a Unicode or *Latin-1* encoded string 136 containing a Python literal or container display. The string or node 137 provided may only consist of the following Python literal structures: 138 strings, numbers, tuples, lists, dicts, booleans, and ``None``. 139 140 This can be used for safely evaluating strings containing Python values from 141 untrusted sources without the need to parse the values oneself. It is not 142 capable of evaluating arbitrarily complex expressions, for example involving 143 operators or indexing. 144 145 146 .. function:: get_docstring(node, clean=True) 147 148 Return the docstring of the given *node* (which must be a 149 :class:`FunctionDef`, :class:`ClassDef` or :class:`Module` node), or ``None`` 150 if it has no docstring. If *clean* is true, clean up the docstring's 151 indentation with :func:`inspect.cleandoc`. 152 153 154 .. function:: fix_missing_locations(node) 155 156 When you compile a node tree with :func:`compile`, the compiler expects 157 :attr:`lineno` and :attr:`col_offset` attributes for every node that supports 158 them. This is rather tedious to fill in for generated nodes, so this helper 159 adds these attributes recursively where not already set, by setting them to 160 the values of the parent node. It works recursively starting at *node*. 161 162 163 .. function:: increment_lineno(node, n=1) 164 165 Increment the line number of each node in the tree starting at *node* by *n*. 166 This is useful to "move code" to a different location in a file. 167 168 169 .. function:: copy_location(new_node, old_node) 170 171 Copy source location (:attr:`lineno` and :attr:`col_offset`) from *old_node* 172 to *new_node* if possible, and return *new_node*. 173 174 175 .. function:: iter_fields(node) 176 177 Yield a tuple of ``(fieldname, value)`` for each field in ``node._fields`` 178 that is present on *node*. 179 180 181 .. function:: iter_child_nodes(node) 182 183 Yield all direct child nodes of *node*, that is, all fields that are nodes 184 and all items of fields that are lists of nodes. 185 186 187 .. function:: walk(node) 188 189 Recursively yield all descendant nodes in the tree starting at *node* 190 (including *node* itself), in no specified order. This is useful if you only 191 want to modify nodes in place and don't care about the context. 192 193 194 .. class:: NodeVisitor() 195 196 A node visitor base class that walks the abstract syntax tree and calls a 197 visitor function for every node found. This function may return a value 198 which is forwarded by the :meth:`visit` method. 199 200 This class is meant to be subclassed, with the subclass adding visitor 201 methods. 202 203 .. method:: visit(node) 204 205 Visit a node. The default implementation calls the method called 206 :samp:`self.visit_{classname}` where *classname* is the name of the node 207 class, or :meth:`generic_visit` if that method doesn't exist. 208 209 .. method:: generic_visit(node) 210 211 This visitor calls :meth:`visit` on all children of the node. 212 213 Note that child nodes of nodes that have a custom visitor method won't be 214 visited unless the visitor calls :meth:`generic_visit` or visits them 215 itself. 216 217 Don't use the :class:`NodeVisitor` if you want to apply changes to nodes 218 during traversal. For this a special visitor exists 219 (:class:`NodeTransformer`) that allows modifications. 220 221 222 .. class:: NodeTransformer() 223 224 A :class:`NodeVisitor` subclass that walks the abstract syntax tree and 225 allows modification of nodes. 226 227 The :class:`NodeTransformer` will walk the AST and use the return value of 228 the visitor methods to replace or remove the old node. If the return value 229 of the visitor method is ``None``, the node will be removed from its 230 location, otherwise it is replaced with the return value. The return value 231 may be the original node in which case no replacement takes place. 232 233 Here is an example transformer that rewrites all occurrences of name lookups 234 (``foo``) to ``data['foo']``:: 235 236 class RewriteName(NodeTransformer): 237 238 def visit_Name(self, node): 239 return copy_location(Subscript( 240 value=Name(id='data', ctx=Load()), 241 slice=Index(value=Str(s=node.id)), 242 ctx=node.ctx 243 ), node) 244 245 Keep in mind that if the node you're operating on has child nodes you must 246 either transform the child nodes yourself or call the :meth:`generic_visit` 247 method for the node first. 248 249 For nodes that were part of a collection of statements (that applies to all 250 statement nodes), the visitor may also return a list of nodes rather than 251 just a single node. 252 253 Usually you use the transformer like this:: 254 255 node = YourTransformer().visit(node) 256 257 258 .. function:: dump(node, annotate_fields=True, include_attributes=False) 259 260 Return a formatted dump of the tree in *node*. This is mainly useful for 261 debugging purposes. The returned string will show the names and the values 262 for fields. This makes the code impossible to evaluate, so if evaluation is 263 wanted *annotate_fields* must be set to ``False``. Attributes such as line 264 numbers and column offsets are not dumped by default. If this is wanted, 265 *include_attributes* can be set to ``True``. 266
