1 :mod:`ast` --- Abstract Syntax Trees 2 ==================================== 3 4 .. module:: ast 5 :synopsis: Abstract Syntax Tree classes and manipulation. 6 7 .. sectionauthor:: Martin v. Lwis <martin (a] v.loewis.de> 8 .. sectionauthor:: Georg Brandl <georg (a] python.org> 9 10 **Source code:** :source:`Lib/ast.py` 11 12 -------------- 13 14 The :mod:`ast` module helps Python applications to process trees of the Python 15 abstract syntax grammar. The abstract syntax itself might change with each 16 Python release; this module helps to find out programmatically what the current 17 grammar looks like. 18 19 An abstract syntax tree can be generated by passing :data:`ast.PyCF_ONLY_AST` as 20 a flag to the :func:`compile` built-in function, or using the :func:`parse` 21 helper provided in this module. The result will be a tree of objects whose 22 classes all inherit from :class:`ast.AST`. An abstract syntax tree can be 23 compiled into a Python code object using the built-in :func:`compile` function. 24 25 26 Node classes 27 ------------ 28 29 .. class:: AST 30 31 This is the base of all AST node classes. The actual node classes are 32 derived from the :file:`Parser/Python.asdl` file, which is reproduced 33 :ref:`below <abstract-grammar>`. They are defined in the :mod:`_ast` C 34 module and re-exported in :mod:`ast`. 35 36 There is one class defined for each left-hand side symbol in the abstract 37 grammar (for example, :class:`ast.stmt` or :class:`ast.expr`). In addition, 38 there is one class defined for each constructor on the right-hand side; these 39 classes inherit from the classes for the left-hand side trees. For example, 40 :class:`ast.BinOp` inherits from :class:`ast.expr`. For production rules 41 with alternatives (aka "sums"), the left-hand side class is abstract: only 42 instances of specific constructor nodes are ever created. 43 44 .. index:: single: ? (question mark); in AST grammar 45 .. index:: single: * (asterisk); in AST grammar 46 47 .. attribute:: _fields 48 49 Each concrete class has an attribute :attr:`_fields` which gives the names 50 of all child nodes. 51 52 Each instance of a concrete class has one attribute for each child node, 53 of the type as defined in the grammar. For example, :class:`ast.BinOp` 54 instances have an attribute :attr:`left` of type :class:`ast.expr`. 55 56 If these attributes are marked as optional in the grammar (using a 57 question mark), the value might be ``None``. If the attributes can have 58 zero-or-more values (marked with an asterisk), the values are represented 59 as Python lists. All possible attributes must be present and have valid 60 values when compiling an AST with :func:`compile`. 61 62 .. attribute:: lineno 63 col_offset 64 65 Instances of :class:`ast.expr` and :class:`ast.stmt` subclasses have 66 :attr:`lineno` and :attr:`col_offset` attributes. The :attr:`lineno` is 67 the line number of source text (1-indexed so the first line is line 1) and 68 the :attr:`col_offset` is the UTF-8 byte offset of the first token that 69 generated the node. The UTF-8 offset is recorded because the parser uses 70 UTF-8 internally. 71 72 The constructor of a class :class:`ast.T` parses its arguments as follows: 73 74 * If there are positional arguments, there must be as many as there are items 75 in :attr:`T._fields`; they will be assigned as attributes of these names. 76 * If there are keyword arguments, they will set the attributes of the same 77 names to the given values. 78 79 For example, to create and populate an :class:`ast.UnaryOp` node, you could 80 use :: 81 82 node = ast.UnaryOp() 83 node.op = ast.USub() 84 node.operand = ast.Num() 85 node.operand.n = 5 86 node.operand.lineno = 0 87 node.operand.col_offset = 0 88 node.lineno = 0 89 node.col_offset = 0 90 91 or the more compact :: 92 93 node = ast.UnaryOp(ast.USub(), ast.Num(5, lineno=0, col_offset=0), 94 lineno=0, col_offset=0) 95 96 97 .. _abstract-grammar: 98 99 Abstract Grammar 100 ---------------- 101 102 The abstract grammar is currently defined as follows: 103 104 .. literalinclude:: ../../Parser/Python.asdl 105 :language: none 106 107 108 :mod:`ast` Helpers 109 ------------------ 110 111 Apart from the node classes, the :mod:`ast` module defines these utility functions 112 and classes for traversing abstract syntax trees: 113 114 .. function:: parse(source, filename='<unknown>', mode='exec') 115 116 Parse the source into an AST node. Equivalent to ``compile(source, 117 filename, mode, ast.PyCF_ONLY_AST)``. 118 119 .. warning:: 120 It is possible to crash the Python interpreter with a 121 sufficiently large/complex string due to stack depth limitations 122 in Python's AST compiler. 123 124 125 .. function:: literal_eval(node_or_string) 126 127 Safely evaluate an expression node or a string containing a Python literal or 128 container display. The string or node provided may only consist of the 129 following Python literal structures: strings, bytes, numbers, tuples, lists, 130 dicts, sets, booleans, and ``None``. 131 132 This can be used for safely evaluating strings containing Python values from 133 untrusted sources without the need to parse the values oneself. It is not 134 capable of evaluating arbitrarily complex expressions, for example involving 135 operators or indexing. 136 137 .. warning:: 138 It is possible to crash the Python interpreter with a 139 sufficiently large/complex string due to stack depth limitations 140 in Python's AST compiler. 141 142 .. versionchanged:: 3.2 143 Now allows bytes and set literals. 144 145 146 .. function:: get_docstring(node, clean=True) 147 148 Return the docstring of the given *node* (which must be a 149 :class:`FunctionDef`, :class:`AsyncFunctionDef`, :class:`ClassDef`, 150 or :class:`Module` node), or ``None`` if it has no docstring. 151 If *clean* is true, clean up the docstring's indentation with 152 :func:`inspect.cleandoc`. 153 154 .. versionchanged:: 3.5 155 :class:`AsyncFunctionDef` is now supported. 156 157 158 .. function:: fix_missing_locations(node) 159 160 When you compile a node tree with :func:`compile`, the compiler expects 161 :attr:`lineno` and :attr:`col_offset` attributes for every node that supports 162 them. This is rather tedious to fill in for generated nodes, so this helper 163 adds these attributes recursively where not already set, by setting them to 164 the values of the parent node. It works recursively starting at *node*. 165 166 167 .. function:: increment_lineno(node, n=1) 168 169 Increment the line number of each node in the tree starting at *node* by *n*. 170 This is useful to "move code" to a different location in a file. 171 172 173 .. function:: copy_location(new_node, old_node) 174 175 Copy source location (:attr:`lineno` and :attr:`col_offset`) from *old_node* 176 to *new_node* if possible, and return *new_node*. 177 178 179 .. function:: iter_fields(node) 180 181 Yield a tuple of ``(fieldname, value)`` for each field in ``node._fields`` 182 that is present on *node*. 183 184 185 .. function:: iter_child_nodes(node) 186 187 Yield all direct child nodes of *node*, that is, all fields that are nodes 188 and all items of fields that are lists of nodes. 189 190 191 .. function:: walk(node) 192 193 Recursively yield all descendant nodes in the tree starting at *node* 194 (including *node* itself), in no specified order. This is useful if you only 195 want to modify nodes in place and don't care about the context. 196 197 198 .. class:: NodeVisitor() 199 200 A node visitor base class that walks the abstract syntax tree and calls a 201 visitor function for every node found. This function may return a value 202 which is forwarded by the :meth:`visit` method. 203 204 This class is meant to be subclassed, with the subclass adding visitor 205 methods. 206 207 .. method:: visit(node) 208 209 Visit a node. The default implementation calls the method called 210 :samp:`self.visit_{classname}` where *classname* is the name of the node 211 class, or :meth:`generic_visit` if that method doesn't exist. 212 213 .. method:: generic_visit(node) 214 215 This visitor calls :meth:`visit` on all children of the node. 216 217 Note that child nodes of nodes that have a custom visitor method won't be 218 visited unless the visitor calls :meth:`generic_visit` or visits them 219 itself. 220 221 Don't use the :class:`NodeVisitor` if you want to apply changes to nodes 222 during traversal. For this a special visitor exists 223 (:class:`NodeTransformer`) that allows modifications. 224 225 226 .. class:: NodeTransformer() 227 228 A :class:`NodeVisitor` subclass that walks the abstract syntax tree and 229 allows modification of nodes. 230 231 The :class:`NodeTransformer` will walk the AST and use the return value of 232 the visitor methods to replace or remove the old node. If the return value 233 of the visitor method is ``None``, the node will be removed from its 234 location, otherwise it is replaced with the return value. The return value 235 may be the original node in which case no replacement takes place. 236 237 Here is an example transformer that rewrites all occurrences of name lookups 238 (``foo``) to ``data['foo']``:: 239 240 class RewriteName(NodeTransformer): 241 242 def visit_Name(self, node): 243 return copy_location(Subscript( 244 value=Name(id='data', ctx=Load()), 245 slice=Index(value=Str(s=node.id)), 246 ctx=node.ctx 247 ), node) 248 249 Keep in mind that if the node you're operating on has child nodes you must 250 either transform the child nodes yourself or call the :meth:`generic_visit` 251 method for the node first. 252 253 For nodes that were part of a collection of statements (that applies to all 254 statement nodes), the visitor may also return a list of nodes rather than 255 just a single node. 256 257 Usually you use the transformer like this:: 258 259 node = YourTransformer().visit(node) 260 261 262 .. function:: dump(node, annotate_fields=True, include_attributes=False) 263 264 Return a formatted dump of the tree in *node*. This is mainly useful for 265 debugging purposes. The returned string will show the names and the values 266 for fields. This makes the code impossible to evaluate, so if evaluation is 267 wanted *annotate_fields* must be set to ``False``. Attributes such as line 268 numbers and column offsets are not dumped by default. If this is wanted, 269 *include_attributes* can be set to ``True``. 270 271 .. seealso:: 272 273 `Green Tree Snakes <https://greentreesnakes.readthedocs.io/>`_, an external documentation resource, has good 274 details on working with Python ASTs. 275
