1 :mod:`token` --- Constants used with Python parse trees 2 ======================================================= 3 4 .. module:: token 5 :synopsis: Constants representing terminal nodes of the parse tree. 6 7 .. sectionauthor:: Fred L. Drake, Jr. <fdrake (a] acm.org> 8 9 **Source code:** :source:`Lib/token.py` 10 11 -------------- 12 13 This module provides constants which represent the numeric values of leaf nodes 14 of the parse tree (terminal tokens). Refer to the file :file:`Grammar/Grammar` 15 in the Python distribution for the definitions of the names in the context of 16 the language grammar. The specific numeric values which the names map to may 17 change between Python versions. 18 19 The module also provides a mapping from numeric codes to names and some 20 functions. The functions mirror definitions in the Python C header files. 21 22 23 .. data:: tok_name 24 25 Dictionary mapping the numeric values of the constants defined in this module 26 back to name strings, allowing more human-readable representation of parse trees 27 to be generated. 28 29 30 .. function:: ISTERMINAL(x) 31 32 Return true for terminal token values. 33 34 35 .. function:: ISNONTERMINAL(x) 36 37 Return true for non-terminal token values. 38 39 40 .. function:: ISEOF(x) 41 42 Return true if *x* is the marker indicating the end of input. 43 44 45 The token constants are: 46 47 .. data:: ENDMARKER 48 NAME 49 NUMBER 50 STRING 51 NEWLINE 52 INDENT 53 DEDENT 54 LPAR 55 RPAR 56 LSQB 57 RSQB 58 COLON 59 COMMA 60 SEMI 61 PLUS 62 MINUS 63 STAR 64 SLASH 65 VBAR 66 AMPER 67 LESS 68 GREATER 69 EQUAL 70 DOT 71 PERCENT 72 LBRACE 73 RBRACE 74 EQEQUAL 75 NOTEQUAL 76 LESSEQUAL 77 GREATEREQUAL 78 TILDE 79 CIRCUMFLEX 80 LEFTSHIFT 81 RIGHTSHIFT 82 DOUBLESTAR 83 PLUSEQUAL 84 MINEQUAL 85 STAREQUAL 86 SLASHEQUAL 87 PERCENTEQUAL 88 AMPEREQUAL 89 VBAREQUAL 90 CIRCUMFLEXEQUAL 91 LEFTSHIFTEQUAL 92 RIGHTSHIFTEQUAL 93 DOUBLESTAREQUAL 94 DOUBLESLASH 95 DOUBLESLASHEQUAL 96 AT 97 ATEQUAL 98 RARROW 99 ELLIPSIS 100 OP 101 AWAIT 102 ASYNC 103 ERRORTOKEN 104 N_TOKENS 105 NT_OFFSET 106 107 .. versionchanged:: 3.5 108 Added :data:`AWAIT` and :data:`ASYNC` tokens. Starting with 109 Python 3.7, "async" and "await" will be tokenized as :data:`NAME` 110 tokens, and :data:`AWAIT` and :data:`ASYNC` will be removed. 111
