1 :mod:`token` --- Constants used with Python parse trees 2 ======================================================= 3 4 .. module:: token 5 :synopsis: Constants representing terminal nodes of the parse tree. 6 7 .. sectionauthor:: Fred L. Drake, Jr. <fdrake (a] acm.org> 8 9 **Source code:** :source:`Lib/token.py` 10 11 -------------- 12 13 This module provides constants which represent the numeric values of leaf nodes 14 of the parse tree (terminal tokens). Refer to the file :file:`Grammar/Grammar` 15 in the Python distribution for the definitions of the names in the context of 16 the language grammar. The specific numeric values which the names map to may 17 change between Python versions. 18 19 The module also provides a mapping from numeric codes to names and some 20 functions. The functions mirror definitions in the Python C header files. 21 22 23 .. data:: tok_name 24 25 Dictionary mapping the numeric values of the constants defined in this module 26 back to name strings, allowing more human-readable representation of parse trees 27 to be generated. 28 29 30 .. function:: ISTERMINAL(x) 31 32 Return true for terminal token values. 33 34 35 .. function:: ISNONTERMINAL(x) 36 37 Return true for non-terminal token values. 38 39 40 .. function:: ISEOF(x) 41 42 Return true if *x* is the marker indicating the end of input. 43 44 45 The token constants are: 46 47 .. data:: ENDMARKER 48 NAME 49 NUMBER 50 STRING 51 NEWLINE 52 INDENT 53 DEDENT 54 LPAR 55 RPAR 56 LSQB 57 RSQB 58 COLON 59 COMMA 60 SEMI 61 PLUS 62 MINUS 63 STAR 64 SLASH 65 VBAR 66 AMPER 67 LESS 68 GREATER 69 EQUAL 70 DOT 71 PERCENT 72 LBRACE 73 RBRACE 74 EQEQUAL 75 NOTEQUAL 76 LESSEQUAL 77 GREATEREQUAL 78 TILDE 79 CIRCUMFLEX 80 LEFTSHIFT 81 RIGHTSHIFT 82 DOUBLESTAR 83 PLUSEQUAL 84 MINEQUAL 85 STAREQUAL 86 SLASHEQUAL 87 PERCENTEQUAL 88 AMPEREQUAL 89 VBAREQUAL 90 CIRCUMFLEXEQUAL 91 LEFTSHIFTEQUAL 92 RIGHTSHIFTEQUAL 93 DOUBLESTAREQUAL 94 DOUBLESLASH 95 DOUBLESLASHEQUAL 96 AT 97 ATEQUAL 98 RARROW 99 ELLIPSIS 100 OP 101 ERRORTOKEN 102 N_TOKENS 103 NT_OFFSET 104 105 106 The following token type values aren't used by the C tokenizer but are needed for 107 the :mod:`tokenize` module. 108 109 .. data:: COMMENT 110 111 Token value used to indicate a comment. 112 113 114 .. data:: NL 115 116 Token value used to indicate a non-terminating newline. The 117 :data:`NEWLINE` token indicates the end of a logical line of Python code; 118 ``NL`` tokens are generated when a logical line of code is continued over 119 multiple physical lines. 120 121 122 .. data:: ENCODING 123 124 Token value that indicates the encoding used to decode the source bytes 125 into text. The first token returned by :func:`tokenize.tokenize` will 126 always be an ``ENCODING`` token. 127 128 129 .. versionchanged:: 3.5 130 Added :data:`AWAIT` and :data:`ASYNC` tokens. 131 132 .. versionchanged:: 3.7 133 Added :data:`COMMENT`, :data:`NL` and :data:`ENCODING` tokens. 134 135 .. versionchanged:: 3.7 136 Removed :data:`AWAIT` and :data:`ASYNC` tokens. "async" and "await" are 137 now tokenized as :data:`NAME` tokens. 138
