1 :mod:`token` --- Constants used with Python parse trees 2 ======================================================= 3 4 .. module:: token 5 :synopsis: Constants representing terminal nodes of the parse tree. 6 .. sectionauthor:: Fred L. Drake, Jr. <fdrake (a] acm.org> 7 8 **Source code:** :source:`Lib/token.py` 9 10 -------------- 11 12 This module provides constants which represent the numeric values of leaf nodes 13 of the parse tree (terminal tokens). Refer to the file :file:`Grammar/Grammar` 14 in the Python distribution for the definitions of the names in the context of 15 the language grammar. The specific numeric values which the names map to may 16 change between Python versions. 17 18 The module also provides a mapping from numeric codes to names and some 19 functions. The functions mirror definitions in the Python C header files. 20 21 22 .. data:: tok_name 23 24 Dictionary mapping the numeric values of the constants defined in this module 25 back to name strings, allowing more human-readable representation of parse trees 26 to be generated. 27 28 29 .. function:: ISTERMINAL(x) 30 31 Return true for terminal token values. 32 33 34 .. function:: ISNONTERMINAL(x) 35 36 Return true for non-terminal token values. 37 38 39 .. function:: ISEOF(x) 40 41 Return true if *x* is the marker indicating the end of input. 42 43 44 The token constants are: 45 46 .. data:: ENDMARKER 47 NAME 48 NUMBER 49 STRING 50 NEWLINE 51 INDENT 52 DEDENT 53 LPAR 54 RPAR 55 LSQB 56 RSQB 57 COLON 58 COMMA 59 SEMI 60 PLUS 61 MINUS 62 STAR 63 SLASH 64 VBAR 65 AMPER 66 LESS 67 GREATER 68 EQUAL 69 DOT 70 PERCENT 71 BACKQUOTE 72 LBRACE 73 RBRACE 74 EQEQUAL 75 NOTEQUAL 76 LESSEQUAL 77 GREATEREQUAL 78 TILDE 79 CIRCUMFLEX 80 LEFTSHIFT 81 RIGHTSHIFT 82 DOUBLESTAR 83 PLUSEQUAL 84 MINEQUAL 85 STAREQUAL 86 SLASHEQUAL 87 PERCENTEQUAL 88 AMPEREQUAL 89 VBAREQUAL 90 CIRCUMFLEXEQUAL 91 LEFTSHIFTEQUAL 92 RIGHTSHIFTEQUAL 93 DOUBLESTAREQUAL 94 DOUBLESLASH 95 DOUBLESLASHEQUAL 96 AT 97 OP 98 ERRORTOKEN 99 N_TOKENS 100 NT_OFFSET 101 102 103 .. seealso:: 104 105 Module :mod:`parser` 106 The second example for the :mod:`parser` module shows how to use the 107 :mod:`symbol` module. 108 109
