Lines Matching full:token
84 * Semantic Values:: Each token or syntactic grouping can have
179 * Token Decl:: Declaring terminal symbols.
203 * Token Values:: How `yylex' must return the semantic value
204 of the token it has read.
205 * Token Locations:: How `yylex' must return the text location
206 (line number, etc.) of the token, if the
213 * Look-Ahead:: Parser looks one token ahead when deciding what to do.
232 * Semantic Tokens:: Token parsing can depend on the semantic context.
233 * Lexical Tie-ins:: Token parsing can depend on the syntactic context.
731 * Semantic Values:: Each token or syntactic grouping can have
768 input string with just a single token of look-ahead. Strictly
793 "terminal symbols" or "token types". We call a piece of input
794 corresponding to a single terminal symbol a "token", and a piece
851 result is that the entire token sequence reduces to a single grouping
871 "token type". Token types as well can be represented as C-like
880 just like a C character constant. You should do this whenever a token
882 same character in a literal as the terminal symbol for that token.
890 semicolon in quotes is a literal character token, representing part of
915 token in a Bison grammar has both a token type and a "semantic value".
918 The token type is a terminal symbol defined in the grammar, such as
920 to decide where the token may validly appear and how to group it with
925 meaning of the token, such as the value of an integer, or the name of an
926 identifier. (A token such as `','' which is just punctuation doesn't
929 For example, an input token might be classified as token type
930 `INTEGER' and have the semantic value 4. Another input token might
931 have the same token type `INTEGER' but value 3989. When a grammar rule
933 because each is an `INTEGER'. When the parser accepts the token, it
934 keeps track of the token's semantic value.
1058 These two declarations look identical until the `..' token. With
1059 normal LALR(1) one-token look-ahead it is not possible to decide
1082 error. If there is a `..' token before the next `;', the rule for
1084 the subrange type rule fails since it requires a `..' token. So one of
1110 %token TYPE DOTDOT ID
1192 %token TYPENAME ID
1318 type of the look-ahead token present at the time of the associated
1321 look-ahead token's semantic value and location, if any. In a
1381 Each token has a semantic value. In a similar fashion, each token
1421 parser calls the lexical analyzer each time it wants a new token. It
1436 Aside from the token type names and the symbols in the actions you
1603 %token NUM
1614 the default. Because we specify `double', each token and each
1626 about the token types (*note The Bison Declarations Section: Bison
1631 terminal symbol that needs to be declared is `NUM', the token type for
1732 The first alternative is a token which is a newline character; this
1815 isn't part of a number is a separate token. Note that the token-code
1816 for such a single-character token is the character itself.
1819 which represents a token
1820 stand for this token type is also a C expression for the numeric code
1821 for the type. This works in two ways. If the token type is a
1824 the number. If the token type is an identifier, that identifier is
1829 The semantic value of the token (if it has one) is stored into the
1835 A token type code of zero is returned if the end-of-input is
1842 number on the stack and the token NUM, or the numeric code
2000 %token NUM
2030 In the second section (Bison declarations), `%left' declares token
2032 `%left' and `%right' (right associativity) take the place of `%token'
2033 which is used to declare a token type name without associativity.
2136 %token NUM
2212 feed the parser with the token locations, as it already does for
2263 `YYLTYPE') containing the token's location.
2265 Now, each time this function returns a token, the parser has its
2348 %token <val> NUM /* Simple double precision number. */
2349 %token <tptr> VAR FNCT /* Variable and Function. */
2375 symbols, just as `%token' is used for declaring token types. We have
2617 /* Any other character is a token by itself. */
2794 A "terminal symbol" (also known as a "token type") represents a
2796 rules to mean that a token in that class is allowed. The symbol is
2798 function returns a token type code to indicate what kind of token has
2811 * A "named token type" is written with an identifier, like an
2814 `%token'. *Note Token Type Names: Token Decl.
2816 * A "character token type" (or "literal character token") is written
2818 constants; for example, `'+'' is a character token type. A
2819 character token type doesn't need to be declared unless you need to
2824 By convention, a character token type is used only to represent a
2825 token that consists of that particular character. Thus, the token
2826 type `'+'' is used to represent the character `+' as a token.
2838 * A "literal string token" is written like a C string constant; for
2839 example, `"<="' is a literal string token. A literal string token
2844 You can associate the literal string token with a symbolic name as
2845 an alias, using the `%token' declaration (*note Token
2846 Declarations: Token Decl.). If you don't do that, the lexical
2847 analyzer has to retrieve the token number for the literal string
2848 token from the `yytname' table (*note Calling Convention::).
2852 By convention, a literal string token is used only to represent a
2853 token that consists of that particular string. Thus, you should
2854 use the token type `"<="' to represent the string `<=' as a token.
2862 backslash-newline allowed. A literal string token must contain
2863 two or more characters; for a token containing just one character,
2864 use a character token (see above).
2872 way you write the token type in the grammar rules, you write it the
2874 character token type is simply the positive numeric code of the
2877 avoid sign-extension on hosts where `char' is signed. Each named token
2884 token-type macro definitions to be available there. Use the `-d'
2911 value of the error token is 256, unless you explicitly assigned 256 to
2912 one of your tokens with a `%token' declaration.
2934 says that two groupings of type `exp', with a `+' token in between, can
3104 tokens with the `%token' Bison declaration (*note Token Type
3105 Names: Token Decl.) and for groupings with the `%type' Bison
3142 connected by a plus-sign token. In the action, `$1' and `$3' refer to
3147 useful semantic value associated with the `+' token, it could be
3183 token, if any, from a semantic action. This semantic value is stored
3316 the open-brace token and look at what follows before deciding whether
3334 information to do it correctly. (The open-brace token is what is called
3335 the "look-ahead" token at this time, since the parser is still deciding
3352 statement by the first token (which is true in C), then one solution
3360 Now the first token of the following declaration or statement, which
3480 It is also possible to access the location of the look-ahead token,
3562 All token type names (but not single-character literal tokens such as
3575 * Token Decl:: Declaring terminal symbols.
3587 File: bison.info, Node: Require Decl, Next: Token Decl, Up: Declarations
3599 File: bison.info, Node: Token Decl, Next: Precedence Decl, Prev: Require Decl, Up: Declarations
3601 3.7.2 Token Type Names
3604 The basic way to declare a token type name (terminal symbol) is as
3607 %token NAME
3611 to stand for this token type's code.
3614 of `%token', if you wish to specify associativity and precedence.
3617 You can explicitly specify the numeric code for a token type by
3619 immediately following the token name:
3621 %token NUM 300
3622 %token XNUM 0x12d // a GNU extension
3625 all token types. Bison will automatically select codes that don't
3629 `%token' or other token declaration to include the data type
3639 %token <val> NUM /* define token NUM and its type */
3641 You can associate a literal string token with a token type name by
3642 writing the literal string at the end of a `%token' declaration which
3645 %token arrow "=>"
3650 %token <operator> OR "||"
3651 %token <operator> LE 134 "<="
3654 Once you equate the literal string and the token name, you can use them
3656 `yylex' function can use the token name or the literal string to obtain
3657 the token type code number (*note Calling Convention::).
3660 File: bison.info, Node: Precedence Decl, Next: Union Decl, Prev: Token Decl, Up: Declarations
3665 Use the `%left', `%right' or `%nonassoc' declaration to declare a token
3671 `%token': either
3679 And indeed any of these declarations serves the purposes of `%token'.
3717 `%token' and `%type' declarations to pick one of the types for a
3758 terminal symbol. All kinds of token declarations allow `<TYPE>'.
3818 %token <string> STRING
3956 -- Directive: %token
3957 Declare a terminal symbol (token type name) with no precedence or
3958 associativity specified (*note Token Type Names: Token Decl.).
3961 Declare a terminal symbol (token type name) that is
3965 Declare a terminal symbol (token type name) that is
3969 Declare a terminal symbol (token type name) that is nonassociative
3995 Write a header file containing macro definitions for the token type
4019 declarations and to the token type codes. *Note Semantic Values
4020 of Tokens: Token Values.
4076 -- Directive: %token-table
4077 Generate an array of token names in the parser file. The name of
4078 the array is `yytname'; `yytname[I]' is the name of the token
4079 whose internal Bison token code number is I. The first three
4085 represent the token in Bison. For single-character literals and
4093 When you specify `%token-table', Bison also generates macro
4098 The highest token number, plus one.
4111 parser states and what is done for each type of look-ahead token in
4251 need to arrange for the token-type macro definitions to be available
4260 * Token Values:: How `yylex' must return the semantic value
4261 of the token it has read.
4262 * Token Locations:: How `yylex' must return the text location
4263 (line number, etc.) of the token, if the
4269 File: bison.info, Node: Calling Convention, Next: Token Values, Up: Lexical
4275 the type of token it has just found; a zero or negative value signifies
4278 When a token is referred to in the grammar rules by a name, that name
4280 numeric code for that token type. So `yylex' can use the name to
4283 When a token is referred to in the grammar rules by a character
4285 token type. So `yylex' can simply return that character code, possibly
4300 return c; /* Assume token type for `+' is '+'. */
4302 return INT; /* Return the type of the token. */
4310 `yylex' can determine the token type codes for them:
4312 * If the grammar defines symbolic token names as aliases for the
4317 * `yylex' can find the multicharacter token in the `yytname' table.
4318 The index of the token in the table is the token type's code. The
4319 name of a multicharacter token is recorded in `yytname' with a
4320 double-quote, the token's characters, and another double-quote.
4321 The token's characters are escaped as necessary to be suitable as
4324 Here's code for looking up a multicharacter token in `yytname',
4325 assuming that the characters of the token are stored in
4326 `token_buffer', and assuming that the token does not contain any
4341 `%token-table' declaration. *Note Decl Summary::.
4344 File: bison.info, Node: Token Values, Next: Token Locations, Prev: Calling Convention, Up: Lexical
4349 In an ordinary (nonreentrant) parser, the semantic value of the token
4356 return INT; /* Return the type of the token. */
4361 Types: Union Decl.). So when you store a token's value, you must use
4375 return INT; /* Return the type of the token. */
4379 File: bison.info, Node: Token Locations, Next: Pure Calling, Prev: Token Values, Up: Lexical
4387 function `yyparse' expects to find the textual location of a token just
4400 File: bison.info, Node: Pure Calling, Prev: Token Locations, Up: Lexical
4417 return INT; /* Return the type of the token. */
4461 reads a token which cannot satisfy any syntax rule. An action in the
4586 -- Macro: YYBACKUP (TOKEN, VALUE);
4587 Unshift a token. This macro is allowed only for rules that reduce
4588 a single value, and only when there is no look-ahead token. It is
4589 also disallowed in GLR parsers. It installs a look-ahead token
4590 with token type TOKEN and semantic value VALUE; then it discards
4594 look-ahead token already, then it reports a syntax error with a
4600 Value stored in `yychar' when there is no look-ahead token.
4620 Variable containing either the look-ahead token, or `YYEOF' when
4622 no look-ahead has been performed so the next token is not yet
4628 Discard the current look-ahead token. This is useful primarily in
4638 Variable containing the look-ahead token location when `yychar' is
4644 Variable containing the look-ahead token semantic value when
4726 token is traditionally called "shifting".
4729 `3' to come. The stack will have four elements, one for each token
4732 But the stack does not always have an element for each token read.
4745 and the next input token is a newline character, then the last three
4755 value 16. Then the newline token can be shifted.
4766 * Look-Ahead:: Parser looks one token ahead when deciding what to do.
4786 token in order to decide what to do.
4788 When a token is read, it is not immediately shifted; first it
4789 becomes the "look-ahead token", which is not on the stack. Now the
4791 the stack, while the look-ahead token remains off to the side. When no
4792 more reductions should take place, the look-ahead token is shifted onto
4794 done; depending on the token type of the look-ahead token, some rules
4811 should be done? If the following token is `)', then the first three
4816 If the following token is `!', then it must be shifted immediately so
4823 The look-ahead token is stored in the variable `yychar'. Its
4845 When the `ELSE' token is read and becomes the look-ahead token, the
4893 %token IF THEN ELSE variable
4943 the next token. Of course, if the next token is `)', we must reduce;
4944 shifting is invalid because no single rule can reduce the token
4945 sequence `- 2 )' or anything starting with that. But if the next token
4950 If the next operator token OP is shifted, then it must be reduced first
4964 token is `-': shifting makes right-associativity.
5025 token. If the token's precedence is higher, the choice is to shift.
5032 rule or the look-ahead token has no precedence, then the default is to
5048 `%nonassoc', can only be used once for a given token; so a token has
5090 The values pushed on the parser stack are not simply token type codes;
5096 Each time a look-ahead token is read, the current parser state
5097 together with the type of look-ahead token are looked up in a table.
5098 This table entry can say, "Shift the look-ahead token." In this case,
5106 look-ahead token is erroneous in the current state. This causes error
5223 %token ID
5246 token of look-ahead: when a `param_spec' is being read, an `ID' is a
5273 %token BOGUS
5288 long as the token `BOGUS' is never generated by `yylex', the added rule
5321 input and on one extra token of look-ahead. As a result, normal Bison
5355 segment of the input token stream.
5459 recognize the special token `error'. This is a terminal symbol that is
5461 handling. The Bison parser generates an `error' token whenever a
5463 token in the current context, the parse can continue.
5486 which the `error' token is acceptable. (This means that the
5488 `stmnts'.) At this point the `error' token can be shifted. Then, if
5489 the old look-ahead token is not acceptable to be shifted next, the
5490 parser reads tokens and discards them until it finds a token which is
5527 Note that rules which accept the `error' token may have actions, just
5535 The previous look-ahead token is reanalyzed immediately after an
5537 to clear this token. Write the statement `yyclearin;' in the error
5544 lexical scanner is probably correct. The previous look-ahead token
5558 syntactic units. In many languages, the meaning of a token is affected
5565 * Semantic Tokens:: Token parsing can depend on the semantic context.
5566 * Lexical Tie-ins:: Token parsing can depend on the syntactic context.
5576 7.1 Semantic Info in Token Types
5588 The method used in GNU C is to have two different token types,
5591 which token type to return: `TYPENAME' if the identifier is declared as
5595 choice of token type to recognize. `IDENTIFIER' is accepted as an
5600 accepted--there is one rule for each of the two token types.
5662 particular, the token `a1b' must be treated as an integer rather than
5792 %token NUM STR
5820 Conflict in state 8 between rule 2 and token '+' resolved as reduce.
5821 Conflict in state 8 between rule 2 and token '-' resolved as reduce.
5822 Conflict in state 8 between rule 2 and token '*' resolved as shift.
5896 symbol, and the look-ahead is a `NUM', then this token is shifted on
5901 report lists `NUM' as a look-ahead token because `NUM' can be at the
5928 the rule 5, `exp: NUM;', is completed. Whatever the look-ahead token
5951 Since there is no default action, any other token than those listed
6032 either shifting the next token and going to the corresponding state, or
6039 token is `*', since we specified that `*' has higher precedence than
6157 * Each time the parser calls `yylex', what kind of token was read.
6159 * Each time a token is shifted, the depth and complete contents of
6169 possible input token. As you read the successive trace messages, you
6181 The debugging information normally gives the token type of each token
6185 standard I/O stream, the numeric code for the token type, and the token
6330 `--token-table'
6331 Pretend that `%token-table' was specified. *Note Decl Summary::.
6338 file containing macro definitions for the token type names defined
6412 `--token-table' `-k'
6643 Return the next token. Its type is the return value, its semantic
6878 The token numbered as 0 corresponds to end of file; the following line
6884 %token END 0 "end of file"
6885 %token ASSIGN ":="
6886 %token <sval> IDENTIFIER "identifier"
6887 %token <ival> NUMBER "number"
6954 #define yyterminate() return token::END
6990 `yy::calcxx_parser::token::identifier' into `token::identifier' for
6994 typedef yy::calcxx_parser::token token;
6998 ":=" return token::ASSIGN;
7005 return token::NUMBER;
7007 {id} yylval->sval = new std::string (yytext); return token::IDENTIFIER;
7129 /* One token only. */
7267 %token START_FOO START_BAR;
7489 The predefined token marking the end of the token stream. It
7493 A token name reserved for error recovery. This token may be used
7497 syntax error, the token `error' becomes the current look-ahead
7498 token. Actions corresponding to `error' are then executed, and
7499 the look-ahead token is reset to the token that originally caused
7519 Bison declaration to assign left associativity to token(s). *Note
7543 Bison declaration to assign nonassociativity to token(s). *Note
7568 Bison declaration to assign right associativity to token(s).
7575 -- Directive: %token
7576 Bison declaration to declare token(s) without specifying
7577 precedence. *Note Token Type Names: Token Decl.
7579 -- Directive: %token-table
7580 Bison declaration to include a token name table in the parser file.
7588 The predefined token onto which all undefined values returned by
7609 look-ahead token. *Note Special Features for Use in Actions:
7614 look-ahead token. (In a pure parser, it is a local variable within
7620 look-ahead token. *Note Error Recovery::.
7658 to get the next token. *Note The Lexical Analyzer Function
7669 numbers associated with a token. (In a pure parser, it is a local
7672 the grammar actions. *Note Textual Locations of Tokens: Token
7674 look-ahead token. *Note Actions and Locations: Actions and
7683 associated with a token. (In a pure parser, it is a local
7685 *Note Semantic Values of Tokens: Token Values. In semantic
7686 actions, it stores the semantic value of the look-ahead token.
7806 Parsing a sentence of a language by analyzing it token by token
7817 Literal string token
7818 A token which consists of two or more fixed characters. *Note
7821 Look-ahead token
7822 A token already read but not yet shifted. *Note Look-Ahead
7831 The class of context-free grammars in which at most one token of
7838 words, a construct that is not a token. *Note Symbols::.
7898 Token
7900 that describes a token in the grammar is a terminal symbol. The
7907 token. *Note Languages and Context-Free Grammars: Language and
8432 * %token <1>: Table of Symbols. (line 156)
8433 * %token <2>: Decl Summary. (line 13)
8434 * %token: Token Decl. (line 6)
8435 * %token-table <1>: Table of Symbols. (line 160)
8436 * %token-table: Decl Summary. (line 133)
8505 * character token: Symbols. (line 31)
8536 * declaring literal string tokens: Token Decl. (line 6)
8539 * declaring token type names: Token Decl. (line 6)
8644 * literal string token: Symbols. (line 53)
8645 * literal token: Symbols. (line 31)
8656 * look-ahead token: Look-Ahead. (line 6)
8750 * string token: Symbols. (line 53)
8766 * token: Language and Grammar.
8768 * token type: Symbols. (line 6)
8769 * token type names, declaring: Token Decl. (line 6)
8770 * token, useless: Understanding. (line 62)
8776 * useless token: Understanding. (line 62)
8832 * yylloc <4>: Token Locations. (line 6)
8840 * YYLTYPE: Token Locations. (line 19)
8844 * yylval <4>: Token Values. (line 6)
8929 Node: Token Decl148626
8944 Node: Token Values177831
8945 Node: Token Locations178995
