Lines Matching refs:token
9 /** Track the set of token types that can follow any rule invocation.
18 * matched a token. Prevents generation of more than one error message
25 * but no token is consumed during recovery...another error is found,
27 * one token/tree node is consumed for two errors.
31 /** In lieu of a return value, this indicates that a rule or token
32 * has failed to match. Reset to false upon valid token match.
45 * the stop token index for each rule. ruleMemo[ruleIndex] is
47 * get back the stop token for associated rule or MEMO_RULE_FAILED.
58 /** The goal of all lexer rules/methods is to create a token object.
60 * create a single token. nextToken will return this object after
61 * matching lexer rule(s). If you subclass to allow multiple token
62 * emissions, then set this to the last token to be matched or
63 * something nonnull so that the auto token emit mechanism will not
64 * emit another token.
66 this.token = null;
68 /** What character index in the stream did the current token start at?
69 * Needed, for example, to get the text for current token. Set at
74 /** The line on which the first character of the token resides */
80 /** The channel number for the current token */
83 /** The token type for the current token */
86 /** You can set the text for the current token to override what is in
