Lines Matching refs:Tokens
276 /// A source of tokens must provide a sequence of tokens via NextToken()
307 /// Where are you getting tokens from? normally the implication will simply
313 /// <summary>A stream of tokens accessing tokens from a TokenSource </summary>
325 /// I < 0 indicates tokens in the past. So -1 is previous token and -2 is
326 /// two tokens ago. LT(0) is undefined. For I>=N, return Token.EOFToken.
335 /// If you don't want to buffer up tokens, then this method makes no
342 /// <summary>Return the text of all tokens from start to stop, inclusive.
343 /// If the stream does not buffer all the tokens then it can just
819 /// <summary>Consume tokens until one matches the given token set </summary>
852 function ToStrings(const Tokens: IList<IToken>): IList<String>;
938 /// The most common stream of tokens is one where every token is buffered up
939 /// and tokens are prefiltered for a certain channel (the parser will only
940 /// see these tokens and cannot change the filter channel number during the
943 /// TODO: how to access the full token stream? How to track all tokens matched per rule?
950 /// to force all tokens of type TType to be on Channel.
968 /// <summary>Given a start and stop index, return a List of all tokens in
969 /// the token type BitSet. Return null if no tokens were found. This
970 /// method looks at both on and off channel tokens.
1003 /// As a point of reference, the Tokens rule DFA for the lexer in the Java grammar
1077 /// NextToken (to push tokens into a list and pull from that list rather
1201 /// to get the .text property anyway if the rule matches tokens from multiple
1219 /// all the time. As the buffer of tokens is converted to strings, the
1227 /// index I does not change the index values for tokens I+1..N-1.
1238 /// Tokens: ITokenRewriteStream;
1242 /// Tokens := TTokenRewriteStream.Create(Lex);
1243 /// Parser := TParser.Create(tokens);
1252 /// WriteLn(Tokens.ToString());
1262 /// Tokens.InsertAfter('pass1', t, 'text to put after t');
1263 /// Tokens.InsertAfter('pass2', u, 'text after u');
1264 /// WriteLn(Tokens.ToString('pass1'));
1265 /// WriteLn(Tokens.ToString('pass2'));
1848 /// <summary>What token number is this from 0..n-1 tokens; < 0 implies invalid index </summary>
1898 /// <summary>What token number is this from 0..n-1 tokens </summary>
1942 /// All tokens go to the parser (unless skip() is called in that rule)
2054 /// is in the set of tokens that can follow the ')' token
2073 /// over the tokens returned for missing tokens. Mostly,
2075 /// tokens. For literals such as '{' and ',', the default
2078 /// If you change what tokens must be created by the lexer,
2079 /// override this method to create the appropriate tokens.
2094 /// viable tokens that can possibly come next (lookahead depth 1)
2102 /// FOLLOW(r) is the set of all tokens that can possibly follow
2106 /// than covering superset) set of following tokens.
2146 * rule invocation, the parser pushes the set of tokens that can
2149 * enclosing rule. This local follow set only includes tokens
2157 * thing to do is to consume tokens until you see something that
2159 * You don't want the exact set of viable next tokens because the
2161 * rest of the input looking for one of the missing tokens.
2173 * At each rule invocation, the set of tokens that could follow
2198 * all context-sensitive FOLLOW sets--the set of all tokens that
2200 * resync to one of those tokens. Note that FOLLOW(c)='^' and if
2277 function ToStrings(const Tokens: IList<IToken>): IList<String>; virtual;
2305 /// <summary><![CDATA[Map<tokentype, channel>]]> to override some Tokens' channel numbers </summary>
2308 /// <summary><![CDATA[Set<tokentype>;]]> discard any tokens with this type </summary>
2311 /// <summary>Skip tokens on any channel but this one; this is how we skip whitespace... </summary>
2314 /// <summary>By default, track all incoming tokens </summary>
2321 /// The index into the tokens list of the current token (next token
2322 /// to consume). p==-1 indicates that the tokens list is empty
2326 /// <summary>Load all tokens from the token source and put in tokens.
2332 /// <summary>Look backwards k tokens on-channel tokens </summary>
2765 /// R.x-y.u = replace x-y indexed tokens with u
2786 /// add tokens in front of a method body '{' and then delete the method
4710 function TBaseRecognizer.ToStrings(const Tokens: IList<IToken>): IList<String>;
4714 if (Tokens = nil) then
4719 for Token in Tokens do
4916 // find k good tokens looking backwards
4919 // skip off-channel tokens
4948 // find k good tokens
4951 // skip off-channel tokens
5164 C := Char(Input.LA(1)); // -1 == \uFFFF, all tokens fit in 65000 space
6207 Tokens: IList<IToken>;
6213 Tokens := GetTokens;
6215 StopIndex := Min(Stop,Tokens.Count - 1);
6230 // Walk buffer, executing instructions and emitting tokens
6232 while (I <= StopIndex) and (I < Tokens.Count) do
6237 T := Tokens[I];
6251 if (StopIndex = Tokens.Count - 1) then
6257 if (Pair.Value.Index >= Tokens.Count - 1) then
