Lines Matching refs:tokens
77 holds newly constructed tokens for lexer rules
534 # rule invocation, the parser pushes the set of tokens that can
537 # enclosing rule. This local follow set only includes tokens
545 # thing to do is to consume tokens until you see something that
547 # You don't want the exact set of viable next tokens because the
549 # rest of the input looking for one of the missing tokens.
561 # At each rule invocation, the set of tokens that could follow
586 # all context-sensitive FOLLOW sets--the set of all tokens that
588 # resync to one of those tokens. Note that FOLLOW(c)='^' and if
677 # over the tokens returned for missing tokens. Mostly,
679 # tokens. For literals such as '{' and ',', the default
682 # If you change what tokens must be created by the lexer,
683 # override this method to create the appropriate tokens.
726 # viable tokens that can possibly come next (look depth 1)
734 # FOLLOW(r) is the set of all tokens that can possibly follow
738 # than covering superset) set of following tokens.
917 matching patterns in the text input and breaking the input into tokens.
925 of tokens. Sometimes lexers are enough to carry out a goal on their own, such as
927 the lexer converts text into tokens for use by a parser, which recognizes larger
948 If the lexer is preparing tokens for use by an ANTLR generated parser, the lexer
976 tokens = ANTLR3::CommonTokenStream.new( lexer )
977 parser = Hypothetical::Parser.new( tokens )
1237 Parsers are set up by providing a stream of tokens, which is usually created by
1240 It iterates through the tokens, verifying the syntax of the sentence and
