Lines Matching defs:Token
37 =begin rdoc ANTLR3::Token
52 originated. Token streams will also provide an index value for the token, which
53 indicates the position of the token relative to other tokens in the stream,
54 starting at zero. For example, the 22nd token pulled from a lexer by
57 == Token as an Interface
59 This library provides a token implementation (see CommonToken). Additionally,
60 you may write your own token class as long as you provide methods that give
61 access to the attributes expected by a token. Even though most of the ANTLR
63 checking, it's a good idea to include this ANTLR3::Token into your customized
64 token class.
68 module Token
72 # the token's associated chunk of text
75 # the integer value associated with the token's type
84 # the integer value of the channel to which the token is assigned
87 # the index of the token with respect to other the other tokens produced during lexing
90 # a reference to the input stream from which the token was extracted
106 # attributes of a token for convenience in quick scripts
108 # @example Match against an integer token type constant
109 # token =~ VARIABLE_NAME => true/false
110 # @example Match against a token type name as a Symbol
111 # token =~ :FLOAT => true/false
112 # @example Match the token text against a Regular Expression
113 # token =~ /^@[a-z_]\w*$/i
114 # @example Compare the token's text to a string
115 # token =~ "class"
171 # Sets the token's channel value to HIDDEN_CHANNEL
218 The base class for the standard implementation of Token. It is implemented as a
224 CommonToken to track token-type names efficiently for debugging, inspection, and
227 subclass named XYZ::Token.
229 Here is the token structure attribute list in order:
244 include Token
262 # allows you to make a copy of a token with a different class
263 def self.from_token( token )
265 token.type, token.channel, token.text ? token.text.clone : nil,
266 token.input, token.start, token.stop, -1, token.line, token.column
287 # End of File / End of Input character and token type
300 iterator methods for token generators. Furthermore, it
302 methods to token generators, like lexers.
314 token = next_token()
315 raise StopIteration if token.nil? || token.type == EOF
316 return token
321 while token = next_token and token.type != EOF
322 yield( token )
340 that need to create token objects This module serves as a mixin that provides
344 including class can create tokens using the token class (which defaults to
345 ANTLR3::CommonToken). Token classes are presumed to have an #initialize method
346 that can be called without any parameters and the token objects are expected to
347 have the standard token attributes (see ANTLR3::Token).
356 self::Token rescue
375 TokenSchemes exist to handle the problem of defining token types as integer
377 dynamically defined modules that map integer values to constants with token-type
387 token's type falls within a range, which is not possible with symbols.
389 The downside of token types being represented as integers is that a developer
394 Since ANTLR requires token type names to follow capital-letter naming
396 the problem of referencing type values by name. Thus, a token type like
398 +VARIABLE+. However, when a recognizer creates tokens, the name of the token's
403 than necessary, as well as making it difficult to change the type of a token
406 TokenSchemes exist as a technique to manage token type referencing and name
409 1. keep token type references clear and understandable in recognizer code
410 2. permit access to a token's type-name independently of recognizer objects
411 3. allow multiple classes to share the same token information
413 == Building Token Schemes
421 1. define a customized token class (more on that below)
444 grammar, ANTLR will add a lexer rule for the literal and give the token a name
476 == Custom Token Classes and Relationship with Tokens
479 and assigned it to the constant name +Token+. This token class will both include
480 and extend the scheme module. Since token schemes define the private instance
481 method <tt>token_name(type)</tt>, instances of the token class are now able to
482 provide their type names. The Token method <tt>name</tt> uses the
487 the token types as named constants, a type-to-name map constant +TOKEN_NAMES+,
489 Token. Thus, when recognizers need to manufacture tokens, instead of using the
490 Token
491 class provided by the token scheme.
493 If you need to use a token class other than CommonToken, you can pass the class
567 # token type has already been defined
572 "new token type definition ``#{ name } = #{ value }'' conflicts " <<
619 "attempted assignment of token type #{ type_value }" <<
648 self::Token
655 const_set( :Token, klass )
