Lines Matching defs:Token
37 # basic token interface
41 class Token(object):
42 """@brief Abstract token baseclass."""
45 """@brief Get the text of the token.
52 """@brief Set the text of the token.
60 """@brief Get the type of the token.
67 """@brief Get the type of the token.
75 """@brief Get the line number on which this token was matched
84 """@brief Set the line number on which this token was matched
109 """@brief Get the channel of the token
116 """@brief Set the channel of the token
126 An index from 0..n-1 of the token object in the input stream.
142 """@brief From what character stream was this token created.
144 You don't have to implement but it's nice to know where a Token
150 """@brief From what character stream was this token created.
152 You don't have to implement but it's nice to know where a Token
160 # token implementations
162 # Token
168 class CommonToken(Token):
169 """@brief Basic token implementation.
179 Token.__init__(self)
200 #What token number is this from 0..n-1 tokens; < 0 implies invalid index
208 # The char position into the input buffer where this token starts
211 # The char position into the input buffer where this token stops
231 Override the text for this token. getText() will return this text
234 was converted to a new string in the token object.
312 class ClassicToken(Token):
313 """@brief Alternative token implementation.
315 A Token object like we'd use in ANTLR 2.x; has an actual string created
317 tree nodes that have payload objects. We need to create a Token object
318 that has a string; the tree node will point at this token. CommonToken
326 Token.__init__(self)
416 # In an action, a lexer rule can set token to this SKIP_TOKEN and ANTLR
417 # will avoid creating a token for this symbol and try to fetch another.
