1 package ANTLR::Runtime::TokenSource; 2 3 use Moose::Role; 4 5 # Return a Token object from your input stream (usually a CharStream). 6 # Do not fail/return upon lexing error; keep chewing on the characters 7 # until you get a good one; errors are not passed through to the parser. 8 requires 'next_token'; 9 10 # Where are you getting tokens from? normally the implication will simply 11 # ask lexers input stream. 12 requires 'get_source_name'; 13 14 no Moose::Role; 15 1; 16 __END__ 17 18 =head1 NAME 19 20 ANTLR::Runtime::TokenSource 21 22 =head1 DESCRIPTION 23 24 A source of tokens must provide a sequence of tokens via nextToken() 25 and also must reveal it's source of characters; CommonToken's text is 26 computed from a CharStream; it only store indices into the char stream. 27 28 Errors from the lexer are never passed to the parser. Either you want 29 to keep going or you do not upon token recognition error. If you do not 30 want to continue lexing then you do not want to continue parsing. Just 31 throw an exception not under RecognitionException and Java will naturally 32 toss you all the way out of the recognizers. If you want to continue 33 lexing then you should not throw an exception to the parser--it has already 34 requested a token. Keep lexing until you get a valid one. Just report 35 errors and keep going, looking for a valid token. 36
