Lines Matching full:lexer
521 either ``Lexer::MeasureTokenLength()`` or ``Lexer::getLocForEndOfToken()``. For
547 The Lexer and Preprocessor Library
550 The Lexer library contains several tightly-connected classes that are involved
559 preprocessor is capable of reading from: a buffer lexer (provided by the
560 :ref:`Lexer <Lexer>` class) and a buffered token stream (provided by the
569 intended to be used by the lexer/preprocess and parser libraries, but are not
580 normal tokens. Normal tokens are those returned by the lexer, annotation
595 identifier lookup was enabled when the token was lexed (e.g., the lexer was
601 lexer. This includes things like ``tok::starequal`` (for the "``*=``"
612 lexer/preprocessor system on a per-token basis:
632 Additionally, the lexer library has no notion of typedef names vs variable
716 The ``Lexer`` class
719 The ``Lexer`` class provides the mechanics of lexing tokens out of a source
720 buffer and deciding what they mean. The ``Lexer`` is complicated by the fact
726 The lexer has a couple of interesting modal features:
728 * The lexer can operate in "raw" mode. This mode has several features that
732 * The lexer can capture and return comments as tokens. This is required to
735 * The lexer can be in ``ParsingFilename`` mode, which happens when
742 * The ``Lexer`` uses a ``LangOptions`` object to know whether trigraphs are
745 In addition to these modes, the lexer keeps track of a couple of other features
748 * The ``Lexer`` uses ``BufferPtr`` to keep track of the current character being
750 * The ``Lexer`` uses ``IsAtStartOfLine`` to keep track of whether the next
752 * The ``Lexer`` keeps track of the current "``#if``" directives that are active
754 * The ``Lexer`` keeps track of an :ref:`MultipleIncludeOpt
