Lines Matching full:token
45 written with the preprocessing token as the fundamental unit; the
61 * Token Spacing:: Spacing and paste avoidance issues.
108 interface for obtaining the next token, `cpp_get_token', takes care of
111 clients of the library can easily spell a given token, such as
115 Lexing a token
118 Lexing of an individual token is handled by `_cpp_lex_direct' and its
126 The job of `_cpp_lex_direct' is simply to lex a token. It is not
133 The lexer places the token it lexes into storage pointed to by the
137 `line' and `col' values of the token just before the location that
140 The lexer does not consider whitespace to be a token in its own
141 right. If whitespace (other than a new line) precedes a token, it sets
142 the `PREV_WHITE' bit in the token's flags. Each token has its `line'
144 of the token. This line number is the line number in the translation
148 The first token on a logical, i.e. unescaped, line has the flag
153 Clients cannot reliably determine this for themselves: the first token
156 token on the line certainly won't have the `BOL' flag set.
162 `in_directive' is set, the lexer returns a `CPP_EOF' token, which is
164 In a directive a `CPP_EOF' token never means end-of-file.
177 `PREV_WHITE' flag of a token if it meets a new line when `parsing_args'
190 This is a good example of the subtlety of getting token spacing
220 anywhere--between the `+' and `=' of the `+=' token, within the
236 `+=' token; it needs to be prepared for an escaped newline of some
255 diagnostic is appropriate. Since we change state on a per-token basis,
259 lexing header names. Normally, a `<' would be lexed as a single token.
261 token as far as the nearest `>' character. Note that we don't allow
267 force. For example, `::' is a single token in C++, but in C it is two
272 Once a token has been lexed, it leads an independent existence. The
274 storage from the original input buffer, so a token remains valid and
289 token stream. For example, after the name of a function-like macro, it
290 wants to check the next token to see if it is an opening parenthesis.
294 pragmas to access the full `#pragma' token stream. The stand-alone
295 preprocessor wants to be able to test the current token with the
298 be sure the pointer to the previous token is still valid. The
300 parsing arbitrarily far ahead in the token stream, and then to be able
304 preprocessor lex all tokens on a line consecutively into a token buffer,
305 which I call a "token run", and when meeting an unescaped new line
321 the token run?
326 expanded by chaining a new token run on to the end of the existing one.
335 stringification and token pasting. I handled this by allocating space
336 for these tokens from the lexer's token run chain. This means they
340 Lexing into a line of tokens solves some of the token memory
344 current line. So cpplib only moves back to the start of the token run
351 share full control over the lifetime of token pointers too.
353 The routine `_cpp_lex_token' handles moving to new token runs,
355 previously-lexed tokens if we stepped back in the token stream. It also
356 checks each token for the `BOL' flag, which might indicate a directive
360 multiple-include optimization if a token was successfully lexed outside
408 token matters they are spelt differently. This spelling
415 token, after lexing, contains a pointer to its hash node, this is used
426 File: cppinternals.info, Node: Macro Expansion, Next: Token Spacing, Prev: Hash Nodes, Up: Top
438 of how things like nested macro expansion, stringification and token
447 contiguously in memory, so a pointer to the first one and a token count
454 special token of type `CPP_MACRO_ARG'. Each such token holds the index
475 contiguous list of tokens delimited by a starting and ending token.
476 When not in base context, cpplib obtains the next token from the list
494 Although these macros expand to a single token which cannot contain any
495 further macros, for reasons of token spacing (*note Token Spacing::)
497 by pushing a context containing just that one token.
521 later scan. This occurs when the identifier is the last token of an
524 parameter in the macro's replacement list, the subsequent token happens
525 to be an opening parenthesis (itself possibly the first token of an
528 It is important to note that when cpplib reads the last token of a
530 looking for the _next_ token do we pop it off the stack and drop to a
531 lower context. This makes backing up by one token easy, but more
533 is still disabled when we are considering the last token of its
544 the macro invocation]. This still leaves the argument token `foo'
546 replacement, the token `foo' is rejected for expansion, and marked
562 read the next token. Unfortunately, because of spacing issues (*note
563 Token Spacing::), there can be fake padding tokens in-between, and if
564 the next real token is not a parenthesis cpplib needs to be able to
565 back up that one token as well as retain the information in any
568 Backing up more than one token when macros are involved is not
573 as it reads tokens. If the next real token is not an opening
574 parenthesis, it backs up that one token, and then pushes an extra
582 have been lexed, it instead makes a copy of the token and adds the flag
587 from the lexer's current token run (*note Lexing a line::) using the
593 list, and cpplib only wants to back-up more than one lexer token in
598 File: cppinternals.info, Node: Token Spacing, Next: Line Numbering, Prev: Macro Expansion, Up: Top
600 Token Spacing
605 preprocessed output results in an identical token stream. Without
624 shown by the `EMPTY' example) from the original lexed token stream, we
625 need to check for accidental token pasting. We call this "paste
626 avoidance". Token addition and removal can only occur because of macro
629 additionally each token created by the `#' and `##' operators.
634 indicates that the token was preceded by whitespace of some form other
656 token, which I call a
657 "padding token", into the token stream to indicate that spacing of the
658 subsequent token is special. The preprocessor inserts padding tokens
660 point to a "source token" from which the subsequent real token should
666 example if a macro's first replacement token expands straight into
674 Here, two padding tokens are generated with sources the `foo' token
675 between the brackets, and the `bar' token from foo's replacement list,
676 respectively. Clearly the first padding token is the one to use, so
677 the output code should contain a rule that the first padding token in a
693 tokens, one per macro invocation, before the token `baz'. We would
695 source token `foo' with no leading space.
703 cpplib insert a padding token with a `NULL' source token when leaving
707 rule so that, if we see a padding token with a `NULL' source token,
708 _and_ that source token has no leading space, then we behave as if we
726 File: cppinternals.info, Node: Line Numbering, Next: Guard Macros, Prev: Token Spacing, Up: Top
735 the line number of a token passed to it:
749 * If the token results from a macro expansion, the line of the macro
755 the token. Consequently, but maybe unexpectedly, a token from the
756 replacement list of a macro expansion carries the location of the token
761 token. This is a because they are allocated from the lexer's token
766 recently _lexed_ token, unless they are passed a specific line and
769 original location in the macro definition that the token came from.
770 Since that is exactly the information each token carries, such an
774 the correct line to output the token on: the position attached to a
775 token is fairly useless if the token came from a macro expansion. All
777 the token's reported location is also wrong if it is part of a physical
781 whenever it lexes a preprocessing token that starts a new logical line
782 other than a directive. It passes this token (which may be a `CPP_EOF'
783 token indicating the end of the translation unit) to the callback
784 routine, which can then use the line and column of this token to
790 As mentioned above, cpplib stores with each token the line number that
874 When about to return a token that is not part of a directive,
902 at `EOF' without returning a token, if the `#endif' directive was not
1015 * paste avoidance: Token Spacing. (line 6)
1016 * spacing: Token Spacing. (line 6)
1017 * token run: Lexer. (line 192)
1018 * token spacing: Token Spacing. (line 6)
1030 Node: Token Spacing30082
