Expand description
Macro expansion: a shadow token stream, with every token knowing where it really came from.
§Why the result is not written back into the tree
A compiler expands and then parses the result. This does not, and the reason is the losslessness invariant: the syntax tree is the file, byte for byte, and replacing a macro call with its expansion would delete the call site from the tree. So expansion produces a second stream that sits beside the tree, and the tree is never touched.
§Why every token carries an origin
Once tokens come from two places, “where is this?” stops being answerable from the token. A token of
int z = MAX(1, 2);is written in the file, while the tokens it expands to were written in the #define — possibly in
another file, possibly years ago. Three features depend on telling them apart, and each of them is
wrong without it:
- Go to definition in an expansion should reach the macro, not the definition of whatever the expansion mentions.
- Diagnostics inside an expansion belong at the call site, which is the only place the reader can act. A note can point at the macro body.
- Renaming and refactoring must not rewrite a macro body when the user asked about one call site, and vice versa.
§The three operations, and why they are not string manipulation
- Substitution replaces a parameter with the tokens of an argument, so an argument containing a comma stays one argument.
- Stringizing (
#x) turns those tokens back into spelling, because that is what it means. - Pasting (
a ## b) joins two tokens and re-lexes the result:+pasted with=is+=, one token, and>pasted with>is>>. A text substitution that concatenated the spelling and stopped would leave two tokens where the language has one.
§Termination
Three separate guards, because each catches a different mistake: a depth limit (runaway nesting), a
per-macro hide set (the standard’s rule, which stops #define A A), and a total work budget (a
pathological input that is neither). All three decline to expand rather than failing, and say which
one fired — see Diagnostic.
Structs§
- Diagnostic
- A note, and where it happened.
- Expanded
Token - A token in an expanded stream.
- Expansion
- The result of expanding a token stream.
- Macro
Invocation - One macro call that was expanded.
Enums§
- Expansion
Note - Why the expander declined to expand something.
- Origin
- Where a token in an expanded stream came from.
Constants§
- MAX_
DEPTH - How deep expansions may nest before the expander stops.
- MAX_
TOKENS - How many tokens one expansion may produce in total.
Functions§
- could_
be_ a_ macro_ name - Is this token a name that could be a macro, as opposed to punctuation or a literal?
- expand
- Expand the macros in a token stream.
- expand_
with_ budget - Expand with no limit on the number of tokens, for callers that pass their own bound.