Skip to main content

Module expand

Module expand 

Source
Expand description

Macro expansion: a shadow token stream, with every token knowing where it really came from.

§Why the result is not written back into the tree

A compiler expands and then parses the result. This does not, and the reason is the losslessness invariant: the syntax tree is the file, byte for byte, and replacing a macro call with its expansion would delete the call site from the tree. So expansion produces a second stream that sits beside the tree, and the tree is never touched.

§Why every token carries an origin

Once tokens come from two places, “where is this?” stops being answerable from the token. A token of

int z = MAX(1, 2);

is written in the file, while the tokens it expands to were written in the #define — possibly in another file, possibly years ago. Three features depend on telling them apart, and each of them is wrong without it:

  • Go to definition in an expansion should reach the macro, not the definition of whatever the expansion mentions.
  • Diagnostics inside an expansion belong at the call site, which is the only place the reader can act. A note can point at the macro body.
  • Renaming and refactoring must not rewrite a macro body when the user asked about one call site, and vice versa.

§The three operations, and why they are not string manipulation

  • Substitution replaces a parameter with the tokens of an argument, so an argument containing a comma stays one argument.
  • Stringizing (#x) turns those tokens back into spelling, because that is what it means.
  • Pasting (a ## b) joins two tokens and re-lexes the result: + pasted with = is +=, one token, and > pasted with > is >>. A text substitution that concatenated the spelling and stopped would leave two tokens where the language has one.

§Termination

Three separate guards, because each catches a different mistake: a depth limit (runaway nesting), a per-macro hide set (the standard’s rule, which stops #define A A), and a total work budget (a pathological input that is neither). All three decline to expand rather than failing, and say which one fired — see Diagnostic.

Structs§

Diagnostic
A note, and where it happened.
ExpandedToken
A token in an expanded stream.
Expansion
The result of expanding a token stream.
MacroInvocation
One macro call that was expanded.

Enums§

ExpansionNote
Why the expander declined to expand something.
Origin
Where a token in an expanded stream came from.

Constants§

MAX_DEPTH
How deep expansions may nest before the expander stops.
MAX_TOKENS
How many tokens one expansion may produce in total.

Functions§

could_be_a_macro_name
Is this token a name that could be a macro, as opposed to punctuation or a literal?
expand
Expand the macros in a token stream.
expand_with_budget
Expand with no limit on the number of tokens, for callers that pass their own bound.