Module lexer 

Error-tolerant lexer for the C language family.

Whitespace and comments are dropped. Preprocessor directives are dropped whole (through their \ line continuations): Fast mode does not preprocess, so both sides of an #if stay in the stream as ordinary tokens while the directive lines themselves never pollute clone content. Every other lexeme becomes a token carrying its preprocessor-normalized text and a reporting-only source span. Malformed spans (unterminated strings, characters and block comments) are recorded as diagnostics and lexing resumes, so a single broken construct never discards the rest of the file. Macros are not expanded: an invocation’s name and delimiters are ordinary tokens.

The lexer is parameterized by a Dialect, which supplies the keyword set, the operator inventory and the dialect-only literal forms (raw strings, digit separators), so the same machinery lexes both C and C++.

Functions

conditional_paths

Record the preprocessor arm active at each token.

lex

Lex source under dialect into tokens and diagnostics.