Expand description
The Fast-frontend interface.
A frontend turns a source file into a flat token stream with comments and
whitespace removed, plus the coarse unit boundaries (functions, methods,
impl blocks, closures) used later as clone-report anchors. Lexing is
error-tolerant: a malformed span becomes a Diagnostic and lexing
continues, so one broken construct never discards the rest of a file.
The frontend deliberately stops at lexing. Macros and templates are not expanded; their invocations pass through as ordinary tokens. Normalization (identifier renaming, literal folding) is applied downstream at fragment scope, so the stream here carries each token’s raw lexeme unchanged.
Source positions are recorded for reporting only. They are never used as stable identifiers: fingerprints are built from token kinds and normalized text, never from a line number or token offset.
Structs§
- Diagnostic
- A recoverable Fast-frontend problem. Analysis continues past it.
- Lexed
File - The result of lexing one source file.
- Lexeme
- A shared, immutable lexeme.
- Lexeme
Interner - Deduplicating store of
Lexemes, typically one per lexed file. - Source
Span - A source position span, recorded for reporting only.
- Token
- One lexical token.
- Unit
- A coarse code unit: a token range plus its source span.
Enums§
- Diagnostic
Kind - The kind of a recoverable Fast-frontend problem.
- Literal
Kind - Category of a literal token, used by literal-normalization strategies.
- Token
Kind - The lexical category of a token.
- Unit
Kind - The kind of a coarse code unit used as a clone-report anchor.
Traits§
- Frontend
- A Fast-mode lexer for one language.