Crate rucc_lex 

Translation phases 1 to 3, pp-tokens, and the fast scanner.

Design: spec/05-preprocessor.md sections 5.1 and 5.2, and spec/06-lexer-and-parser.md section 6.1 for what happens to these tokens next. Layer rank 4, see spec/18-package-layout.md.

This is the hottest loop in the compiler at -O0, and it is also the place where being clever costs correctness, so the two shapes it takes are worth stating plainly.

Phases 1 and 2 are resolved lazily by the cursor, never by rewriting the buffer. A span is always a range of real bytes in the file the user wrote, even when the token’s spelling is not those bytes read in order, and a token that crossed a splice or a trigraph says so through TokenFlags::SPLICED.

Phase 3 is a loop over a 256-entry dispatch table, and identifiers are interned during the scan rather than in a second pass, so nothing after this crate ever compares identifier text. Whitespace and comment bodies, which are most of the bytes and none of the meaning, are skipped a word at a time rather than a byte at a time.

use rucc_base::Interner;
use rucc_lex::{Options, PpTokenKind, tokenize};

let mut interner = Interner::new();
let (tokens, diagnostics) = tokenize(b"int x = 1;", 0, Options::new(), &mut interner);
assert!(diagnostics.is_empty());
assert_eq!(tokens[0].kind, PpTokenKind::Ident);
assert_eq!(interner.resolve(tokens[0].value.unwrap()), "int");

Status

Phases 1 to 3 are real, along with the pp-token model, the dispatch table, interning during the scan, and the word at a time skips for whitespace and comment bodies. The bytes arrive as a memory mapping when the file is large enough for that to be worth it, which the driver decides and nothing here can tell. Phases 4 to 6, which is directives and macro expansion, belong to rucc-pp.

Every crate in the workspace is published, and publishing implies a promise. This one is tier 3: its Rust API is explicitly unstable and will change without a major version bump. Depend on the rucc binary’s behaviour, not on this.

Structs

Lexer

A scanner over one file.

Options

The dialect knobs phase 1 cares about.

PpToken

One preprocessing token.

TokenFlags

Things about a token that its own bytes do not say.

Enums

PpTokenKind

What a preprocessing token is.

Punct

A punctuator.

Constants

MILESTONE

The milestone in spec/17-milestones.md that fills this crate in.

Functions

tokenize

Scans src to the end and returns every preprocessing token and every complaint.