Translation phases 1 to 3, pp-tokens, and the fast scanner.

Design: spec/05-preprocessor.md sections 5.1 and 5.2, and spec/06-lexer-and-parser.md section 6.1 for what happens to these tokens next. Layer rank 4, see spec/18-package-layout.md.

This is the hottest loop in the compiler at -O0, and it is also the place where being clever costs correctness, so the two shapes it takes are worth stating plainly.

Phases 1 and 2 are resolved lazily by the cursor, never by rewriting the buffer. A span is always a range of real bytes in the file the user wrote, even when the token's spelling is not those bytes read in order, and a token that crossed a splice or a trigraph says so through [TokenFlags::SPLICED].

Phase 3 is a loop over a 256-entry dispatch table, and identifiers are interned during the scan rather than in a second pass, so nothing after this crate ever compares identifier text.

use rucc_base::Interner;
use rucc_lex::{Options, PpTokenKind, tokenize};

let mut interner = Interner::new();
let (tokens, diagnostics) = tokenize(b"int x = 1;", 0, Options::new(), &mut interner);
assert!(diagnostics.is_empty());
assert_eq!(tokens[0].kind, PpTokenKind::Ident);
assert_eq!(interner.resolve(tokens[0].value.unwrap()), "int");

Status

Phases 1 to 3 are real, along with the pp-token model, the dispatch table and interning during the scan. Memory mapped input and the SIMD skips for whitespace and comment bodies are the remaining performance items in M1 and are tracked on the milestone issue. Phases 4 to 6, which is directives and macro expansion, belong to rucc-pp.

Every crate in the workspace is published, and publishing implies a promise. This one is tier 3: its Rust API is explicitly unstable and will change without a major version bump. Depend on the rucc binary's behaviour, not on this.