Crate rucc_lex 

Translation phases 1 to 3, pp-tokens, the fast scanner, the keyword table and the constants.

Design: spec/05-preprocessor.md sections 5.1 and 5.2, and spec/06-lexer-and-parser.md section 6.1 for what happens to these tokens next. Layer rank 4, see spec/18-package-layout.md.

This is the hottest loop in the compiler at -O0, and it is also the place where being clever costs correctness, so the two shapes it takes are worth stating plainly.

Phases 1 and 2 are resolved lazily by the cursor, never by rewriting the buffer. A span is always a range of real bytes in the file the user wrote, even when the token’s spelling is not those bytes read in order, and a token that crossed a splice or a trigraph says so through TokenFlags::SPLICED.

Phase 3 is a loop over a 256-entry dispatch table, and identifiers are interned during the scan rather than in a second pass, so nothing after this crate ever compares identifier text. Whitespace and comment bodies, which are most of the bytes and none of the meaning, are skipped a word at a time rather than a byte at a time.

Keywords is the first half of phase 7 and the reason the interner is here rather than in the parser. The keyword spellings are interned before any source is read, so they are one run of symbols at the bottom of the table and recognising one is a subtraction and a bounds check. Which of them the dialect actually has is resolved once, when the table is built, rather than at every identifier.

integer is the next piece of it. A preprocessing number is deliberately looser than a constant, so nothing before this point has asked what 0x1p+3 or 1.2.3 means, and the type a constant ends up with is a table walk whose candidate list depends on the base, the suffix and the dialect. The value is accumulated in a hundred and twenty eight bits with every step checked, so a constant too large for any type is a diagnostic rather than a number nobody wrote.

use rucc_base::Interner;
use rucc_lex::{Options, PpTokenKind, tokenize};

let mut interner = Interner::new();
let (tokens, diagnostics) = tokenize(b"int x = 1;", 0, Options::new(), &mut interner);
assert!(diagnostics.is_empty());
assert_eq!(tokens[0].kind, PpTokenKind::Ident);
assert_eq!(interner.resolve(tokens[0].value.unwrap()), "int");

Status

Phases 1 to 3 are real, along with the pp-token model, the dispatch table, interning during the scan, and the word at a time skips for whitespace and comment bodies. The bytes arrive as a memory mapping when the file is large enough for that to be worth it, which the driver decides and nothing here can tell. Phases 4 to 6, which is directives and macro expansion, belong to rucc-pp.

Of phase 7, the keywords, the dialect gate and the integer constants are here. The floating constants, the escape sequences and the encoding prefixes are not yet, and neither is the Token the parser will read.

Every crate in the workspace is published, and publishing implies a promise. This one is tier 3: its Rust API is explicitly unstable and will change without a major version bump. Depend on the rucc binary’s behaviour, not on this.

Structs

IntConstant

A converted integer constant.

Keywords

The keywords of one dialect, ready to be looked up by symbol.

Lexer

A scanner over one file.

Options

The dialect knobs phase 1 cares about.

PpToken

One preprocessing token.

Remarks

What a constant does that the dialect being compiled has an opinion about.

TokenFlags

Things about a token that its own bytes do not say.

Enums

IntConstantType

The type of an integer constant.

IntError

Why a preprocessing number is not an integer constant.

Keyword

A keyword, meaning a spelling the grammar knows rather than a name a program chose.

PpTokenKind

What a preprocessing token is.

Punct

A punctuator.

Constants

MILESTONE

The milestone in spec/17-milestones.md that fills this crate in.

Functions

integer

Converts the spelling of a preprocessing number into an integer constant.

tokenize

Scans src to the end and returns every preprocessing token and every complaint.