Crate rucc_lex 

Translation phases 1 to 3, pp-tokens, the fast scanner, the keyword table and the constants.

Design: spec/05-preprocessor.md sections 5.1 and 5.2, and spec/06-lexer-and-parser.md section 6.1 for what happens to these tokens next. Layer rank 4, see spec/18-package-layout.md.

This is the hottest loop in the compiler at -O0, and it is also the place where being clever costs correctness, so the two shapes it takes are worth stating plainly.

Phases 1 and 2 are resolved lazily by the cursor, never by rewriting the buffer. A span is always a range of real bytes in the file the user wrote, even when the token’s spelling is not those bytes read in order, and a token that crossed a splice or a trigraph says so through TokenFlags::SPLICED.

Phase 3 is a loop over a 256-entry dispatch table, and identifiers are interned during the scan rather than in a second pass, so nothing after this crate ever compares identifier text. Whitespace and comment bodies, which are most of the bytes and none of the meaning, are skipped a word at a time rather than a byte at a time.

Keywords is the first half of phase 7 and the reason the interner is here rather than in the parser. The keyword spellings are interned before any source is read, so they are one run of symbols at the bottom of the table and recognising one is a subtraction and a bounds check. Which of them the dialect actually has is resolved once, when the table is built, rather than at every identifier.

integer and floating are the next piece of it. A preprocessing number is deliberately looser than a constant, so nothing before this point has asked what 0x1p+3 or 1.2.3 means. The type an integer constant ends up with is a table walk whose candidate list depends on the base, the suffix and the dialect, and the value is accumulated in a hundred and twenty eight bits with every step checked, so a constant too large for any type is a diagnostic rather than a number nobody wrote. A floating constant takes its type from its suffix, of which there are many more than the standard’s three, and its value from the correctly rounded software conversion in rucc-base, so that the bits do not depend on the machine the compiler is running on.

character and string finish the spellings. What an element of a literal is depends on the encoding prefix and, for a wide one, on the target, so a wide string is UTF-16 on Windows and UTF-32 everywhere else and is not even the same length in both. The escapes divide into the ones that name a character, which get encoded, and the ones that write a value, which do not and are truncated to the element instead.

convert is the end of it. It walks a stream of pp-tokens and produces Tokens: an identifier becomes a keyword when the dialect has that spelling, a number becomes a typed value, a run of adjacent string literals becomes the one literal it is, and a stray byte becomes the error it always was. A Token is sixteen bytes like a pp-token, so the values do not live in it; they live in vectors beside it and the token holds an index.

use rucc_base::Interner;
use rucc_lex::{Options, PpTokenKind, tokenize};

let mut interner = Interner::new();
let (tokens, diagnostics) = tokenize(b"int x = 1;", 0, Options::new(), &mut interner);
assert!(diagnostics.is_empty());
assert_eq!(tokens[0].kind, PpTokenKind::Ident);
assert_eq!(interner.resolve(tokens[0].value.unwrap()), "int");

Status

Phases 1 to 3 are real, along with the pp-token model, the dispatch table, interning during the scan, and the word at a time skips for whitespace and comment bodies. The bytes arrive as a memory mapping when the file is large enough for that to be worth it, which the driver decides and nothing here can tell. Phases 4 to 6, which is directives and macro expansion, belong to rucc-pp.

Phase 7 is here too, all of it: the keywords and the dialect gate, the numeric constants, the literals with their escapes and encoding prefixes, the concatenation of adjacent literals, and convert, which turns a stream of pp-tokens into the Tokens the parser reads and is where a remark from a conversion becomes a diagnostic. Decimal floating constants are recognised and refused, because nothing in the compiler has a decimal floating value to put one in, and \N{NAME} is refused because GCC 13.3 only has it in C++. A universal character name above the end of Unicode is an error here and a warning in GCC, which is the one place this crate follows clang instead.

Every crate in the workspace is published, and publishing implies a promise. This one is tier 3: its Rust API is explicitly unstable and will change without a major version bump. Depend on the rucc binary’s behaviour, not on this.

Structs

CharConstant

A converted character constant.

Convert

Everything phase 7 needs that is not the tokens.

FloatConstant

A converted floating constant.

IntConstant

A converted integer constant.

Keywords

The keywords of one dialect, ready to be looked up by symbol.

Lexer

A scanner over one file.

Options

The dialect knobs phase 1 cares about.

PpToken

One preprocessing token.

Remarks

What a constant does that the dialect being compiled has an opinion about, or that happened to it on the way to a value.

StringLiteral

A converted string literal.

Token

One token, as the parser reads it.

TokenFlags

Things about a token that its own bytes do not say.

Tokens

The tokens of a translation unit, with the values they refer to.

Enums

Encoding

The encoding prefix of a character constant or a string literal.

FloatConstantType

The type of a floating constant.

FloatError

Why a preprocessing number is not a floating constant.

IntConstantType

The type of an integer constant.

IntError

Why a preprocessing number is not an integer constant.

Keyword

A keyword, meaning a spelling the grammar knows rather than a name a program chose.

LiteralError

Why a spelling is not a literal.

PpTokenKind

What a preprocessing token is.

Punct

A punctuator.

TokenKind

What a token is.

Constants

MILESTONE

The milestone in spec/17-milestones.md that fills this crate in.

Functions

character

Converts the spelling of a character constant into a value.

convert

Converts a stream of preprocessing tokens into tokens.

floating

Converts the spelling of a preprocessing number into a floating constant.

integer

Converts the spelling of a preprocessing number into an integer constant.

string

Converts the spelling of a string literal into its elements.

strings

Converts a run of adjacent string literals into the one literal they are.

tokenize

Scans src to the end and returns every preprocessing token and every complaint.