Expand description
String interning.
Identifiers are compared constantly: on every macro lookup, every scope lookup, every
typedef disambiguation. Interning turns those comparisons into an integer compare and
turns the storage into one arena instead of a String per occurrence. The lexer interns
during the scan rather than after it, per spec/06-lexer-and-parser.md, so an identifier
is never materialised as a String at all.
§Reserved names
Every interner starts with the names in RESERVED already in it, in that order, which is
what makes the constants in sym the symbols they are. The reason is that a pass past the
lexer holds the interner through a shared reference and cannot add to it, and a pass that
builds a type of its own still has to name it: the members of the target’s va_list are
named by the ABI and never by the source, so the names have to exist before anything is read.
§Determinism
Symbol ordering is allocation order, which is the order the source was read in. That
is deterministic for a given input, and it is the reason the compiler can sort by symbol
anywhere it needs a stable order without reaching for the string. Hashing a Symbol must
never leak into output ordering, because hash order is not stable across runs, and
spec/02-the-goal.md makes byte-identical output a requirement rather than a nicety.
§Spellings that are not text
A source file is UTF-8 and an identifier in it is text, but the body of a string literal is
bytes and does not have to be text at all: "\xff" may be written as the byte itself, and
the object it initialises is one byte long whatever that byte is. So Interner::intern_bytes
takes a spelling that is not UTF-8 and Interner::resolve_bytes gives it back exactly,
while Interner::resolve still hands back a &str, because almost everything that holds a
symbol wants to print it. What it hands back for such a symbol is the lossy reading, with the
bytes that are not characters replaced, which is right for a message and wrong for an object,
and the object is what resolve_bytes is for.
Modules§
Structs§
- Interner
- An append-only set of strings, each mapped to a
Symbol. - Symbol
- An interned string.
- Symbol
Table - Marker for the symbol table, so that
Idx<SymbolTable>cannot be confused with any other index.
Constants§
- RESERVED
- The names every interner is built with, in the order they are interned.