intern-lang 0.3.0

Installation

[dependencies]
intern-lang = "0.3"

Quick start

use intern_lang::Interner;

let mut interner = Interner::new();

// Intern returns a small Copy handle; the same string always returns the same one.
let while_kw = interner.intern("while");
let for_kw = interner.intern("for");
assert_eq!(interner.intern("while"), while_kw);

// Name comparisons are now integer comparisons, not byte walks.
assert_ne!(while_kw, for_kw);

// Resolve borrows the original bytes straight out of the store.
assert_eq!(interner.resolve(while_kw), Some("while"));

Folding a token stream down to symbols, so later passes compare integers and a name in an AST node costs four bytes instead of an owned String:

use intern_lang::{Interner, Symbol};

let mut interner = Interner::new();
let source = ["let", "x", "=", "x", "+", "x"];
let tokens: Vec<Symbol> = source.iter().map(|t| interner.intern(t)).collect();

// The three `x` occurrences collapsed to one symbol.
assert_eq!(tokens[1], tokens[3]);
assert_eq!(interner.len(), 4); // let, x, =, +

Features

• Four-byte handle. A Symbol is a NonZeroU32 newtype — Copy, integer equality / ordering / hashing, and Option<Symbol> is four bytes too.
• Bytes stored once. Interned strings are appended end to end in a single contiguous buffer; the dedup index stores symbol ids, not a second copy.
• Allocation-free hits. Interning a string already seen is a hash lookup with no allocation and no copy.
• Growth-stable symbols. A symbol keeps resolving to the same string for the interner's whole lifetime, including after the backing store reallocates.
• Thread-safe variant. ConcurrentInterner lets many threads intern into one shared symbol space; the warm read path runs concurrently and racing threads never mint a duplicate symbol. Both interners share the Lookup read trait.
• no_std. Relies only on alloc; the default std feature is additive. No runtime dependencies beyond an optional serde (planned).
• #![forbid(unsafe_code)]. The contiguous store is implemented without any unsafe.

Performance

Resolution is a side-table index plus a slice — measured at roughly 1.1 ns per resolve (Criterion mean, Windows x86_64, Rust stable, on the development machine). Interning an already-seen string is allocation-free: a hash over the bytes, an open-addressing probe, and one byte comparison to confirm the hit. The numbers below are a v0.x baseline, tracked over time rather than advertised as final:

Run them yourself with cargo bench. The warm read path scales across threads: at 8 threads the ConcurrentInterner sustains roughly 4× the single-thread intern throughput, since hits are served under a shared read lock.

Concurrent interning

use std::sync::Arc;
use std::thread;

use intern_lang::ConcurrentInterner;

let interner = Arc::new(ConcurrentInterner::new());

let handles: Vec<_> = (0..4)
    .map(|_| {
        let interner = Arc::clone(&interner);
        thread::spawn(move || interner.intern("shared"))
    })
    .collect();

let symbols: Vec<_> = handles.into_iter().map(|h| h.join().unwrap()).collect();

// Every thread agrees on one symbol; "shared" was interned exactly once.
assert!(symbols.iter().all(|&s| s == symbols[0]));
assert_eq!(interner.len(), 1);

API overview

For the complete reference with examples, see docs/API.md.

• Symbol — four-byte Copy handle to an interned string.
• Interner — single-threaded interner: intern, get, resolve, len, with_capacity.
• ConcurrentInterner — thread-safe interner sharing one symbol space (requires the std feature).
• Lookup — read-side trait both interners implement.

Status

v0.3.0 ships the core interner, the symbol, and the thread-safe ConcurrentInterner. Optional serde for Symbol and the defined symbol-space-exhaustion result land across the remaining 0.x series per the ROADMAP; the public surface freezes at 1.0.0.

Contributing

See dev/DIRECTIVES.md for engineering standards and the definition of done. Before a PR: cargo fmt --all, cargo clippy --all-targets --all-features -- -D warnings, and cargo test --all-features must be clean.