Struct Interner

Source

pub struct Interner<'a> { /* private fields */ }

Expand description

A deduplicating string store over a BlobHeap and a flat hash table.

use plugmem_arena::{BlobHeapCfg, Interner};

let mut terms = Interner::new(BlobHeapCfg::new());
let apple = terms.intern("apple").unwrap();
let banana = terms.intern("banana").unwrap();
assert_eq!(terms.intern("apple").unwrap(), apple); // stable id
assert_ne!(apple, banana);
assert_eq!(terms.resolve(apple), "apple");
assert_eq!(terms.len(), 2);

Clone is cheap-ish (two flat memcpys) and safe: unlike the arena, every stored byte is initialized.

Implementations§

Source §

impl<'a> Interner<'a>

Source

pub fn new(cfg: BlobHeapCfg) -> Self

Creates an empty interner; cfg bounds the underlying string heap (BlobHeapCfg::max_blob caps a single term’s byte length). The hash table itself is small (4 bytes per slot) and not counted against cfg.max_bytes.

Source

pub fn intern(&mut self, s: &str) -> Result<TermId, Error>

Returns the id for s, storing it on first sight.

§Errors

Error::BlobTooLarge if s is longer than the configured BlobHeapCfg::max_blob;
Error::CapacityExceeded if storing a new term would grow the heap past BlobHeapCfg::max_bytes.

A failed intern leaves the interner unchanged.

Source

pub fn lookup(&self, s: &str) -> Option<TermId>

Returns the id of an already-interned string, without creating it.

The read-only sibling of Interner::intern — query paths must not grow the vocabulary (a query is not a mutation).

Source

pub fn resolve(&self, id: TermId) -> &str

Returns the string behind an id. O(1).

§Panics

Panics if id was not returned by this interner’s Interner::intern — a dangling id is a caller bug.

Source

pub fn len(&self) -> usize

Number of distinct strings interned.

Source

pub fn is_empty(&self) -> bool

true when nothing has been interned.

Source

pub fn pool_bytes(&self) -> usize

Total bytes of interned string content (the underlying heap’s pool size; the hash table’s few bytes per slot are not counted).

Source

pub fn dump_index(&self, out: &mut Vec<u8>)

Appends the string heap’s index section to out; see BlobHeap::dump_index.

Source

pub fn dump_pool(&self, out: &mut Vec<u8>)

Appends the string heap’s pool section to out; see BlobHeap::dump_pool.

Source

pub fn dump_table(&self, out: &mut Vec<u8>)

Appends the hash-table section to out.

Layout (little-endian): [slots u32][len u32] then slots × u32 entries (0 = empty, else TermId + 1). The table is persisted rather than rebuilt on load: rebuilding is O(terms × hash) and the cold-start budget is tighter than the 4 bytes per slot.

Source

pub fn load( cfg: BlobHeapCfg, index: &[u8], pool: &[u8], table: &[u8], ) -> Result<Self, Error>

Rebuilds an interner from its three dumped sections.

The input is untrusted — no panic on arbitrary bytes. On top of BlobHeap::load, validation covers:

every stored blob is valid UTF-8 (O(pool bytes), SIMD-speed) — this is what keeps Interner::resolve infallible;
table shape: power-of-two slot count >= 16, exact section length, the ≤ 0.7 load factor, len equal to the heap’s blob count;
table entries: in bounds, no id stored twice, and exactly len of them (checked with a visited bitmap).

Entry placement (that each id sits on its hash’s probe path) is not re-verified — that would cost the full rebuild the stored table exists to avoid. A well-formed but misplaced table cannot cause memory unsafety or a panic; it degrades intern to assigning a duplicate id for the affected terms. Section checksums cover accidental corruption.

§Errors

Error::Corrupt for any inconsistency.

Source

pub fn load_borrowed( cfg: BlobHeapCfg, index: &[u8], pool: &'a [u8], table: &[u8], ) -> Result<Self, Error>

Rebuilds an interner that borrows its string bytes from a longer-lived buffer (a memory-mapped snapshot). The table stays owned; validation is identical to Interner::load. The term dictionary is small (Zipf vocabulary), so paging it in to check UTF-8 is cheap and the large pools (texts, vectors) still load lazily.