Crate plugmem_arena 

Flat byte-pool storage structures for plugmem.

This crate is the storage foundation of the plugmem engine, but it is deliberately generic: nothing here knows about facts, vectors or LLMs. If you need a compact, allocation-frugal, no_std sorted container whose in-memory representation is its serialized form, you can lift it into your own project as-is — see examples/ for self-contained walkthroughs.

Philosophy

1. State is flat bytes. A container is one contiguous byte pool plus a few small metadata arrays. No per-element allocations, no pointers, no Box/Rc graphs. Persisting a container is memcpy; loading it back is bounds-checking the metadata and adopting the bytes.
2. Costs are local and visible. Every operation touches one 4 KiB page (one cache-friendly unit). Worst cases are small, fixed and measured — the optional counters feature exposes deterministic work counters used as CI performance gates.
3. Keys are big-endian. Byte-wise comparison of an encoded key must equal the numeric comparison of its source value, so binary search and ordered iteration work directly on raw bytes. Helpers live in key.

The four structures

Structure	Shape	Typical use
Arena	sorted fixed-size records, sharded 4 KiB pages	primary record store, ordered indexes
BlobHeap	append-only variable-length blobs, dense ids	texts, names, raw vectors
ChunkPool	many small growable lists over 64-byte chunks	posting lists, adjacency lists
Interner	string -> dense u32 (heap + flat hash table)	terms, tags, entity names

Quick start

use plugmem_arena::{Arena, ArenaCfg, ShardMode, Slot, key};

/// A tiny fixed-size record: 4-byte big-endian key + 1-byte payload.
#[derive(Debug, PartialEq)]
struct Rec {
    id: u32,
    level: u8,
}

impl Slot for Rec {
    const SIZE: usize = 5;
    const KEY_LEN: usize = 4;
    fn write(&self, out: &mut [u8]) {
        key::write_u32(out, self.id);
        out[4] = self.level;
    }
    fn read(bytes: &[u8]) -> Self {
        Rec { id: key::read_u32(bytes), level: bytes[4] }
    }
}

let mut arena = Arena::<Rec>::new(ArenaCfg::new(64, ShardMode::Ordered)).unwrap();
arena.insert(&Rec { id: 7, level: 3 }).unwrap();
arena.insert(&Rec { id: 1, level: 9 }).unwrap();

let mut key_buf = [0u8; 4];
key::write_u32(&mut key_buf, 7);
assert_eq!(arena.get(&key_buf), Some(Rec { id: 7, level: 3 }));

// Ordered mode: iteration yields ascending keys across all shards.
let ids: Vec<u32> = arena.iter().map(|r| r.id).collect();
assert_eq!(ids, [1, 7]);

The one unsafe

The single default unsafe in this crate is page allocation without zeroing (Vec::reserve + set_len). It is kept because it was measured, not assumed: on the wasm target (our primary portability target) zeroing freshly grown pages made the allocation path 12x slower (wasmtime, 32k pages: 3889 us zeroed vs 316 us uninit), while on native x86-64 the difference is noise. The safety invariant is simple and local: bytes of a page beyond count * Slot::SIZE are never read — every read is bounded by the per-shard element count, and a slot is fully written before count is incremented. See Arena::ensure_page for the full safety comment. A consequence worth knowing: Arena intentionally implements neither Clone nor PartialEq, because a byte-wise clone or comparison would read those uninitialized tails.

Bounds-check elimination (get_unchecked) was measured on the same harness and rejected: <= 1% on native, slower under wasm (the runtime bounds-checks linear memory anyway). Safe indexing everywhere else.

Feature flags

• std (default) — nothing yet beyond linking std for consumers’ convenience; the crate is fully functional as no_std + alloc.
• counters — deterministic work counters (Counters) on every container: key comparisons, bytes shifted, pages allocated. Zero cost when disabled (the increments compile away).

Modules

key

Big-endian key encoding helpers.

Structs

Arena

A sharded collection of fixed-size records in one contiguous byte pool, sorted by key prefix within each shard.

ArenaCfg

Arena configuration.

BlobHeap

Append-only byte heap addressed by dense BlobIds.

BlobHeapBuilder

The index side of a BlobHeap, built without holding the pool.

BlobHeapCfg

BlobHeap configuration.

BlobId

Handle to one blob in a BlobHeap: a dense index assigned in push order, starting at 0.

ChunkIter

Iterator over one list’s chunks; see ChunkPool::iter.

ChunkPool

A pool of 64-byte chunks backing many independent append-only lists.

ChunkPoolCfg

ChunkPool configuration.

Interner

A deduplicating string store over a BlobHeap and a flat hash table.

Iter

Iterator over all records of an Arena; see Arena::iter for ordering guarantees.

ListHandle

Owner-held handle to one list inside a ChunkPool.

TermId

Handle to one interned string: dense, assigned in first-seen order, starting at 0. Numerically equal to the underlying heap’s BlobId.

Enums

Error

Errors returned by storage operations. No operation in this crate panics on resource exhaustion — capacity problems are always typed errors. (Contract violations — a wrong key length, an out-of-range id — panic, because they are caller bugs, not runtime conditions; each method documents its panics.)

ShardMode

How keys are mapped to shards.

Constants

CHUNK_BYTES

Size of one chunk in bytes: a chain link ([LINK_BYTES]) plus the payload.

CHUNK_PAYLOAD

Payload bytes per chunk (CHUNK_BYTES minus the [LINK_BYTES] link). Also the maximum length of a single pushed value.

PAGE_BYTES

Size of one arena page in bytes.

Traits

Slot

A fixed-size record that can be stored in an Arena.