Module expert_store 

A bounded, lease-protected byte cache for routed-expert weights – the storage foundation for running MoE checkpoints whose experts do not all fit in RAM at once (stream cold experts from SSD, keep hot ones resident under one global byte budget).

Status: wired into both decode paths as an opt-in (Decoder::from_gguf_with_expert_cache for GGUF, load_kimi_checkpoint_with_expert_cache for Kimi safetensors, both behind the server’s FERROX_EXPERT_CACHE_BYTES or FERROX_SSD_STREAMING=1 which defaults the cache to 2 GiB), each proven bit-identical to its resident path on the committed fixtures at both generous and smaller-than-one-expert budgets. Without the opt-in, experts still load resident (mmap) exactly as before. ExpertStore::prefetch warms keys for ds4-style SSD streaming overlap (caller supplies the hotlist). Never yet exercised against a real large checkpoint. Reference clone: .scratch/ds4 (gitignored).

Design:

• ExpertSource abstracts where bytes come from (a file with positional reads, a shard set, an in-memory test source). The store never caches a partial expert: one (layer, expert) key maps to one complete byte buffer (gate+up+down and any auxiliary tensors, concatenated by the source in a layout the consumer defines).
• ExpertStore::acquire returns an ExpertLease – a cheap Arc handle that pins the entry: eviction skips any entry with an outstanding lease, so a slot can never be reused while CPU (or, later, GPU) work still reads it. This is the slot-reuse-corruption guard, enforced structurally (an Arc with strong_count > 1 is simply not freeable), not by convention.
• When an expert doesn’t fit even after evicting every unleased entry (e.g. a cache configured smaller than one decode step’s expert union), acquire still succeeds: the bytes are read and returned as an uncached pass-through lease. Capacity pressure degrades to more I/O, never to a wrong answer or a deadlock.
• Reads happen outside the store lock, so concurrent misses on different experts overlap their I/O. Two concurrent misses on the same expert may both read it; the first to insert wins and the loser’s buffer becomes that caller’s private pass-through copy – duplicated work under a rare race, never wrong bytes.

Structs

ExpertKey

Stable identity of one routed expert within a checkpoint.

ExpertLease

A pinned handle to one expert’s bytes. While any lease for an entry is alive, the store cannot evict or reuse that entry’s memory.

ExpertStore

The bounded cache. See the module docs for the design contract.

ExpertStoreStats

Monotonic counters, readable at any time without taking the store lock. resident_bytes is a gauge (current cache footprint).

FileRangeSource

A file-backed ExpertSource: each expert is one contiguous (offset, len) byte range in a single file, read with positional reads (pread-style on unix, so concurrent misses never contend on a shared seek cursor). This is the portable buffered-I/O default the storage design starts from; O_DIRECT/F_NOCACHE/io_uring variants are future work behind the same trait and must produce identical bytes.

Traits

ExpertSource

Where expert bytes come from. Implementations must be cheap to call concurrently (positional reads, not a shared seek cursor).