Skip to main content

Crate ramvamp_core

Crate ramvamp_core 

Source
Expand description

ramvamp-core: a runtime for streaming fine-grained MoE experts from NVMe.

The design target is a 26-30B-parameter fine-grained MoE model (128 experts per layer, top-8 routing, ~3-4B active parameters per token) running in a ~3 GB memory budget. The always-needed “common core” (embeddings, attention, routers, norms, any shared expert) stays memory-mapped; routed experts live on disk in a page-aligned packed layout and are read on demand with explicit parallel I/O into a small per-layer LFU cache.

Design provenance: the streaming architecture follows the measured results published by TurboFieldfare (https://github.com/drumih/turbo-fieldfare), most importantly: explicit reads beat mmap demand paging (3.54x per cold expert read, ~8x end to end in their full-token simulator); a per-layer expert cache roughly halves expert I/O; cross-layer expert prediction does not work (~7% accuracy), so there is no speculative prefetch; and I/O overlaps only with compute that is guaranteed to run (cache-hit experts, shared expert when the model has one).

Two things that follow from our own measurements rather than theirs. The eviction policy is LFU over frequency counters indexed by expert id whose counts survive eviction, which is where the entire win over LRU comes from (EXP-005: without ghost history, per-slot LFU is worth -1.7 to 0.0 points against LRU). And the cache dial is specified as a total byte budget rather than a slot count, so that one config stays meaningful across models: 1,438.6 MiB of expert pool, which is 11 slots/layer on Qwen3-30B-A3B (EXP-012). That last one is design intent, not current fact - SlotPool::new and LayerCache::new take slot counts today, and wave 2 is where the configured quantity becomes bytes.

Modules§

format
The ramvamp packed model format.
generate
Generation orchestration: prefill, token-by-token decode, sampling.
io
Expert streaming and caching.
kernels
Compute kernels behind a backend trait.
kv
KV cache.
model
Architecture configuration and the per-layer forward pass.
threads
Compute pool and CPU topology.
tokenizer
Tokenizer and chat-template rendering for installed .rvmp models.