ramvamp_core/lib.rs
1//! ramvamp-core: a runtime for streaming fine-grained MoE experts from NVMe.
2//!
3//! The design target is a 26-30B-parameter fine-grained MoE model (128 experts
4//! per layer, top-8 routing, ~3-4B active parameters per token) running in a
5//! ~3 GB memory budget. The always-needed "common core" (embeddings, attention,
6//! routers, norms, any shared expert) stays memory-mapped; routed experts live
7//! on disk in a page-aligned packed layout and are read on demand with
8//! explicit parallel I/O into a small per-layer LFU cache.
9//!
10//! Design provenance: the streaming architecture follows the measured results
11//! published by TurboFieldfare (<https://github.com/drumih/turbo-fieldfare>),
12//! most importantly: explicit reads beat mmap demand paging (3.54x per cold
13//! expert read, ~8x end to end in their full-token simulator); a per-layer
14//! expert cache roughly halves expert I/O; cross-layer expert prediction does
15//! not work (~7% accuracy), so there is no speculative prefetch; and I/O
16//! overlaps only with compute that is guaranteed to run (cache-hit experts,
17//! shared expert when the model has one).
18//!
19//! Two things that follow from our own measurements rather than theirs. The
20//! eviction policy is LFU over frequency counters indexed by expert id whose
21//! counts survive eviction, which is where the entire win over LRU comes from
22//! (EXP-005: without ghost history, per-slot LFU is worth -1.7 to 0.0 points
23//! against LRU). And the cache dial is specified as a total byte budget
24//! rather than a slot count, so that one config stays meaningful across
25//! models: 1,438.6 MiB of expert pool, which is 11 slots/layer on
26//! Qwen3-30B-A3B (EXP-012). That last one is **design intent, not current
27//! fact** - `SlotPool::new` and `LayerCache::new` take slot counts today, and
28//! wave 2 is where the configured quantity becomes bytes.
29
30// `chunks_exact_to_as_chunks` is newer than the toolchains some contributors
31// run, hence `unknown_lints`. The flagged loops are hot kernel code, and
32// rewriting them to `as_chunks` is a codegen change that gets measured, not
33// slipped in to quiet a lint.
34#![allow(unknown_lints, clippy::chunks_exact_to_as_chunks)]
35
36pub mod format;
37pub mod generate;
38pub mod io;
39pub mod kernels;
40pub mod kv;
41pub mod model;
42pub mod threads;
43pub mod tokenizer;