Status: v0, and it runs. Qwen3-30B-A3B generates coherent text on CPU inside a 4 GB cgroup with a cold page cache, on a machine in ordinary use, validated against llama.cpp on identical weights. The shipping surface is
ramvamp serve: the OpenAI Chat Completions API on loopback, with streaming SSE, tool calls, and named configuration profiles.
ramvamp is a CPU-first local LLM inference runtime, written in Rust, for fine-grained
Mixture-of-Experts models. It runs a 30B-parameter quantized model on an ordinary Linux x86_64
machine with an NVMe SSD, no GPU and no 32 GB of RAM, by never loading the full checkpoint into
memory. The always-needed common weights (about 1 GiB) are read into memory once; the routed experts, which are most
of the model, live on disk in a page-aligned packed format and are fetched with io_uring and
O_DIRECT only when the router asks for them, through a small per-layer LFU cache.
Qwen3-30B-A3B is the only model. A second one has to be faster or better than it on this
runtime: Gemma 4 26B-A4B was costed against that bar and set aside (docs/roadmap.md, v1 item
1), and the repacker accepts qwen3moe architecture GGUFs and refuses anything else by name.
At a glance
Qwen3-30B-A3B Q4_K_M, cold, inside memory.max=4G with memory.swap.max=0, the default 19
expert slots per layer, on the reference machine (Intel Core Ultra 9 185H, CPU only, Micron
2400 DRAM-less QLC NVMe), with the machine in ordinary use and every run hygiene-clean (EXP-033):
| measured | |
|---|---|
| Decode | 2.78 to 2.81 tok/s at ctx 512 (--max-new 128), 1.96 tok/s at ctx 3,961, on this DRAM-less QLC drive |
| Prefill | about 25 tok/s at ctx 512 (24.48 to 25.27), 18.2 at ctx 3,961 |
| Peak RAM | 3,690 to 3,697 MiB at ctx 512 and 4,008 MiB at ctx 3,961, of a 4,096 MiB ceiling |
| Model on disk | 17.35 GiB, a ~6x memory saving |
| Fidelity | mean full-vocab KL 1.04e-2 vs llama.cpp, top-1 agreement 8/8 (EXP-004) |
Throughput is stated per drive on purpose: decode is I/O-bound and this part sustains about
1.6 GB/s at decode's own read geometry, so no figure appears without its drive. With
--cache-bytes 1440M (11 slots) it fits the earlier 3 GB contract, at 2.2 tok/s decode at ctx 512.
Full table, conditions, the v0 figures and drive detail: docs/benchmarks.md.
Install
|
Puts ramvamp and ramvamp-repack in /usr/local/bin. Pin a tag with
RAMVAMP_VERSION=v0.1.0, move the destination with RAMVAMP_INSTALL_DIR, re-run the same
line to upgrade. Linux x86_64 only, and on anything else it refuses and installs nothing:
io_uring is a Linux interface, the fast kernels are AVX2 plus F16C, and a binary elsewhere
would run on fallback paths that no number here describes.
Then install the model. That is a 17.35 GiB download, so the installer prints this and stops rather than starting it for you, and the path wants to be on an NVMe SSD:
# Download and repack the pinned Qwen3-30B-A3B in one streaming pass.
# Generate. Text streams to stdout; timing and expert stats go to stderr.
Prebuilt tarball. What the one-liner does for you. Every tagged release publishes
ramvamp-linux-x86_64.tar.gz on Releases,
holding both binaries plus the README and both licences.
From source. Needs Rust 1.88 or newer, which a prebuilt binary does not. Nothing else: the tree has no C dependencies.
Requirements, the three I/O modes and the repacker's other subcommands: docs/install.md.
Chat in the terminal
The quickest way to try it, with nothing to wire up:
# Same thing with a live status panel: prefill progress, tokens per second,
# context used, and the expert cache hit rate, updated while it generates.
--tui exists because a long prompt is minutes of prefill before the first character
appears, and silence is indistinguishable from a hang. It needs a terminal on stdout.
Plain chat is the reference behaviour and stays that way.
Serve and configure
That is /v1/chat/completions (streaming SSE and buffered), /v1/models and /health, on
127.0.0.1 with no --host flag, on purpose. It serves one request at a time, reuses the KV cache
across requests by longest common prefix, and refuses fields it cannot honour exactly as typed
400s rather than accepting them quietly. Routes, refusals and positioning: docs/server.md.
--context (default 4096) is the dial that trades conversation length against resident
bytes, and plan prices any value of it without loading the model; a 32K profile projects about
6,695 MiB and needs a bigger budget than 4 GB, which is what named profiles exist to express.
Profiles, --profile / --config / --no-config, the four-layer precedence chain and the three
environment variables: docs/configuration.md.
How it works
- Install repacks, it does not convert. The GGUF is streamed into the
.rvmplayout: acommon.binfor what every token needs, one file per layer holding 128 fixed-stride, page-aligned expert blobs. Nothing is requantized and no shard is materialized in heap memory. - Common weights are read in once. Embeddings, attention, routers, norms and the lm_head
are about 1,023 MiB, touched every token, so they are read into anonymous memory at load,
where a swap-off cgroup cannot reclaim them (EXP-031).
--common-memory mappedmaps them instead. - Experts are read explicitly, never demand-paged. Per layer the router picks 8 of 128. Hits in that layer's LFU cache dispatch to the compute pool immediately; the misses go to io_uring as one batch of O_DIRECT reads, so they bypass the page cache and cost the memory budget nothing, and the hit compute hides their latency.
- The cache is small on purpose. At the default 2,485 MiB budget that is 19 slots per layer out of 128 experts. Worst case is 1,097 MB of expert weights per decode token, and the cache absorbs 67.1% of requests at ctx 512 and 68.3% at ctx 3,961 (EXP-033).
Explicit reads rather than mmap demand paging is a measured choice, and so is having no speculative cross-layer expert prefetch; docs/landscape.md carries both measurements. docs/architecture.md is the design and docs/roadmap.md the plan of record.
Workspace
| Crate | Purpose |
|---|---|
ramvamp-core |
Runtime library: packed format, expert streaming, LFU cache, CPU kernels, KV cache, generation |
ramvamp-repack |
Streaming installer: ranged Hugging Face downloads repacked directly into the .rvmp layout |
ramvamp-server |
OpenAI-compatible HTTP layer: request validation, SSE framing, tool-call extraction, KV prefix reuse |
ramvamp |
CLI: serve, chat, generate, plan |
Development
All four must pass before a push. CONTRIBUTING.md covers the rest, including the measurement rules a performance change has to follow; security issues go through SECURITY.md.
Prior art
Inspired by TurboFieldfare, a Swift + Metal runtime that proved the physics on Apple Silicon: Gemma 4 26B-A4B in about 2 GB at 5.1 to 6.3 tok/s on an 8 GB M2 Air. ramvamp exists because that proof is locked to Macs with M-series chips.
License
Licensed under either of Apache License 2.0 or MIT License at your option. Model weights are not included and remain governed by their own terms.