ramvamp-server 0.1.1

OpenAI-compatible HTTP server for the ramvamp runtime: request/response shapes, SSE streaming, prompt assembly, prefix-cached generation, and error mapping
Documentation

Status: v0, and it runs. Qwen3-30B-A3B generates coherent text on CPU inside a 4 GB cgroup with a cold page cache, on a machine in ordinary use, validated against llama.cpp on identical weights. The shipping surface is ramvamp serve: the OpenAI Chat Completions API on loopback, with streaming SSE, tool calls, and named configuration profiles.

ramvamp is a CPU-first local LLM inference runtime, written in Rust, for fine-grained Mixture-of-Experts models. It runs a 30B-parameter quantized model on an ordinary Linux x86_64 machine with an NVMe SSD, no GPU and no 32 GB of RAM, by never loading the full checkpoint into memory. The always-needed common weights (about 1 GiB) are read into memory once; the routed experts, which are most of the model, live on disk in a page-aligned packed format and are fetched with io_uring and O_DIRECT only when the router asks for them, through a small per-layer LFU cache. Qwen3-30B-A3B is the only model. A second one has to be faster or better than it on this runtime: Gemma 4 26B-A4B was costed against that bar and set aside (docs/roadmap.md, v1 item 1), and the repacker accepts qwen3moe architecture GGUFs and refuses anything else by name.

At a glance

Qwen3-30B-A3B Q4_K_M, cold, inside memory.max=4G with memory.swap.max=0, the default 19 expert slots per layer, on the reference machine (Intel Core Ultra 9 185H, CPU only, Micron 2400 DRAM-less QLC NVMe), with the machine in ordinary use and every run hygiene-clean (EXP-033):

measured
Decode 2.78 to 2.81 tok/s at ctx 512 (--max-new 128), 1.96 tok/s at ctx 3,961, on this DRAM-less QLC drive
Prefill about 25 tok/s at ctx 512 (24.48 to 25.27), 18.2 at ctx 3,961
Peak RAM 3,690 to 3,697 MiB at ctx 512 and 4,008 MiB at ctx 3,961, of a 4,096 MiB ceiling
Model on disk 17.35 GiB, a ~6x memory saving
Fidelity mean full-vocab KL 1.04e-2 vs llama.cpp, top-1 agreement 8/8 (EXP-004)

Throughput is stated per drive on purpose: decode is I/O-bound and this part sustains about 1.6 GB/s at decode's own read geometry, so no figure appears without its drive. With --cache-bytes 1440M (11 slots) it fits the earlier 3 GB contract, at 2.2 tok/s decode at ctx 512. Full table, conditions, the v0 figures and drive detail: docs/benchmarks.md.

Install

curl -sSL https://y0sif.github.io/ramvamp/install.sh | bash

Puts ramvamp and ramvamp-repack in /usr/local/bin. Pin a tag with RAMVAMP_VERSION=v0.1.0, move the destination with RAMVAMP_INSTALL_DIR, re-run the same line to upgrade. Linux x86_64 only, and on anything else it refuses and installs nothing: io_uring is a Linux interface, the fast kernels are AVX2 plus F16C, and a binary elsewhere would run on fallback paths that no number here describes.

Then install the model. That is a 17.35 GiB download, so the installer prints this and stops rather than starting it for you, and the path wants to be on an NVMe SSD:

# Download and repack the pinned Qwen3-30B-A3B in one streaming pass.
ramvamp-repack install --output ~/models/qwen3-30b-a3b.rvmp

# Generate. Text streams to stdout; timing and expert stats go to stderr.
ramvamp generate --model ~/models/qwen3-30b-a3b.rvmp \
    --prompt "Explain io_uring in two sentences."

Prebuilt tarball. What the one-liner does for you. Every tagged release publishes ramvamp-linux-x86_64.tar.gz on Releases, holding both binaries plus the README and both licences.

curl -sSL -o ramvamp.tar.gz \
  https://github.com/y0sif/ramvamp/releases/latest/download/ramvamp-linux-x86_64.tar.gz
tar xzf ramvamp.tar.gz
sudo install -m755 ramvamp ramvamp-repack /usr/local/bin/

From source. Needs Rust 1.88 or newer, which a prebuilt binary does not. Nothing else: the tree has no C dependencies.

git clone https://github.com/y0sif/ramvamp
cd ramvamp
cargo build --release

Requirements, the three I/O modes and the repacker's other subcommands: docs/install.md.

Chat in the terminal

The quickest way to try it, with nothing to wire up:

ramvamp chat --model ~/models/qwen3-30b-a3b.rvmp

# Same thing with a live status panel: prefill progress, tokens per second,
# context used, and the expert cache hit rate, updated while it generates.
ramvamp chat --model ~/models/qwen3-30b-a3b.rvmp --tui

--tui exists because a long prompt is minutes of prefill before the first character appears, and silence is indistinguishable from a hang. It needs a terminal on stdout. Plain chat is the reference behaviour and stays that way.

Serve and configure

ramvamp serve --model ~/models/qwen3-30b-a3b.rvmp --port 8080

That is /v1/chat/completions (streaming SSE and buffered), /v1/models and /health, on 127.0.0.1 with no --host flag, on purpose. It serves one request at a time, reuses the KV cache across requests by longest common prefix, and refuses fields it cannot honour exactly as typed 400s rather than accepting them quietly. Routes, refusals and positioning: docs/server.md.

--context (default 4096) is the dial that trades conversation length against resident bytes, and plan prices any value of it without loading the model; a 32K profile projects about 6,695 MiB and needs a bigger budget than 4 GB, which is what named profiles exist to express. Profiles, --profile / --config / --no-config, the four-layer precedence chain and the three environment variables: docs/configuration.md.

How it works

  1. Install repacks, it does not convert. The GGUF is streamed into the .rvmp layout: a common.bin for what every token needs, one file per layer holding 128 fixed-stride, page-aligned expert blobs. Nothing is requantized and no shard is materialized in heap memory.
  2. Common weights are read in once. Embeddings, attention, routers, norms and the lm_head are about 1,023 MiB, touched every token, so they are read into anonymous memory at load, where a swap-off cgroup cannot reclaim them (EXP-031). --common-memory mapped maps them instead.
  3. Experts are read explicitly, never demand-paged. Per layer the router picks 8 of 128. Hits in that layer's LFU cache dispatch to the compute pool immediately; the misses go to io_uring as one batch of O_DIRECT reads, so they bypass the page cache and cost the memory budget nothing, and the hit compute hides their latency.
  4. The cache is small on purpose. At the default 2,485 MiB budget that is 19 slots per layer out of 128 experts. Worst case is 1,097 MB of expert weights per decode token, and the cache absorbs 67.1% of requests at ctx 512 and 68.3% at ctx 3,961 (EXP-033).

Explicit reads rather than mmap demand paging is a measured choice, and so is having no speculative cross-layer expert prefetch; docs/landscape.md carries both measurements. docs/architecture.md is the design and docs/roadmap.md the plan of record.

Workspace

Crate Purpose
ramvamp-core Runtime library: packed format, expert streaming, LFU cache, CPU kernels, KV cache, generation
ramvamp-repack Streaming installer: ranged Hugging Face downloads repacked directly into the .rvmp layout
ramvamp-server OpenAI-compatible HTTP layer: request validation, SSE framing, tool-call extraction, KV prefix reuse
ramvamp CLI: serve, chat, generate, plan

Development

cargo build
cargo test
cargo clippy --all-targets -- -D warnings
cargo fmt --check

All four must pass before a push. CONTRIBUTING.md covers the rest, including the measurement rules a performance change has to follow; security issues go through SECURITY.md.

Prior art

Inspired by TurboFieldfare, a Swift + Metal runtime that proved the physics on Apple Silicon: Gemma 4 26B-A4B in about 2 GB at 5.1 to 6.3 tok/s on an 8 GB M2 Air. ramvamp exists because that proof is locked to Macs with M-series chips.

License

Licensed under either of Apache License 2.0 or MIT License at your option. Model weights are not included and remain governed by their own terms.