joshua
A pure-Rust LLM inference engine — a Rust clone of Cactus.
No C or C++ dependencies. CPU inference runs entirely in safe Rust via candle (HuggingFace's native Rust ML framework) and tokenizers.
Features
| Feature | Details |
|---|---|
| Pure Rust | Zero C/C++ dependencies — cargo build requires only a Rust toolchain |
| mmap loading | The GGUF file is memory-mapped like llama.cpp: weights page in lazily and stay in the OS page cache |
| Huge pages | Optional transparent (MADV_HUGEPAGE) or explicit 2 MiB / 1 GiB (MAP_HUGETLB) backing to cut TLB misses on large models |
| OpenAI-compatible | Drop-in replacement for /v1/chat/completions, /v1/embeddings, /v1/models |
| Streaming | Server-Sent Events (SSE) for token-by-token streaming |
| GGUF support | Llama/Mistral/Mixtral, Gemma 1–3, GLM-4, LFM2, Phi-2, Phi-3, Qwen2, Qwen3, Qwen3-MoE, DeepSeek-V2/V3, DeepSeek-V4, Kimi-K2 |
| Exotic quant dtypes | In-mapping decoders for IQ2_XXS (DeepSeek-V4's 2.0625-bit expert weights) and MXFP4 (Kimi-K3-class), with matmuls that keep the blocks in the mmap instead of materialising f32 |
| Fused SIMD kernels | AVX2 dequant+dot fusion for Q8_0/Q2_K/Q4_K and parallel SIMD quantized matmuls on x86-64 |
| Chat templates | Renders the model's own tokenizer.chat_template from the GGUF (Jinja via pure-Rust minijinja); ChatML fallback |
| Tool calling | OpenAI-compatible tools / tool_calls, parsing Hermes/Qwen, Mistral, and Llama-3 call formats |
| Embeddings | Dense sentence embeddings for llama / qwen2 / qwen3 embedding models, with GGUF pooling metadata |
| KV-cache reuse | Multi-turn requests continue from a warm model pool and prefill only the new suffix; DeepSeek-V2/V3 MLA caches the compressed latent (c_kv + k_pe) instead of the reconstructed per-head K/V, cutting KV memory ~70× |
| GPU (optional) | --features cuda or metal route inference through candle's GPU backends |
| NPU / llama.cpp interop (optional) | Vendor plugins run in a crash-isolated shim process; a llama.cpp adapter brings every ggml backend (Hexagon NPU, CANN, CUDA, Vulkan, …) |
| Vision (optional) | OpenAI-style image messages routed through llama.cpp's mtmd (Qwen2.5-VL, Gemma 3, LLaVA, …) via the same isolated plugin |
| Speech-to-text | Whisper transcription in pure Rust: /v1/audio/transcriptions + joshua transcribe |
| Sampling | Temperature, top-k, min-p, top-p (nucleus), greedy — all in Rust |
| HTTPS (optional) | --features tls terminates TLS in-process via rustls — no reverse proxy needed |
| API-key auth | Optional --api-key guards the /v1 routes with OpenAI-style bearer authentication |
Architecture
┌──────────────────────────┐
│ Joshua (Rust crate) │ ← OpenAI-compatible REST API (axum)
└──────────────────────────┘ Chat completions, embeddings, streaming
│
┌──────────────────────────┐
│ candle (pure Rust) │ ← Tensor operations + quantized GGUF inference
└──────────────────────────┘ Llama / Gemma / GLM-4 / LFM2 / Phi / Qwen loaders
│
┌──────────────────────────┐
│ tokenizers (pure Rust) │ ← BPE tokenisation from tokenizer.json
└──────────────────────────┘ HuggingFace tokenizers library
Requirements
| Tool | Minimum version |
|---|---|
| Rust toolchain | 1.88 |
The minimum is set by the dependency tree — currently zip 8.6.0, which
requires Rust 1.88 (the table previously claimed 1.87 via candle-core's use
of {integer}::is_multiple_of, but the tree has moved on).
No CMake, no C++ compiler, no CUDA toolkit required.
Quick start
1 — Add to Cargo.toml
[]
= { = "https://github.com/rexlunae/joshua" }
2 — Download a model
Any GGUF model with a supported architecture works (see
Supported models below). You also need the
tokenizer.json from the same HuggingFace repository — place it alongside the
.gguf file.
# Using the Hugging Face CLI
# Download GGUF weights + tokenizer into ./weights/
The layout Joshua expects:
weights/
├── gemma-3-1b-it-Q4_K_M.gguf ← quantised weights
└── tokenizer.json ← HuggingFace tokenizer
3 — Library usage
use ;
4 — CLI
# Build (no C++ compiler needed)
# One-shot completion
# Embed texts (dense vectors, llama/qwen2/qwen3 embedding models)
# Transcribe speech (Whisper model directory, pure Rust)
# Start the API server
HTTP API
All endpoints are OpenAI-compatible.
POST /v1/chat/completions
Streaming — add "stream": true and consume SSE events:
POST /v1/embeddings
Note: Embeddings run a hidden-state forward pass with the pooling strategy from the GGUF metadata (mean / CLS / last-token). Supported architectures:
llama(e5-mistral, SFR-Embedding),qwen2(gte-Qwen2), andqwen3(Qwen3-Embedding).
POST /v1/audio/transcriptions
Mounted when the server is started with --whisper-model <dir> (a directory
holding model.safetensors + config.json + tokenizer.json, e.g. from
openai/whisper-tiny). Pure-Rust pipeline: WAV in any sample rate/channel
count → mel spectrogram → greedy decode.
GET /v1/models
GET /health
# {"status":"ok"}
Securing the server
The server binds 127.0.0.1 by default and speaks plaintext HTTP with no
authentication — fine for local use or behind a reverse proxy. Before
binding a public interface (--addr 0.0.0.0:8080), enable authentication
and/or TLS below.
Request limits
Two limits bound how much work a single client can demand, both tunable:
--max-concurrency(default: CPU count) caps simultaneous generations/embeddings; requests over the cap get503 Service Unavailableinstead of piling up model instances and exhausting memory. Lower it for large models on small boxes.--max-output-tokens(default: 4096) caps generated tokens per request regardless of the client'smax_tokens, bounding single-request CPU time.
Uploaded audio is limited to ~30 minutes of 16 kHz-equivalent samples, and
inline image data must be a base64 data: URL (filesystem paths and remote
URLs in image fields are rejected).
API-key authentication
Pass --api-key (or set JOSHUA_API_KEY) and every /v1 route requires the
key as an OpenAI-style bearer token; GET /health stays open for liveness
probes. Requests without the key get a 401 with an OpenAI-format error
body, so standard clients report it cleanly.
TLS (HTTPS)
TLS termination is built in via rustls,
behind an opt-in cargo feature — rustls' ring crypto provider compiles
C/assembly, so it is excluded from the default build to keep the
"only a Rust toolchain required" guarantee (same policy as the cuda/metal
features). Building with --features tls requires a C compiler.
--tls-cert takes a PEM certificate chain and --tls-key the matching
PKCS#8/RSA/SEC1 private key (both flags required together). Library users
can call server::serve_with_state_tls directly.
Huge pages
Large models thrash the TLB: a 7B Q4 model is ~4 GB, which is two million
4 KiB pages. Backing the mapping with huge pages cuts TLB misses. Select a
strategy with --huge-pages (or EngineOptions::huge_pages in the library);
the huge-page modes are Linux-only and fall back to normal pages elsewhere.
| Mode | Mechanism | Trade-off |
|---|---|---|
off (default) |
file-backed mmap, normal pages |
shared via the page cache; no setup |
transparent |
file-backed mmap + MADV_HUGEPAGE |
keeps the shared page cache; best-effort, kernel picks the size (usually 2 MiB); no setup |
2mb / 1gb / huge |
model copied into an anonymous MAP_HUGETLB mapping |
guarantees the page size, but uses private RAM (no shared page cache) and needs a preallocated pool |
# Best-effort transparent huge pages — safe to enable anywhere:
# Explicit 1 GiB pages (reserve the pool first):
transparent is the safe default to try: it preserves Joshua's shared-page-cache
model and simply does nothing if the kernel can't honour it. The explicit
modes give you a guaranteed page size at the cost of a private in-RAM copy and
a preconfigured hugepage pool (vm.nr_hugepages, or hugeadm for 1 GiB pages).
Compressed model files
Mapping only works when the bytes on disk are the model, and there are two common ways for that to stop being true — neither of them visible from the filename:
- the
.ggufis really a gzip/zstd/xz/… stream (a download that was never unpacked). Mapping it maps the compressed bytes, so nothing in the mapping is a tensor and the load fails — historically with a baffling magic-mismatch error from the header parser; - the
.ggufsits on a transparently compressing filesystem (btrfs/ZFScompress=…, NTFS compression). The mapping works, which is why this one goes unnoticed, but every page fault has to decompress a block instead of handing back a shared page-cache page, and loading and inference get dramatically slower.
Joshua checks for both before mapping and reports what it found, naming the format and the way out:
WARN "./weights/model.gguf" cannot be memory-mapped usefully: the file is a
gzip stream, not raw GGUF — mapping it maps the compressed bytes, so no
tensor can be read in place. Decompress it first (gunzip).
By default that is a warning and the load continues. Pass --mmap (or
EngineOptions::mmap(MmapMode::Required)) to say that mapping is the point of
the run: the same finding then fails the load instead, rather than silently
handing you a mapping that decompresses on every page fault.
Filesystem compression is detected from the file's on-disk allocation, so a
model stored sparsely reports the same way — equally bad news for a mapping.
It is not reported for --huge-pages 2mb/1gb/huge, which copy the model into
anonymous memory in one pass and pay for the decompression only once.
Models larger than RAM
A sparse mixture-of-experts model (DeepSeek-V4-Flash, Qwen3-MoE, …) far larger
than RAM has a bimodal access pattern that plain mmap serves poorly. A small
dense set — embeddings, norms, attention, routers, shared experts,
indexer/compressor, output — is touched on every token, while the routed
experts are touched sparsely (a token routes through a handful of the 256 per
layer). The default readahead hint (MADV_SEQUENTIAL) sets "free after use"
on the whole mapping, so the dense set gets evicted between requests; a blanket
random hint kills readahead for everything.
Joshua splits the model into dense and expert ranges and treats them differently:
| Flag | Effect |
|---|---|
--pin-hot-weights |
Prefetches the dense ranges (MADV_WILLNEED) at load and advises MADV_RANDOM on expert ranges, so the per-token working set is resident before the first request. Auto-on when the model file is larger than RAM; --pin-hot-weights=false forces it off. |
--mlock-hot-weights |
Additionally mlock(2)s the dense ranges for a hard residency guarantee. =required fails the load when the memlock limit is too low; the default (on) warns once and degrades to advisory pinning. |
--lazy-weights |
The blanket random-access hint for the whole mapping — no readahead at all. Mostly superseded by --pin-hot-weights, kept for the truly-RAM-starved case. |
The memlock limit is checked against the hot-set size before any mlock
call, with one warning naming limit vs required size (on DeepSeek-V4-Flash Q2_K
the dense set is ~8.2 GiB). Raise it with:
# systemd services
LimitMEMLOCK=infinity
# login session / PAM
# /etc/security/limits.conf:
# tserica - memlock unlimited
# live, without re-login (systemd user session)
Prefill on the sparse MoE loaders streams the experts instead of faulting them
in one 4 KiB page at a time: the layer loop advises MADV_SEQUENTIAL over the
whole expert span for the duration of the pass and dispatches the routed
experts in tensor-major (file) order, so each expert tensor is read as one
clean sequential pass while the layer computes (measured ~1.9 GB/s vs
~175 MB/s for per-page demand faults).
systemd user session trap: the unit's
LimitMEMLOCK=infinityis capped by the user manager's own hard limit, which is inherited from the login session — so on a headless box the unit setting alone does nothing. Add the PAM line and apply theprlimitlive fix (or re-login) for it to take effect.
On DeepSeek-V4-Flash Q2_K the whole file collapses to 2 dense + 1 expert
ranges, so pinning costs two madvise calls and one mlock. examples/tensor_sizes.rs
prints the dense/expert split of any GGUF to sanity-check a new model.
Environment variables
| Variable | Description |
|---|---|
JOSHUA_MODEL_PATH |
Default model path (overrides --model flag) |
JOSHUA_API_KEY |
API key required on /v1 routes (same as --api-key) |
JOSHUA_TLS_CERT |
PEM certificate chain for HTTPS (same as --tls-cert; needs --features tls) |
JOSHUA_TLS_KEY |
PEM private key for HTTPS (same as --tls-key) |
JOSHUA_LAZY_WEIGHTS |
Same as --lazy-weights |
JOSHUA_PIN_HOT_WEIGHTS |
Same as --pin-hot-weights (true/false, or the flag with no value) |
JOSHUA_MLOCK_HOT_WEIGHTS |
Same as --mlock-hot-weights (on, required, or off) |
JOSHUA_MAX_CONCURRENCY |
Cap on simultaneous generations/embeddings (same as --max-concurrency) |
JOSHUA_MAX_OUTPUT_TOKENS |
Hard ceiling on generated tokens per request (same as --max-output-tokens) |
JOSHUA_WHISPER_MODEL |
Whisper model directory mounted at /v1/audio/transcriptions (same as --whisper-model) |
JOSHUA_NPU_PLUGIN |
NPU vendor plugin path (same as --npu-plugin) |
JOSHUA_LLAMA_N_GPU_LAYERS |
llama.cpp adapter layer offload count (default: all) |
JOSHUA_LLAMA_MMPROJ |
Multimodal projector GGUF for vision via llama.cpp's mtmd |
JOSHUA_LLAMA_BACKENDS_DIR |
Directory of libggml-<name> modules to register at adapter startup (llama.cpp dynamic-backends builds) |
RUST_LOG |
Log filter (e.g. info, joshua=debug) |
Generation options
| Field | Type | Default | Description |
|---|---|---|---|
max_tokens |
u32 |
256 |
Maximum tokens to generate |
temperature |
f32 |
0.7 |
Sampling temperature (0 = greedy) |
top_p |
f32 |
0.9 |
Nucleus sampling threshold |
top_k |
i32 |
40 |
Top-k sampling (0 = disabled) |
min_p |
f32 |
0.05 |
Min-p filter relative to top token |
repetition_penalty |
f32 |
1.1 |
Penalise tokens seen in the last 64-token window (1.0 = disabled) |
stop_sequences |
Vec<String> |
[] |
Stop on these strings |
Supported models
Joshua reads general.architecture from the GGUF metadata and dispatches to
the matching pure-Rust candle loader. Currently supported architectures:
general.architecture |
Model families |
|---|---|
llama |
Llama 1/2/3, Mistral, Mixtral, TinyLlama, SmolLM, Vicuna, Zephyr, Yi, and anything else llama.cpp's converters emit as llama |
gemma / gemma2 / gemma3 / gemma-embedding |
Gemma 1, Gemma 2, Gemma 3 |
glm4 |
GLM-4 (dense) |
lfm2 |
Liquid LFM2 |
phi2 |
Phi-1, Phi-1.5, Phi-2 |
phi3 |
Phi-3, Phi-3.5 |
qwen2 |
Qwen1.5, Qwen2, Qwen2.5 |
qwen3 |
Qwen3 (dense) |
qwen3moe |
Qwen3 mixture-of-experts |
deepseek2 |
DeepSeek-V2, DeepSeek-V3, Kimi-K2 (MLA attention + fine-grained MoE) |
deepseek4 |
DeepSeek-V4 (Hyper-Connections residual mixing, alternating sliding-window / learned KV-compressor attention, Lightning-Indexer sparse attention, fine-grained MoE with IQ2_XXS experts) |
The deepseek2 loader is Joshua's own (candle has no quantized DeepSeek
path). It implements Multi-head Latent Attention with Q/KV LoRA, DeepSeek-V3 /
Kimi-K2 sigmoid-with-bias group-limited expert routing, shared experts, and
YaRN RoPE — and keeps the experts quantized (a 1 T-parameter MoE keeps its
on-disk footprint instead of exploding to f32 in RAM). Since PR #30, MLA
attention caches the compressed latent (c_kv + k_pe) rather than the
reconstructed per-head K/V — numerically identical to llama.cpp and ~70× less
KV memory for V3-class models; the full K/V is rebuilt once per forward from
the latent. Both the legacy combined (attn_kv_b) and modern MLA-split
(attn_k_b/attn_v_b) GGUF encodings load, and its logits are cross-checked
against llama.cpp.
The deepseek4 loader handles the architecture's three additions over V3:
Hyper-Connections mix the residual stream to hc_mult parallel copies with
Sinkhorn-normalised per-token weights, CSA/HCA compressor layers pool blocks of
4 / 128 tokens into a compressed KV, and the Lightning Indexer picks the
index_topk compressed positions each query attends to. The routed experts
stay quantized as IQ2_XXS trellis blocks decoded in-mapping during the matmul,
so a 162 B model keeps its on-disk footprint. Activations run in f32 on CPU.
Kimi-K3 (in progress). The correctness-critical primitives — Kimi Delta
Attention (per-channel decay gates), attention residuals, the situ
activation, and an MXFP4 decoder — are implemented in kimi_k3.rs and
unit-tested against the reference formulations, but a full forward pass is not
wired up yet: no released llama.cpp runs K3, and no weights were available to
check end-to-end logits against. It is not yet dispatchable from GGUF
metadata.
Example models:
google/gemma-3-270m-it/1b-it/4b-itQwen/Qwen3-0.6B/1.7BLiquidAI/LFM2-1.2Bmicrosoft/Phi-3-mini-4k-instructmistralai/Mistral-7B-Instruct-v0.3THUDM/GLM-4-9B-0414deepseek-ai/DeepSeek-V2-Lite,moonshotai/Kimi-K2-Instruct(as GGUF)deepseek-ai/DeepSeek-V4-Flash-162B(as GGUF)
Every other architecture name in llama.cpp's registry (Mamba, RWKV, GPT-2,
DeepSeek-V1, Granite, OLMo, StarCoder2, and ~70 more) is recognised at load time
and rejected with an error that names the architecture and lists what is
supported — so an unsupported model fails fast with a clear message instead
of a cryptic missing-tensor error. Coverage grows as candle gains loaders;
adding one is a small patch to src/model.rs.
NPU & llama.cpp backend interop (experimental)
Vendor NPU runtimes are proprietary C/C++ stacks, so Joshua contains them behind three safety layers instead of linking them into the pure-Rust core:
- Trait boundary — generation transparently falls back to the candle CPU/GPU path when a backend is missing or failing; a circuit breaker disables a backend after repeated failures.
- Plugin ABI, loaded at runtime — a backend is any shared library
exporting the four-function
joshua_npu_*C ABI (init/forward/reset/free, documented injoshua::npu). Nothing is linked at build time; the default build stays pure Rust. - Process isolation — by default the plugin runs inside the small
joshua-npu-shimsubprocess: control over pipes, tensors over shared memory, timeouts enforced, child killed on any violation. A crashing or hanging vendor runtime costs one request, never the server.
# Isolated by default:
# Opt into in-process loading (faster, but a plugin crash is fatal):
Vision / multimodal
Vision rides the same plugin mechanism: an optional fifth ABI symbol,
joshua_npu_media_prefill, lets a plugin tokenise-and-prefill a prompt whose
<__media__> markers correspond to attached images. The llama.cpp adapter
implements it with llama.cpp's mtmd — covering Qwen2.5-VL, Gemma 3 vision,
LLaVA, and the rest of its multimodal zoo:
# Point the adapter at the model's multimodal projector:
JOSHUA_LLAMA_MMPROJ=./weights/mmproj.gguf \
Clients send standard OpenAI vision messages (content parts with
image_url data URLs) or ChatMessage.images paths; decoding, sampling,
streaming, and tool calling work unchanged after the multimodal prefill.
Requests with images and no media-capable plugin fail fast with a clear
error.
The llama.cpp adapter
No vendor ships Joshua plugins — they ship llama.cpp/ggml backends. The
joshua-llamacpp-npu crate bridges that: it implements the plugin ABI by
driving llama.cpp itself, so every backend llama.cpp supports (Qualcomm
Hexagon NPU, Huawei CANN, CUDA, Vulkan, OpenCL, Metal, …) works through the
same isolated shim, against the same GGUF file, with no model conversion:
# Compiles llama.cpp — needs CMake + a C++ toolchain, which is exactly why
# it is NOT part of the default build; the C++ only ever runs in the shim.
Layer offload is controlled with JOSHUA_LLAMA_N_GPU_LAYERS (default: all).
NPU backends are enabled the same way as in llama.cpp itself — build it with
the vendor SDK (see llama.cpp's Snapdragon/CANN backend docs). With
llama.cpp's dynamic-backends feature, ready-built libggml-<name> modules
are picked up at startup from JOSHUA_LLAMA_BACKENDS_DIR (when unset, the
compile-time default directory is scanned), so a cross-compiled Hexagon/CANN
backend drops in with no code change.
The test suite proves the stack end to end without real hardware: a mock vendor plugin exercises determinism, crash containment (the plugin aborts — the server survives), hang timeouts, and engine fallback; the llama.cpp adapter is verified to produce byte-identical greedy output to Joshua's own candle path on the same weights.
Roadmap
- Chat completions (non-streaming)
- Chat completions (SSE streaming)
- Legacy text completions
- OpenAI-compatible model list
- mmap-based model loading
- Multi-architecture GGUF dispatch (all candle quantized loaders)
- Per-model chat templates from GGUF metadata
- Dense embeddings (llama / qwen2 / qwen3 embedding models, GGUF pooling metadata)
- Tool / function calling (OpenAI-compatible, Hermes/Mistral/Llama-3 formats)
- GPU acceleration (
cuda/metalcargo features) - KV-cache sharing across requests (warm model pool with prefix reuse)
- DeepSeek-V4 sparse-attention MoE loader (Hyper-Connections, CSA/HCA KV compression, Lightning Indexer, IQ2_XXS experts)
- DeepSeek-V2/V3 MLA latent cache (~70× smaller KV cache, prefill == incremental)
- Fused AVX2 k-quant kernels and SIMD quantized matmuls (CPU prefill/decode speed-ups)
- Sparse-MoE weight management (hot-weight pinning, mlock with memlock-limit check, prefill streaming)
- Vision / multimodal support (OpenAI image messages via llama.cpp
mtmdthrough the plugin shim) - Speech-to-text (Whisper — pure-Rust pipeline,
/v1/audio/transcriptions) - NPU backend architecture (isolated vendor-plugin shim + llama.cpp adapter for Hexagon/CANN/…)
- Kimi-K3 full forward pass (primitives done: Kimi Delta Attention, attention residuals,
situ, MXFP4 — see Supported models)
License
MIT