# hf2q — Architecture
This document is a source-grounded map of the `hf2q` codebase at
`main` HEAD. It exists to bring a new contributor up to "I can find
the code that owns X" in under thirty minutes.
It is **descriptive** (what is) not **prescriptive** (what should be).
For target-state designs see the per-ADR files under `docs/`; for the
prior-art inference path see `docs/arch-current-inference-path.md`.
---
## 1. What `hf2q` is
`hf2q` is a single Rust binary with two cooperating halves:
1. **Convert** — read a HuggingFace model directory (`config.json` +
`*.safetensors`), normalize tensor names per architecture, run a
quantization algorithm, and emit either a GGUF file (single binary,
`llama.cpp` consumers) or an mlx-lm-shaped safetensors directory
(mlx-lm, Candle, vLLM, hf2q's own serve loader).
2. **Serve / Generate** — load a GGUF, run prefill + decode on the
Apple-Silicon GPU through the `mlx-native` crate, and expose
OpenAI-compatible HTTP endpoints (chat completions, embeddings,
models) with SSE streaming, tool calls, vision, grammar-constrained
sampling, and a persistent block-prefix KV cache.
Both halves share the same internal IR (`src/ir/`) and the same arch
registry (`src/arch/`), so adding a new model family is one ADR + one
module under `src/models/` and `src/inference/models/`.
### Sovereignty rule (`docs/arch-onboarding.md`)
- **Pure Rust.** No `llama.cpp` / `candle` code, crate, binary, or
build artifact in hf2q deliverables at build / test / CI time.
Enforced by ADR-008 ("candle divorce").
- **Spec sources are read-only.** `llama-arch.cpp`,
`convert_hf_to_gguf.py`, `clip.cpp`, `clip-model.h` are *read* to
derive specs; every transcribed value carries a `// citation:` line
back to the source file + line.
- **No external oracles in tests.** Correctness is proven by
hand-authored expected values, spec-driven synthetic inputs, or
round-trip gates (emit → load through our own loader).
---
## 2. Crate layout
```
hf2q (one binary `hf2q`, one narrow [lib] facade for tests)
├── src/main.rs process entry, exit-code classification
├── src/lib.rs narrow library facade (kv-persist only,
│ for tests under `tests/`)
├── src/cli.rs clap derive — every subcommand + arg
├── src/doctor.rs `hf2q doctor` runtime diagnostic
├── src/preflight.rs ADR-012 preflight checks (disk, token, …)
├── src/progress.rs indicatif-based progress reporting
├── src/report.rs machine-readable convert-result report
├── src/gguf_patch.rs metadata-only GGUF rewriter (no tensor I/O)
│
├── src/arch/ ADR-012 arch registry (single source of truth)
│ ├── catalog.rs TensorCatalog — expected tensor names + dtypes
│ ├── conformance.rs quality thresholds, smoke prompts, MTP/vision flags
│ ├── entries/ one file per registered arch (qwen35, qwen35moe, …)
│ ├── registry.rs ArchRegistry, ArchEntry, ArchError
│ └── smoke.rs end-gate smoke driver (`hf2q smoke`)
│
├── src/input/ external model I/O — nothing else touches raw model files
│ ├── config_parser.rs HF config.json → ModelMetadata
│ ├── safetensors.rs streaming mmap shard reader → TensorMap
│ ├── hf_download.rs HF Hub download (with x-linked-etag integrity)
│ └── integrity.rs per-shard SHA-256 verification
│
├── src/ir/ internal representation crossing modules
│ ├── mod.rs ModelMetadata, TensorMap, DType, QuantizedTensor, …
│ └── lazy.rs lazy-tensor handle (ADR-014 streaming convert)
│
├── src/models/ per-arch conversion (tensor rename, MoE merge, metadata)
│ ├── qwen35/ ADR-012 Qwen 3.5 / 3.6 convert (dense + MoE)
│ └── vit/ ADR-012 P10 pure-Rust mmproj (ViT) emitter
│
├── src/quantize/ every Q-format codec
│ ├── q_legacy.rs Q4_0 / Q5_0 / Q8_0 (`llama.cpp` block formats)
│ ├── k_quant.rs K-quant common super-block math
│ ├── k_quant_codec.rs Q2_K…Q6_K block encoders / decoders
│ ├── k_quant_codec_quantizer.rs the bind-it-all-together quantizer
│ ├── variant_quantizer.rs K-quant variant dispatch
│ ├── mixed.rs dynamic-quant-{4-6,4-8,6-8,2-8}
│ ├── layer_mix.rs per-layer sensitivity → bit choice
│ ├── dwq_k_quantizer.rs DWQ-overlay K-quant adapter
│ └── static_quant.rs dynamic→static plan freezer
│
├── src/calibrate/ activation capture + DWQ training (ADR-020)
│ ├── imatrix.rs imatrix capture
│ ├── imatrix_calibrator.rs imatrix → sensitivity prior
│ ├── imatrix_xvalidate.rs leave-one-out validation
│ ├── dwq.rs per-Linear DWQ training entry
│ ├── dwq_loop.rs multi-Linear orchestration
│ ├── dwq_e2e.rs end-to-end DWQ run + bookkeeping
│ ├── dwq_targets.rs FP32-teacher target derivation
│ ├── dwq_activation.rs activation cache for hybrid arches
│ ├── dwq_calibrator.rs teacher provider + Adam loop
│ ├── dwq_benchmark.rs benchmark harness
│ ├── adam.rs Adam optimizer (CPU + GPU variants)
│ ├── autograd*.rs forward-mode-tape autograd primitives
│ ├── gguf_teacher.rs GGUF as FP32 teacher provider
│ ├── hf_safetensors_teacher.rs HF safetensors teacher
│ ├── qdq_gpu.rs quantize-dequantize round-trip on GPU
│ ├── fd_sensitivity.rs finite-difference sensitivity
│ ├── sensitivity.rs Hessian-trace sensitivity prior
│ ├── sensitivity_comparison.rs cross-method comparison
│ ├── dynamic_quant{_gpu}.rs dynamic-quant kernels
│ ├── calibrator.rs top-level calibrate orchestration
│ ├── calibration_batcher.rs windowed-corpus tokenizer
│ ├── cache.rs calibrate-side cache
│ ├── apex.rs APEX adaptive-quant planner
│ ├── mlx_safetensors_loader.rs mlx-lm overlay loader
│ ├── qwen35_*.rs qwen35-specific per-layer DWQ ops
│ └── … (~35 files; this is the dense quarter)
│
├── src/backends/ output writers — GGUF + mlx-lm safetensors
│ ├── gguf.rs GGUF emitter (header, tensor stream, sidecars)
│ ├── safetensors_out.rs mlx-lm safetensors directory writer
│ ├── chat_templates.rs per-arch chat template lookup
│ └── chat_templates/ canonical Jinja2 templates
│
├── src/quality/ post-convert quality measurement
│ ├── cosine_sim.rs weight-level cosine similarity
│ ├── kl_divergence.rs output-logit KL
│ ├── perplexity.rs PPL on a corpus
│ ├── ppl_driver.rs forward-pass driver for PPL
│ ├── kernel_parity.rs kernel-equivalence proofs
│ └── regression.rs regression-gate accountant
│
├── src/intelligence/ hardware probe + auto-quant + RuVector
│ ├── hardware.rs chip detection, MTL device, memory probe
│ ├── fingerprint.rs stable model fingerprint (for cache keys)
│ ├── auto_quant.rs auto-mode planner
│ ├── heuristics.rs rule-based fallback when RuVector is silent
│ └── ruvector.rs optional self-learning store (cargo feature)
│
├── src/inference/ runtime model + spec-decode + vision
│ ├── models/ per-arch forward graphs
│ │ ├── gemma4/ dense + MoE 30-layer Gemma 4
│ │ ├── qwen35/ dense + MoE Qwen 3.5 / 3.6
│ │ ├── qwen3vl_text/ Qwen 3-VL text tower (vision lives elsewhere)
│ │ ├── bert/ BERT embedding model
│ │ └── nomic_bert/ Nomic embedding model
│ ├── spec_decode/ ADR-029 speculative-decode primitives
│ │ ├── ngram_proposer.rs pure-CPU n-gram drafter
│ │ ├── dflash/ ADR-030 dFlash block-diffusion drafter
│ │ └── verifier.rs multi-token verify forward
│ └── vision/ mmproj load + image embed
│
├── src/serve/ HTTP API, KV-cache, multi-model
│ ├── api/ axum router + handlers + state
│ │ ├── schema.rs OpenAI wire types
│ │ ├── handlers.rs /v1/* request handlers
│ │ ├── router.rs axum router + middleware
│ │ ├── sse.rs SSE encoder
│ │ ├── engine.rs Gemma 4 engine wrapper
│ │ ├── engine_qwen35.rs Qwen 3.5 engine wrapper
│ │ ├── engine_qwen3vl.rs Qwen 3-VL engine wrapper
│ │ ├── grammar/ grammar-constrained sampling
│ │ ├── kv_spill_descriptor.rs KV-spill metadata
│ │ ├── tq_packed_descriptor.rs TurboQuant packed metadata
│ │ ├── registry.rs model registry (multi-model serve)
│ │ ├── embedding_pool.rs /v1/embeddings request pool
│ │ ├── middleware.rs CORS, request-id, auth
│ │ └── state.rs AppState, ServerConfig
│ ├── forward_mlx.rs Gemma-4 forward via mlx-native
│ ├── forward_prefill.rs per-token prefill
│ ├── forward_prefill_batched.rs ADR-015 batched prefill (35× wins here)
│ ├── kv_persist/ ADR-017 persistent block-prefix cache
│ │ ├── block_store.rs disk-backed block store
│ │ ├── writer.rs async writer + fsync barriers
│ │ ├── recovery.rs crash-recovery on startup
│ │ ├── format.rs envelope + sidecar codecs
│ │ ├── index.rs in-memory index
│ │ ├── lcp_registry.rs longest-common-prefix registry
│ │ ├── spiller.rs KvSpiller<E> trait impl
│ │ └── metrics.rs cache-side telemetry seam
│ ├── multi_model.rs multi-model registry + eviction
│ ├── encoder_worker_singleton.rs Metal encoder worker
│ ├── auto_pipeline.rs serve-time pipeline selection
│ ├── cache.rs global model cache (~/.cache/hf2q)
│ ├── parity_quality.rs ADR-009 parity assertions
│ ├── provenance.rs GGUF → producer fingerprint
│ ├── quant_select.rs model → quant-variant selector
│ ├── sampler_pure.rs temp / top-k / top-p sampling
│ ├── spec_decode_cli.rs generate-time spec-decode driver
│ ├── header.rs GGUF header read + validate
│ ├── gpu.rs shared GPU resource init
│ ├── layer_ctx.rs per-layer mutable context
│ ├── load_info.rs structured `loaded` event
│ ├── config.rs ServeArgs validation
│ └── mod.rs cmd_serve + cmd_generate entry
│
└── src/bin/ one-off audit binaries (iter23/24/25, dump_gguf_*)
```
The library facade (`src/lib.rs`) deliberately re-exports only
`serve::kv_persist::{block_store, format, index, metrics, recovery,
writer, lcp_registry}` — everything else stays binary-private. Tests
under `tests/` are integration-style; they bind to either the public
CLI surface (via `assert_cmd`) or to that narrow lib facade.
---
## 3. The convert pipeline
```
HF input directory or HF Hub repo
│
┌────────────────────┴────────────────────┐
│ src/input/ (mmap, config parse, │
│ integrity, optional HF download) │
└────────────────────┬────────────────────┘
│ ModelMetadata + TensorMap
v
┌─────────────────────────────────────────┐
│ src/arch/ look up arch entry │
│ src/models/<arch>/ rename + MoE merge│
└────────────────────┬────────────────────┘
│ canonical-named TensorMap
v
┌─────────────────────────────────────────┐
│ src/quantize/ pick codec by --quant │
│ [optional] src/calibrate/ DWQ │
│ [optional] imatrix sensitivity │
└────────────────────┬────────────────────┘
│ QuantizedModel
v
┌─────────────────────────────────────────┐
│ src/backends/ │
│ gguf.rs → *.gguf │
│ safetensors_out.rs → mlx-lm dir │
└────────────────────┬────────────────────┘
│
┌────────────────────┴────────────────────┐
│ src/quality/ cosine + (optional) PPL │
│ src/report.rs structured convert log │
└─────────────────────────────────────────┘
```
Streaming is real: `safetensors` shards are mmap'd, tensors are
quantized in rayon-parallel chunks (`src/quantize/`), and the writer
sinks blocks to disk as soon as they're ready. Memory ceiling is set
by the largest individual tensor + its quant scratch — typically a few
GB even for 35 B-parameter models.
### Quantization families
| **Legacy block** (`q4_0`, `q5_0`, `q8_0`) | `src/quantize/q_legacy.rs` | Per-block 32-element row layout. Single scale; no min. |
| **K-quant** (`q2_k`…`q6_k`, `q4_k_m`, `q5_k_m`, …) | `src/quantize/k_quant*.rs`, `variant_quantizer.rs` | 256-element super-blocks with per-row scale + min. Codec maths verified bit-identical to `llama.cpp` via round-trip + spec-citation tests. |
| **Imatrix-K-quant** (`imatrix-q4_k_m`, `imatrix-adaptive`, …) | `src/calibrate/imatrix*.rs` + same codecs | Activation-imatrix-weighted residual minimization. Per-layer sensitivity threshold drives bit assignment. |
| **Dynamic / DWQ-mixed** (`dynamic-quant-{4-6,4-8,6-8,2-8}`) | `src/quantize/mixed.rs`, `layer_mix.rs` | Per-layer mixed-bit; `static_quant.rs` freezes a dynamic plan. |
| **DWQ-overlay** (`dwq-4`, `hf2q dwq-train`) | `src/calibrate/dwq*.rs`, `src/quantize/dwq_k_quantizer.rs` | Trains `<stem>.scales` + `<stem>.biases` per-Linear over an FP32 teacher (GGUF or HF safetensors) with Adam to minimize KL. Emits an mlx-format safetensors overlay. |
The full menu, gated by per-arch availability, lives in
`docs/converting-a-model.md`. The decommissioned variants (e.g. `apex`,
`dwq-mixed-*`) emit structured "did you mean" errors via
`src/cli.rs:map_deleted_quant`.
---
## 4. The inference pipeline
### 4.1 Load
`serve::cmd_serve` / `serve::cmd_generate` →
1. **Header read** (`serve/header.rs`) validates the GGUF magic +
producer fingerprint (`serve/provenance.rs`) against the arch
registry.
2. **Arch dispatch** picks an explicit engine wrapper under
`serve/api/engine*.rs` (Gemma 4, Qwen 3.5/3.6, Qwen 3-VL,
DeepSeek-V4). Unsupported or cross-family shapes fail rather than falling
through an approximately compatible graph.
3. **Weight load** dequantizes-on-demand into `mlx-native` MTL buffers
(`inference/models/<arch>/...`). Fused-kernel pipelines compile at
load time so the first request doesn't pay shader-compile latency.
4. **Warmup** runs a 1-token decode + a 10-token prefill, clears the
KV cache, and emits the structured `loaded` event
(`serve/load_info.rs`).
### 4.2 Prefill
Two paths share the same forward graph but differ in dispatch shape:
- `serve/forward_prefill.rs` — per-token; safe default; the ADR-009
parity reference. Used when `HF2Q_BATCHED_PREFILL=0` or for arches
not yet on the batched path.
- `serve/forward_prefill_batched.rs` — ADR-015's batched prefill;
used by applicable Gemma and Qwen graph paths. Was the single largest serve
speedup in the project (35× over per-token at `pp1024` on Gemma 4)
when the HTTP path was wired in ADR-028 Phase 15. Flash-Attention
(ADR-011) lives in this path.
Slot-aware scheduling adds a transaction boundary above the family forward
graph (`serve/api/engine.rs`, ADR-040). For Qwen 3.5/3.6 text serving, one
prefill transaction contains at most 2,048 new prompt tokens on one slot-local
hybrid KV cache. A successful transaction publishes the scheduler ledger only
after all full-attention and MTP cursors agree. `Mixed` steps decode active
streams before advancing the next cold prefill, and cold prefills rotate
round-robin. This is distinct from the unsafe chunk-scan DeltaNet experiment:
the outer transaction bounds the complete attention/MoE graph and gives the
worker cancellation and fairness boundaries.
Gemma 4 uses the same outer state-machine shape for long plain-text prompts,
with a family-specific candidate cap of 4,096 tokens and mandatory splits at
the stable-prefix boundary. Each successful transaction validates and commits
the engaged HB, hybrid, dense, and MLX cursor rows before scheduler
publication. DeepSeek uses its native verifier transaction width instead of a
generic token cap. Both cold work and meaningful retained-prefix suffixes are
resumable; cached suffixes remain outside the cold-cohort policy.
DeepSeek's `Mixed` step has a separate interactive budget. While a real decode
lane is runnable, one prefill transaction is capped at two 128-token verifier
windows and decode receives up to eight tokens before the next prefill slice.
If all decode owners are parked or absent, the cap is removed and the proven
2,048-token solo-prefill plan resumes. This avoids applying the latency cost of
small transactions to bulk work that has no interactive peer.
### 4.3 Decode
`serve/forward_mlx.rs` is the per-token decode hot loop. It:
1. Runs the per-arch graph (`inference/models/<arch>/`) through
`mlx-native` MTL dispatches.
2. Reads / writes the KV cache through TurboQuant
(`docs/operating-kv-cache.md`, ADR-007): K and V are Hadamard-
transformed and quantized to 8-bit with a per-block scale, giving
~2× memory savings vs an F16 KV cache at negligible quality loss
(Gate A cosine mean 0.9998, Gate B argmax divergence 0.8%). For
Qwen 3.5 / 3.6 (ADR-027) the TQ-HB path drops F32 K/V allocations
entirely, delivering 3.94× savings against the F32-only baseline
(340 MiB vs 1.34 GiB at 32K context). The TQ-HB encode is fused
into the dense KV-store path; the on-load path lazily promotes
from the persisted block store.
3. Samples through `serve/sampler_pure.rs` (temp / top-k / top-p) and
optionally a grammar-constrained `serve/api/grammar/` sampler for
tool calls and JSON-mode.
### 4.4 Speculative decode (ADR-029 / ADR-030)
`inference/spec_decode/`:
- **N-gram proposer** (`ngram_proposer.rs`) — pure-CPU drafter,
cost-free when the suffix repeats.
- **dFlash drafter** (`dflash/`) — ADR-030 block-diffusion neural
drafter. Currently default-OFF; runs through the same verify
forward as the n-gram path.
- **Verifier** (`verifier.rs`) — multi-token verify forward that
returns per-position logits + a rollback handle for the KV cache so
rejected drafts don't bias the production state.
A sourdough byte-identity gate ensures spec-decode never diverges from
the vanilla path at `K=0` — this is the production safety contract
that lets the drafter ship behind a default flag.
### 4.5 Vision
`inference/vision/` loads an mmproj GGUF (emitted by `models/vit/`)
and runs the vision tower as a Metal kernel chain identical to the
text tower's primitive set. `inference/models/qwen3vl_text/` consumes
the projected embeddings via the chat-template's `<|vision_start|>`
markers.
---
## 5. The HTTP server
`serve/api/` is a thin axum 0.7 service. The router
(`serve/api/router.rs`) is fixed at:
| `GET /health` | `handlers::health` — process liveness; remains 200 when generation readiness has failed |
| `GET /readyz` | `handlers::readyz` — 200 only while generation is ready and every pooled engine worker is healthy |
| `GET /metrics` | `handlers::metrics` — Prometheus exposition |
| `GET /v1/models` | `handlers::list_models` |
| `GET /v1/models/:model_id` | `handlers::get_model` |
| `POST /v1/chat/completions` | `handlers::chat_completions` |
| `POST /v1/embeddings` | `handlers::embeddings` |
| `POST /shutdown` | `handlers::shutdown` (auth-gated) |
`AppState` (`serve/api/state.rs`) carries the engine handle, the
multi-model registry, the embedding pool, and a warmed
`KernelRegistry` for `/v1/embeddings` so handlers never pay
shader-compile latency.
Middleware (`serve/api/middleware.rs`) layers CORS, optional Bearer
auth, and request-id propagation. SSE encoding lives in
`serve/api/sse.rs`; the grammar sampler emits tool-call deltas that
the SSE encoder threads into the OpenAI-shaped stream.
Metal watchdog, ignored-submission, and device-loss errors are worker-fatal,
not slot-local. An `EngineSupervisor` outside the model worker also observes
individual Metal transaction leases, so a call that never returns still
poisons readiness and unblocks unary/SSE waiters. Qwen, Gemma, and DeepSeek
workers close admission, terminate active, detached, buffered, and
pre-close-permitted requests once, stop submitting GPU work, and make
`/readyz` return 503. The guarded SSE bridge treats a full downstream queue as
request-local cancellation instead of blocking the sole worker. `/health`
remains a liveness endpoint. OS supervision recreates the process/device
generation; hf2q does not attempt an in-process reset of a poisoned Metal
queue.
The persistent block-prefix cache (`serve/kv_persist/`) is the most
operationally interesting piece: it makes the first prefill of a
recurring system prompt nearly free across process restarts.
`block_store.rs` is the atomic-rename-under-SIGKILL surface that's
proved by a child-process kill-9 integration test
(`tests/kv_persist_writer_kill_minus_9.rs`).
---
## 6. The arch registry (`src/arch/`)
The arch registry is the **single source of truth** for everything an
architecture needs to be a first-class hf2q citizen. The struct
(`src/arch/registry.rs:56-84`):
```rust
pub struct ArchEntry {
pub arch: &'static str, // GGUF arch string ("qwen35", "qwen35moe")
pub hf_architectures: &'static [&'static str], // HF config.json::architectures[0]
pub tensor_catalog: &'static TensorCatalog, // P4 tensor-name templates
pub has_mtp: bool, // emits blk.{L}.nextn.* tensors?
pub has_vision: bool, // --emit-vision-tower path?
pub smoke_prompts: &'static [&'static str], // deterministic inputs for `hf2q smoke`
pub ppl_corpus: EvalCorpus, // Decision-17 PPL eval corpus
pub quality_thresholds: QualityThresholds, // per-arch quality bounds
pub disk_floor_gb: u32, // smoke preflight EXIT_INSUFFICIENT_DISK
pub hf_repos: &'static [&'static str], // expected HF repos for smoke
pub auto_override: Option<&'static str>, // P8 Decision-18 AutoResolver override
}
```
Per-arch GGUF metadata emission and the HF→GGUF tensor-name function
live in `src/models/<arch>/` modules (not on the `ArchEntry` itself);
the layer-map dispatch lives in `src/backends/gguf.rs`.
Adding a new arch is mechanical: add `src/arch/entries/<arch>.rs`
register it in `src/arch/entries/mod.rs`, transcribe the tensor
catalog with `// citation:` lines, add a smoke prompt, and the
following tooling all "just works":
- `hf2q smoke --arch <arch>` (ADR-012 Decision 16 end-gate)
- `hf2q parity --arch <arch>` (ADR-009 parity validation)
- `hf2q dwq-train` (ADR-020) cohort sensitivity priors
- the convert pipeline (rename + metadata emission)
- the `hf2q info` inspector
The contract is "one file per arch + ~50 LOC registration + 200–400
LOC arch-specific transforms" replacing the ~1500-LOC harness rewrite
every new arch paid pre-`src/arch/`. The canonical reference is
`docs/arch-onboarding.md`.
---
## 7. Observability + operator surface
- **Logging.** `--log-format text|json` with `--log-level
debug|info|warn|error`. JSON logs are one object per line; safe for
Loki / Datadog ingest.
- **Serve dashboard.** `--operator-ui auto|dashboard|plain`. `auto` uses an
alternate-screen live view only for interactive text stderr. Engine events
enter a bounded `try_send` channel, never the inference critical path. Each
request exposes family-local identity, slot, phase, cache/new-token split,
prefill completion/rate/ETA, and decode rate without exposing prompt or tool
contents. Pipes, CI, services, and JSON logging retain plain output.
- **Progress.** `indicatif` bars at convert time; suppressed when
stderr is not a TTY.
- **Metrics.** Prometheus exposition on `GET /metrics` covering
request latency, token throughput, KV-cache hit rate, MTL dispatch
count and the regression-gate counters.
- **Structured convert report** (`src/report.rs`): every convert run
emits a machine-readable JSON manifest pointing at the input
hashes, the arch entry version, the chosen quant, per-tensor
drift summaries, and the wall-clock breakdown.
- **Environment flags.** Investigation-only env vars are listed in
`docs/operator-env-vars.md`. Defaults are the safe-production
choice; opt-in flags carry a one-shot ack at startup.
- **Exit codes.**
- `0` success.
- `1` conversion error.
- `2` quality threshold exceeded (ADR-009 parity).
- `3` input / validation error.
- `4–8` `hf2q smoke` preflight failures (per ADR-012 Decision 16:
each failure mode gets a distinct code so CI can tell them apart).
---
## 8. Testing
`tests/` hosts integration tests; `src/**/*.rs` carries unit tests inline. The
harness leans on three patterns:
1. **Spec-citation tests.** Every K-quant codec has a hand-authored
spec-driven test that matches `llama.cpp`'s block layout byte-for-
byte without linking against `llama.cpp`.
2. **Round-trip gates.** Convert → reload via our own GGUF reader →
assert tensor name + shape + dtype + (for float passes) byte
identity.
3. **End-gate smoke prompts.** `hf2q smoke` runs the arch's canonical
prompts and asserts the model emits the expected first / stop
tokens. Failure modes get distinct exit codes (see §7).
Benchmarks live in `benches/` and `scripts/`; the latter directory
also carries every ADR's repro runbook.
---
## 9. ADR index (where the rationale lives)
The "why" of every load-bearing design decision lives in numbered
ADRs under `docs/`. The most architecturally consequential ones:
| **ADR-004** | GGUF compatibility — what we promise to `llama.cpp` consumers. |
| **ADR-005** | Inference server — Phase 1/2/3 of the HTTP API. |
| **ADR-006** | `mlx-native` GPU backend — why Metal, why not MPS-graph. |
| **ADR-007** | TurboQuant KV cache — Hadamard-quantized K/V at 4 bits. |
| **ADR-008** | Candle divorce — sovereignty rule, single-backend invariant. |
| **ADR-009** | Reference parity + coherence recovery — the parity contract. |
| **ADR-010** | Exact batched-kernel parity — verified-kernel ledger. |
| **ADR-011** | Flash-Attention prefill — the prefill speedup. |
| **ADR-012** | Qwen35MoE conversion — and the arch-registry contract. |
| **ADR-013** | Qwen3.5 inference — per-arch inference module pattern. |
| **ADR-014** | Streaming convert pipeline + peer-parity gates (cross-arch). |
| **ADR-015** | mlx-native — general decode-path speed improvements (qwen35 + gemma). |
| **ADR-016** | coreml-native opportunistic encoder offload — P2 ViT + P3 BERT. |
| **ADR-017** | Persistent Block Prefix Cache for serve mode — `serve/kv_persist/`. |
| **ADR-018** | Uniform Model-Load UX Across Families — `hf2q serve --model PATH` invariants. |
| **ADR-019** | mlx-native Encoder Architecture — Per-Stage Fence Design. |
| **ADR-020** | DWQ + Mixed-Precision Quantization (port from mlx-lm) — `hf2q dwq-train`. |
| **ADR-021** | Qwen3VL ViT prelude GPU port — vision tower. |
| **ADR-022** | Kernel-coverage parity with `llama.cpp`. |
| **ADR-027** | Qwen3.5 TQ KV cache + persist family. |
| **ADR-028** | Peer parity, coherence + speed (the perf canonical). |
| **ADR-029** | Gemma4 MoE pipeline is the gap — perf investigation. |
| **ADR-030** | dFlash block-diffusion spec-decode. |
| **ADR-040** | Full-context agent slots, scheduler admission, fairness, and per-slot state. |
Each ADR carries phase status, acceptance tests, and a "what comes
next" section. ADRs are append-only; superseded ones are linked
forward rather than deleted.
---
## 10. Where to look first
| Read the public CLI surface | `src/cli.rs` |
| Trace a `convert` request | `src/serve/mod.rs` → `cmd_generate` is the wrong one; `src/main.rs` dispatches `Command::Convert` into `quantize::cmd_convert`. |
| Trace a serve chat request | `src/serve/api/handlers.rs::chat_completions` → `engine*.rs` → `inference/models/<arch>/forward.rs` |
| Add a new model family | `docs/arch-onboarding.md` |
| Add a new quant variant | `src/quantize/` + register in `src/cli.rs::QuantArg` |
| Tune the KV cache | `docs/operating-kv-cache.md` + `src/serve/kv_persist/` |
| Add a new HTTP route | `src/serve/api/router.rs` + `handlers.rs` |
| Find a perf number | the ADR-028 / ADR-029 iter-logs under `docs/` |
For anything time-sensitive, prefer reading the relevant ADR over
this document. ADRs are the system of record; this file is the map.