Ferrite
Ferrite is a CPU-native large language model inference engine written in Rust. It loads GGUF models, runs quantized inference with architecture-specific SIMD kernels, and exposes both a command-line interface and an OpenAI-compatible HTTP server.
Ferrite is currently alpha software. The supported surface is deliberately small and well tested, but model coverage and sampling features are not yet complete.
Install
Ferrite's supported user installs are versioned release archives and the
official server image. Do not use cargo install ferrite: that crates.io name
belongs to an unrelated project.
Download the archive for your platform from the
GitHub Releases page, verify it
against the release's SHA256SUMS file, and then verify its provenance:
The initial supported archive targets are macOS arm64 and statically linked Linux x86_64. See installation and verification for extraction, container, and model-integrity instructions.
Highlights
- Llama and Qwen2 model architectures, including Qwen2.5 GGUF artifacts.
- F32, F16, BF16, Q4_K, Q5_0, Q6_K, and Q8_0 tensor loading.
- NEON, Arm I8MM, and x86_64 AVX2 kernels with portable fallbacks.
- Greedy text generation, token streaming, profiling, and reproducible evals.
- OpenAI-compatible models, completions, and chat-completions endpoints.
- Optional prompt-prefix caching and experimental continuous batching.
- No model or binary test assets committed to the repository. Test GGUF data is generated in memory, and real-model tests use local artifacts explicitly.
Quick start
Install Rust 1.96.1 through rustup, then build the release binaries:
Place a supported GGUF model under target/models/, then run a completion:
Or start the HTTP server:
For the measured, machine-specific fast path, follow the performance golden path. Do not assume that an experimental kernel is faster or compatible on every CPU.
Documentation
- Documentation index
- Getting started
- Performance golden path
- Command-line interface
- HTTP server
- OpenAI API compatibility
- Models and tensor formats
- Architecture
- Library API
- Operational tools
- Evaluation and regression gates
- Development guide
- Safety policy
- Current limitations
- Troubleshooting
- Release process
- Changelog
Architecture decisions and curated benchmark evidence live beside the guides
under docs/. Raw reproducible eval records remain under scripts/evals/.
Transient plans, session notes, private tool state, and model binaries are not
repository artifacts.
Repository layout
crates/ferrite-model GGUF parsing and tokenization
crates/ferrite-inference model loading, sessions, cache, and CPU kernels
crates/ferrite-cli local generation, profiling, and benchmarking
crates/ferrite-server OpenAI-compatible serving and eval clients
crates/ferrite-fixtures generated test fixtures
docs maintained user and contributor documentation
docs/adr durable architecture decisions
docs/benchmarks curated benchmark protocols and milestone evidence
scripts eval harness, raw eval records, and repository checks
Quality gates
The required local checks are documented in CONTRIBUTING.md. CI runs formatting, strict Clippy, default and all-feature tests on Linux and macOS, plus a separate Rust 1.96 MSRV check. Linux jobs also enforce rustdoc, doctests, documentation, repository hygiene, eval-harness tests, package contents, RustSec advisories, licenses, duplicate dependencies, and sources.
License
Ferrite is available under the Apache License 2.0.