# Ferrite
Ferrite is a CPU-native large language model inference engine written in Rust.
It loads GGUF models, runs quantized inference with architecture-specific SIMD
kernels, and exposes both a command-line interface and an OpenAI-compatible
HTTP server.
Ferrite is currently alpha software. The supported surface is deliberately
small and well tested, but model coverage and sampling features are not yet
complete.
## Install
Ferrite's supported user installs are versioned release archives and the
official server image. Do not use `cargo install ferrite`: that crates.io name
belongs to an unrelated project.
Download the archive for your platform from the
[GitHub Releases page](https://github.com/vicotrbb/ferrite/releases), verify it
against the release's `SHA256SUMS` file, and then verify its provenance:
```sh
gh release verify-asset v<version> ferrite-v<version>-<target>.tar.gz \
--repo vicotrbb/ferrite
gh attestation verify ferrite-v<version>-<target>.tar.gz \
--repo vicotrbb/ferrite
```
The initial supported archive targets are macOS arm64 and statically linked
Linux x86_64. See [installation and verification](docs/install.md) for
extraction, container, and model-integrity instructions.
## Highlights
- Llama and Qwen2 model architectures, including Qwen2.5 GGUF artifacts.
- F32, F16, BF16, Q4_K, Q5_0, Q6_K, and Q8_0 tensor loading.
- NEON, Arm I8MM, and x86_64 AVX2 kernels with portable fallbacks.
- Greedy text generation, token streaming, profiling, and reproducible evals.
- OpenAI-compatible models, completions, and chat-completions endpoints.
- Optional prompt-prefix caching and experimental continuous batching.
- No model or binary test assets committed to the repository. Test GGUF data
is generated in memory, and real-model tests use local artifacts explicitly.
## Quick start
Install Rust 1.96.1 through [rustup](https://rustup.rs/), then build the release
binaries:
```sh
git clone https://github.com/vicotrbb/ferrite.git
cd ferrite
cargo build --release --locked -p ferrite-cli -p ferrite-server
```
Place a supported GGUF model under `target/models/`, then run a completion:
```sh
target/release/ferrite \
--model target/models/qwen2.5-0.5b-instruct-q4_k_m.gguf \
--prompt 'Write one sentence about Rust.' \
--generate-tokens 32 \
--stream
```
Or start the HTTP server:
```sh
target/release/ferrite-server \
--model target/models/qwen2.5-0.5b-instruct-q4_k_m.gguf \
--model-id qwen2.5-0.5b-q4_k_m \
--bind 127.0.0.1:8080 \
--api-key local-secret
```
```sh
curl http://127.0.0.1:8080/v1/chat/completions \
-H 'authorization: Bearer local-secret' \
-H 'content-type: application/json' \
-d '{"model":"qwen2.5-0.5b-q4_k_m","messages":[{"role":"user","content":"Hello"}],"max_completion_tokens":32}'
```
For the measured, machine-specific fast path, follow the
[performance golden path](docs/performance.md). Do not assume that an
experimental kernel is faster or compatible on every CPU.
## Documentation
- [Documentation index](docs/README.md)
- [Getting started](docs/getting-started.md)
- [Performance golden path](docs/performance.md)
- [Command-line interface](docs/cli.md)
- [HTTP server](docs/server.md)
- [OpenAI API compatibility](docs/openai-api.md)
- [Models and tensor formats](docs/models.md)
- [Architecture](docs/architecture.md)
- [Library API](docs/library-api.md)
- [Operational tools](docs/benchmark-tools.md)
- [Evaluation and regression gates](docs/evaluation.md)
- [Development guide](docs/development.md)
- [Safety policy](docs/safety.md)
- [Current limitations](docs/limitations.md)
- [Troubleshooting](docs/troubleshooting.md)
- [Release process](docs/releasing.md)
- [Changelog](CHANGELOG.md)
Architecture decisions and curated benchmark evidence live beside the guides
under `docs/`. Raw reproducible eval records remain under `scripts/evals/`.
Transient plans, session notes, private tool state, and model binaries are not
repository artifacts.
## Repository layout
```text
crates/ferrite-model GGUF parsing and tokenization
crates/ferrite-inference model loading, sessions, cache, and CPU kernels
crates/ferrite-cli local generation, profiling, and benchmarking
crates/ferrite-server OpenAI-compatible serving and eval clients
crates/ferrite-fixtures generated test fixtures
docs maintained user and contributor documentation
docs/adr durable architecture decisions
docs/benchmarks curated benchmark protocols and milestone evidence
scripts eval harness, raw eval records, and repository checks
```
## Quality gates
The required local checks are documented in [CONTRIBUTING.md](CONTRIBUTING.md).
CI runs formatting, strict Clippy, default and all-feature tests on Linux and
macOS, plus a separate Rust 1.96 MSRV check. Linux jobs also enforce rustdoc,
doctests, documentation, repository hygiene, eval-harness tests, package
contents, RustSec advisories, licenses, duplicate dependencies, and sources.
## License
Ferrite is available under the [Apache License 2.0](LICENSE).