ferrite-model 0.2.0

GGUF parsing and tokenization for the Ferrite inference engine
Documentation
  • Coverage
  • 100%
    125 out of 125 items documented0 out of 31 items with examples
  • Size
  • Source code size: 105.2 kB This is the summed size of all the files inside the crates.io package for this release.
  • Documentation size: 902.7 kB This is the summed size of all files generated by rustdoc for all configured targets
  • Ø build duration
  • this release: 4s Average build duration of successful builds.
  • all releases: 5s Average build duration of successful builds in releases after 2024-10-23.
  • Links
  • vicotrbb/ferrite
    4 2 0
  • crates.io
  • Dependencies
  • Versions
  • Owners
  • vicotrbb

Ferrite

Ferrite is a CPU-native large language model inference engine written in Rust. It loads GGUF models, runs quantized inference with architecture-specific SIMD kernels, and exposes both a command-line interface and an OpenAI-compatible HTTP server.

Ferrite is currently alpha software. The supported surface is deliberately small and well tested, but model coverage and sampling features are not yet complete.

Install

Ferrite's supported user installs are versioned release archives and the official server image. Do not use cargo install ferrite: that crates.io name belongs to an unrelated project.

Download the archive for your platform from the GitHub Releases page, verify it against the release's SHA256SUMS file, and then verify its provenance:

gh release verify-asset v<version> ferrite-v<version>-<target>.tar.gz \
  --repo vicotrbb/ferrite
gh attestation verify ferrite-v<version>-<target>.tar.gz \
  --repo vicotrbb/ferrite

The initial supported archive targets are macOS arm64 and statically linked Linux x86_64. See installation and verification for extraction, container, and model-integrity instructions.

Highlights

  • Llama and Qwen2 model architectures, including Qwen2.5 GGUF artifacts.
  • F32, F16, BF16, Q4_K, Q5_0, Q6_K, and Q8_0 tensor loading.
  • NEON, Arm I8MM, and x86_64 AVX2 kernels with portable fallbacks.
  • Greedy text generation, token streaming, profiling, and reproducible evals.
  • OpenAI-compatible models, completions, and chat-completions endpoints.
  • Optional prompt-prefix caching and experimental continuous batching.
  • No model or binary test assets committed to the repository. Test GGUF data is generated in memory, and real-model tests use local artifacts explicitly.

Quick start

Install Rust 1.96.1 through rustup, then build the release binaries:

git clone https://github.com/vicotrbb/ferrite.git
cd ferrite
cargo build --release --locked -p ferrite-cli -p ferrite-server

Place a supported GGUF model under target/models/, then run a completion:

target/release/ferrite \
  --model target/models/qwen2.5-0.5b-instruct-q4_k_m.gguf \
  --prompt 'Write one sentence about Rust.' \
  --generate-tokens 32 \
  --stream

Or start the HTTP server:

target/release/ferrite-server \
  --model target/models/qwen2.5-0.5b-instruct-q4_k_m.gguf \
  --model-id qwen2.5-0.5b-q4_k_m \
  --bind 127.0.0.1:8080 \
  --api-key local-secret
curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'authorization: Bearer local-secret' \
  -H 'content-type: application/json' \
  -d '{"model":"qwen2.5-0.5b-q4_k_m","messages":[{"role":"user","content":"Hello"}],"max_completion_tokens":32}'

For the measured, machine-specific fast path, follow the performance golden path. Do not assume that an experimental kernel is faster or compatible on every CPU.

Documentation

Architecture decisions and curated benchmark evidence live beside the guides under docs/. Raw reproducible eval records remain under scripts/evals/. Transient plans, session notes, private tool state, and model binaries are not repository artifacts.

Repository layout

crates/ferrite-model       GGUF parsing and tokenization
crates/ferrite-inference   model loading, sessions, cache, and CPU kernels
crates/ferrite-cli         local generation, profiling, and benchmarking
crates/ferrite-server      OpenAI-compatible serving and eval clients
crates/ferrite-fixtures    generated test fixtures
docs                       maintained user and contributor documentation
docs/adr                   durable architecture decisions
docs/benchmarks            curated benchmark protocols and milestone evidence
scripts                    eval harness, raw eval records, and repository checks

Quality gates

The required local checks are documented in CONTRIBUTING.md. CI runs formatting, strict Clippy, default and all-feature tests on Linux and macOS, plus a separate Rust 1.96 MSRV check. Linux jobs also enforce rustdoc, doctests, documentation, repository hygiene, eval-harness tests, package contents, RustSec advisories, licenses, duplicate dependencies, and sources.

License

Ferrite is available under the Apache License 2.0.