ferrox-inference

Facade crate for Ferrox, a pure-Rust GGUF / MoE inference engine: mmap loaders, quantized CPU + Apple Metal + CUDA kernels, and an OpenAI-compatible server.

It contains no logic. It re-exports the workspace under one name, so a dependent writes one line instead of six, and so the project is findable on crates.io, since the name ferrox belongs to an unrelated crate.

[dependencies]
ferrox-inference = "0.10"

use ferrox_inference::gguf::ShardedGguf;

let file = ShardedGguf::open("model.gguf")?;

The binaries are elsewhere

cargo install ferrox-cli      # installs the `ferrox` binary
cargo install ferrox-server   # OpenAI-compatible HTTP server

They are not shipped from this crate on purpose: two crates installing a binary called ferrox would fight over the same path in ~/.cargo/bin.

Features

Neither GPU feature is on by default. metal does not build off Apple Silicon, and cuda needs a toolkit most machines do not have.

The bar CUDA is held to is "must compile". There is no pinned benchmark host for it and no published timings, so treat a Windows or Linux install as CPU-only in practice. See docs/FEATURES.md.

The rest of the workspace

Speed claims live in benchmarks/RESULTS.md, measured against llama.cpp on the same host and the same GGUF. That table is generated from the raw timing files each run writes, and it says so wherever nothing has been measured.

Apache-2.0.