ferrox-inference 0.9.1

Pure-Rust GGUF / MoE inference engine: mmap loaders, quantized CPU + Metal + CUDA kernels
Documentation

ferrox-inference

Facade crate for Ferrox, a pure-Rust GGUF / MoE inference engine: mmap loaders, quantized CPU + Apple Metal + CUDA kernels, and an OpenAI-compatible server.

It contains no logic. It re-exports the workspace under one name, so a dependent writes one line instead of six, and so the project is findable on crates.io, since the name ferrox belongs to an unrelated crate.

[dependencies]
ferrox-inference = "0.9"

use ferrox_inference::gguf::ShardedGguf;

let file = ShardedGguf::open("model.gguf")?;

The binaries are elsewhere

cargo install ferrox-cli      # installs the `ferrox` binary
cargo install ferrox-server   # OpenAI-compatible HTTP server

They are not shipped from this crate on purpose: two crates installing a binary called ferrox would fight over the same path in ~/.cargo/bin.

Features

Feature Effect
metal Apple Metal kernels. Apple Silicon only.
cuda CUDA/NVRTC kernels. Needs a CUDA toolkit at build time.
api Re-export ferrox-api (route constants + wire DTOs).

Neither GPU feature is on by default: metal does not build off Apple Silicon, and cuda needs a toolkit most machines do not have.

CUDA is held to "must compile", there is no pinned benchmark host and no published receipts for it. Treat a Windows or Linux install as CPU-only in practice. See docs/FEATURES.md.

The rest of the workspace

Crate What it is
ferrox-gguf GGUF mmap reader, sharded checkpoints
ferrox-quant Block layouts, fused dequant+dot
ferrox-safetensors SafeTensors mmap reader
ferrox-core Tensor ops, RoPE, GQA, KV cache
ferrox-moe Expert routing and dispatch
ferrox-models Loaders and decoder stacks
ferrox-api Route constants + wire DTOs
ferrox-metal Apple Metal kernels
ferrox-cuda CUDA/NVRTC kernels

Speed claims live in benchmarks/RESULTS.md, measured against llama.cpp on the same host and GGUF. If there is no receipt, the table says so.

Apache-2.0.