ferrox-inference 0.9.1

Pure-Rust GGUF / MoE inference engine: mmap loaders, quantized CPU + Metal + CUDA kernels
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90

//! Ferrox — a pure-Rust GGUF / MoE inference engine.
//!
//! This crate is a **facade**. It contains no logic of its own: it
//! re-exports the workspace under one name so a dependent writes one
//! line in `Cargo.toml` instead of six, and so the project is findable
//! on crates.io (the name `ferrox` belongs to an unrelated crate).
//!
//! The command-line tools are not here. `cargo install ferrox-cli`
//! installs the `ferrox` binary; `ferrox-server` is the
//! OpenAI-compatible HTTP server. Shipping a second binary called
//! `ferrox` from this crate would just fight the first one over
//! `~/.cargo/bin`.
//!
//! # Layout
//!
//! The stack, bottom to top:
//!
//! | Module | Crate | What it is |
//! |---|---|---|
//! | [`gguf`] | `ferrox-gguf` | GGUF mmap reader, sharded checkpoints |
//! | [`quant`] | `ferrox-quant` | Block layouts and fused dequant+dot |
//! | [`safetensors`] | `ferrox-safetensors` | SafeTensors mmap reader |
//! | [`core`] | `ferrox-core` | Tensor ops, RoPE, GQA, KV cache |
//! | [`moe`] | `ferrox-moe` | Expert routing and dispatch |
//! | [`models`] | `ferrox-models` | Loaders and decoder stacks |
//! | [`api`] | `ferrox-api` | Route constants + wire DTOs (feature `api`) |
//!
//! # Example
//!
//! ```no_run
//! use ferrox_inference::gguf::ShardedGguf;
//!
//! let file = ShardedGguf::open("model.gguf")?;
//! println!("{} tensors", file.tensor_count());
//! # Ok::<(), Box<dyn std::error::Error>>(())
//! ```
//!
//! # Features
//!
//! - `metal` — Apple Metal kernels. Apple Silicon only.
//! - `cuda` — CUDA/NVRTC kernels. Needs a CUDA toolkit at build time.
//!   Held to "must compile": there is no pinned benchmark host and no
//!   published receipts for it. See `docs/FEATURES.md`.
//! - `api` — pull in `ferrox-api` for client-side route constants.
//!
//! Neither GPU feature is on by default, because both are wrong to
//! assume: `metal` does not build off Apple Silicon and `cuda` needs a
//! toolkit that most machines do not have.

#![forbid(unsafe_code)]

pub use ferrox_core as core;
pub use ferrox_gguf as gguf;
pub use ferrox_models as models;
pub use ferrox_moe as moe;
pub use ferrox_quant as quant;
pub use ferrox_safetensors as safetensors;

#[cfg(feature = "api")]
pub use ferrox_api as api;

/// The workspace version this facade was built from.
///
/// Every crate in the workspace shares one version, so this is the
/// version of the whole engine, not just of the facade.
pub const VERSION: &str = env!("CARGO_PKG_VERSION");

#[cfg(test)]
mod tests {
    /// The facade is only useful if it actually re-exports. These
    /// paths would fail to COMPILE -- not merely assert false -- if a
    /// module were dropped from `lib.rs`, which is the failure mode
    /// worth catching: a facade that silently stops re-exporting one
    /// layer looks fine until a dependent upgrades and cannot build.
    #[allow(dead_code)]
    fn every_layer_is_reachable_through_the_facade() {
        let _: Option<super::gguf::ShardedGguf> = None;
        let _: Option<super::quant::QuantError> = None;
        let _: Option<super::core::cache::KvCache> = None;
        let _: Option<super::safetensors::SafetensorsFile> = None;
        let _ = std::mem::size_of::<super::models::Decoder>();
        let _ = std::mem::size_of::<super::moe::MoeLayerConfig>();
    }

    #[test]
    fn version_is_the_workspace_version() {
        assert_eq!(super::VERSION, env!("CARGO_PKG_VERSION"));
        assert!(!super::VERSION.is_empty());
    }
}