1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
// SPDX-License-Identifier: MIT OR Apache-2.0
//! # fuel-inference
//!
//! **Layer**: Inference | **Stability**: experimental
//!
//! Inference orchestration for the Fuel ML framework. This crate provides
//! the building blocks for running autoregressive text generation and
//! transformer inference pipelines on top of `fuel-core` and
//! `fuel-transformers`.
//!
//! ## What is here
//!
//! - **KV cache** — [`kv_cache`] re-exports and unifies all cache variants
//! (`Cache`, `KvCache`, `RotatingKvCache`, `ConcatKvCache`, `ScatteredKvCache`)
//! currently living in `fuel-core`.
//! currently living in `fuel-core`.
//! - **Logits processing** — [`generation`] holds `LogitsProcessor`, `Sampling`,
//! and all decode-time logit strategies currently living in `fuel-transformers`.
//! - **Eviction policies** — [`eviction`] provides composable KV cache eviction
//! strategies (LRU, H2O Heavy-Hitter Oracle, weighted voting aggregation).
//! - **Prefix caching** — [`prefix_cache`] provides hash-based KV state reuse
//! for shared prompt prefixes (system prompts, few-shot examples).
//! - **StreamingLLM** — [`streaming`] implements the sink-token + recent-window
//! attention strategy for stable generation beyond the training context window.
//! - **Speculative decoding** — [`speculative`] implements the draft-then-verify
//! parallel token generation algorithm for reduced decode latency.
//! - **Chunked prefill** — [`chunked_prefill`] splits long prompts into bounded
//! chunks to reduce TTFT and allow decode interleaving.
//! - **Segmented eviction** — [`segmented_eviction`] provides span-level KV cache
//! management where logical segments are evicted as complete units.
//! - **KV compression** — [`kv_compress`] provides three orthogonal compression
//! strategies (KIVI quantization, R-KV importance pruning, low-rank approximation).
//! - **Scheduler** — [`scheduler`] is a memory-aware inference scheduler with
//! priority queuing and eviction-pressure admission control.
//! - **MoE routing** — [`moe_routing`] provides capacity-aware top-K token routing
//! for Mixture-of-Experts models.
//! - **Tiered storage** — [`tiered_storage`] tracks KV cache segments across
//! GPU → CPU → Disk tiers with budget-aware demotion/promotion and position
//! ID preservation for RoPE re-injection.
//! - **Context compression** — [`context_compress`] provides token-budget-aware
//! turn selection and compression for conversations exceeding the context window.
//! - **Tool calls** — [`tool_call`] provides structured tool call parsing,
//! validation, dispatch, and result injection for function-calling models.
//! - **Pipelines** — [`pipelines`] is placeholder for future session and
//! batching abstractions.
//!
//! ## What is NOT here
//!
//! - Model definitions (stay in `fuel-transformers`)
//! - Training loop or gradient utilities (use `fuel-training`)
//! - Tokenisation (use `tokenizers` crate directly)
//! - Data loading (use `fuel-datasets`)
//!
//! ## Quick start
//!
//! ```no_run
//! use fuel_inference::generation::{LogitsProcessor, Sampling};
//! use fuel::{Device, lazy::Tensor};
//!
//! # fn main() -> fuel::Result<()> {
//! let device = Device::cpu();
//! let mut lp = LogitsProcessor::new(42, Some(0.7), None);
//!
//! // logits: shape [vocab_size]
//! let logits = Tensor::zeros(32_000usize, fuel::DType::F32, &device)?;
//! // let next_token = lp.sample(&logits)?;
//! # Ok(())
//! # }
//! ```
//!
//! ## Layer placement
//!
//! ```text
//! fuel-inference ← you are here (inference orchestration)
//! fuel-transformers (model definitions)
//! fuel-core (tensors, devices, autograd, AND the NN surface:
//! `fuel_nn::varbuilder::VarBuilder`,
//! `fuel_nn::varmap::VarMap`, and `fuel_nn::modules`
//! — linear, embedding, norm, activation, lora,
//! quantizable_linear, moe, conv, sequential)
//! ```
//!
//! Note: a separate `fuel-nn` crate is **planned, not shipped** — there is no
//! such crate in the workspace today, and the NN surface lives in `fuel-core`
//! as shown above. Extracting it is a step of the `fuel-core` retirement
//! program. (Corrected 2026-07-29: the previous layer diagram listed
//! `fuel-nn` as an existing layer, which sent a consumer looking for a crate
//! that does not exist. See `docs/architecture/02-layers.md`.)
//!
//! Nothing in `fuel-core` or `fuel-transformers` depends on
//! this crate. It is a leaf that aggregates; it does not define.
// ── Re-export inference building blocks ───────────────────────────────────
//
// `kv_cache` and `sampling` now live in `fuel-core` (no fuel-nn dep
// required here). `generation` lives in `fuel-transformers` (moving it here
// would require fuel-transformers to depend on fuel-inference, violating
// the leaf-crate principle). `fuel-inference` aggregates; it does not define.
/// Token-sampling and logit-processing strategies for autoregressive decode.
///
/// Re-exported from `fuel_transformers::generation`.
/// KV cache implementations for efficient transformer inference.
///
/// Re-exported from `fuel-core`. The eager `fuel::kv_cache` module
/// retired with the eager-Tensor program (Phase β, `1ab1d0c9`); the
/// pipelined-era equivalents are `fuel::lazy_kv_cache::LazyKvCache`
/// (graph-level KV state) and `fuel::inference_context::KvCache`
/// (persistent per-context storage on the production executor).
// ── Native inference modules ──────────────────────────────────────────────
/// Speculative decoding: draft-then-verify parallel token generation.
/// Memory-aware inference scheduler with priority queuing and
/// eviction-pressure admission control.
/// Placeholder for future batching, streaming-decode, and session abstractions.