ferrox-server 0.14.0

OpenAI-compatible HTTP server for the Ferrox inference engine
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

//! Running a request: admitting it, prefilling it, decoding it, and
//! deciding where its answer ends.

/// Admission and chunked prefill for the WINDOW and RECURRENT memory
/// models, which [`batch`] excludes.
///
/// Wired: nothing yet -- `PrefillPass`, `SlotTable`, `Capacity`,
/// `Geometry`, `PromptAdmission`, `NotAdmitted` and `FinishReason` are
/// the largest unwired block left in this crate, and they are one
/// roadmap item: `sched-time-debt` (roadmap `c3-serving-and-kv`), whose
/// quantum is chunk DURATION because a GPU cannot preempt a running
/// kernel.
#[allow(dead_code)]
pub(crate) mod admission;

pub(crate) mod batch;