Crate ferrox_server 

ferrox-server: OpenAI-compatible HTTP surface (/health, /v1/models, /v1/chat/completions, /v1/completions, /v1/tokenize, /v1/detokenize, /v1/embeddings) over the ferrox-models decoder, plus a whole-response cache for exact-repeat requests (see cache module). Loads a real GGUF checkpoint and its own real tokenizer when -m/--model or FERROX_MODEL_PATH is set (see model module). Supports sampling (temperature/top_p/top_k/repetition_penalty), stop sequences, and SSE streaming (see generate module).

Concurrency: the loaded model (Model) is immutable once loaded and shared via Arc, not locked behind a Mutex – there is no shared mutable decoder state for concurrent requests to contend on or for one panicking request to poison. The pointer to it is swappable (AppState::active, behind an RwLock held only long enough to clone one Arc), which is what /admin/models/load swaps; a request that has already cloned its handle finishes against the exact weights it started on, and the old model is freed when the last such request lets go. Each request builds its own KV cache (see generate::generate) and runs its decode loop on tokio’s blocking-thread pool via spawn_blocking, so CPU-bound generation no longer blocks the async reactor threads – multiple requests can decode genuinely concurrently, bounded by that pool rather than serialized through one lock. Only the small whole-response cache is still mutable shared state, and it’s locked only for the brief get/put around it, never across a decode.

Streaming scope: when stream: true and tools are inactive, each decoded chunk is pushed through a bounded mpsc channel from the blocking generate task into the SSE writer so time-to-first-byte overlaps with ongoing decode. Under continuous batching the batch worker emits the same incremental chunks as the private decode loop.

Structs

ServerArgs

Constants

BUILT_WITH_CUDA

Whether this build of the server has the CUDA kernels compiled in. See BUILT_WITH_METAL.

BUILT_WITH_METAL

Whether this build of the server has the Metal kernels compiled in.

Functions

run_server

Runs the server to completion.