Crate cortiq_server

Expand description

Cortiq Server — OpenAI-compatible API + web management dashboard.

Modules§

api: Cortiq extension API endpoints.
dashboard: Web management dashboard — served as embedded static content.
openai: OpenAI-compatible API endpoints, backed by the real inference pipeline. Generation runs in spawn_blocking behind a Mutex — a panic inside the pipeline becomes a 500, never a dead process.
streaming: SSE (Server-Sent Events) streaming for OpenAI-compatible chat completions.

AppState: Shared application state: runtime (masks, metrics), a tokenizer handle that never blocks on generation, and the slot pool.
PipelinePool: Fixed pool of pipeline slots over ONE shared mmap’d model (roadmap §3 «serving полностью сериализован», этап 5.1): the weights are zero-copy shared through Arc<CmfModel>, each slot owns its KV-cache / recurrent state / sampler / workspace. A request checks a slot out for the duration of one generation, so up to slots requests decode CONCURRENTLY; excess requests queue fairly on the semaphore. This is bounded-concurrency serving, not yet continuous batching (этап 5.2+).
SlotGuard: A checked-out slot: holds both the concurrency permit and the pipeline lock until dropped.