Module engine_pool 

Engine-replica pool for concurrent serving.

The HTTP server historically wrapped a single InferenceEngine in a tokio::sync::Mutex, which serialized every request: a long generation blocked all others for its full duration. This module replaces that single mutex with a pool of N engine replicas guarded by a semaphore, so up to N requests can generate concurrently.

Why a pool works (and is safe)

• Weights are process-global and immutable: InferenceEngine::from_gguf_path leaks the mmap and parsed GgufFile to 'static, so every replica borrows the same &'static GgufFile zero-copy. The immutable, dequantized token_embd table is likewise shared across replicas via one Arc<[f32]> (see build_pool_from_gguf). Only the per-replica KvCache and light wrappers are duplicated.
• On CPU tiers (Reference / AVX / NEON), BonsaiModel::forward mutates only self.kv_cache over a shared &dyn OneBitKernel on immutable weights, so distinct engine instances run fully parallel.
• On the GPU tier (Metal / CUDA), decode funnels through a process-global singleton graph owning one KV cache and shared scratch. N > 1 GPU replicas give no compute parallelism and would corrupt each other’s KV, so the pool size is clamped to 1 on the GPU tier (see resolve_pool_size).

Back-compatibility

The default path wraps exactly one engine in a 1-element pool whose EngineLease calls the identical generate* methods on the identical engine. Single-request behavior — including RNG progression — is therefore byte-identical to the previous single-mutex design.

Lock discipline

idle is a synchronous std::sync::Mutex held only for the duration of a pop/push — never across an .await. Async waiting happens purely on the tokio::sync::Semaphore, whose permit count equals the pool size.

Structs

EngineLease

An exclusive lease on one engine from an EnginePool.

EnginePool

A pool of InferenceEngine replicas guarded by a semaphore.

Enums

PoolError

Errors that can occur while acquiring an engine from the pool.

Functions

build_pool_from_gguf

Build an EnginePool from a GGUF file, sizing it for the detected tier.

default_cpu_pool_size

Default pool size on CPU tiers: the host’s available parallelism, capped at 4, with a floor of 1.

resolve_pool_size

Resolve the effective pool size for a given kernel tier.