Skip to main content

run_chain_blocking

Function run_chain_blocking 

Source
pub fn run_chain_blocking<T, E>(
    worker: &GpuWorker,
    model_name: &str,
    config: &Config,
    hint: Option<ActivationHint>,
    with_engine: impl FnOnce(&mut dyn InferenceEngine) -> Result<T, E>,
) -> ChainPrep<T, E>
Expand description

Run a blocking chain operation on a specific GPU worker.

Acquires worker.model_load_lock for the full duration, binds the current thread to worker.gpu.ordinal (so reclaim_gpu_memory debug asserts are satisfied), ensures the model is loaded on GPU, takes the engine out of the worker’s cache, passes it to with_engine, and restores the engine unconditionally on both success and closure failure.

Safe to call from inside tokio::task::spawn_blocking. The calling thread can be any thread — the ThreadGpuGuard clears the thread-local on return.

§Errors

Returns Err(anyhow::Error) from the outer Result if:

  • ensure_model_ready_sync fails (bad config, disk IO, load error).
  • The engine vanishes from the cache between ensure and take (cache race).

Returns Ok(Err(E)) if the closure itself returned an error — caller preserves the closure’s typed error for precise HTTP status mapping.