Struct GpuWorker 

pub struct GpuWorker {
    pub gpu: DiscoveredGpu,
    pub model_cache: Arc<Mutex<ModelCache>>,
    pub active_generation: Arc<RwLock<Option<ActiveGeneration>>>,
    pub model_load_lock: Arc<Mutex<()>>,
    pub shared_pool: Arc<Mutex<SharedPool>>,
    pub in_flight: AtomicUsize,
    pub consecutive_failures: AtomicUsize,
    pub poisoned: AtomicBool,
    pub fatal_cuda_error: Arc<AtomicBool>,
    pub fatal_cuda_shutdown: Arc<Notify>,
    pub degraded_until: RwLock<Option<Instant>>,
    pub job_tx: SyncSender<GpuJob>,
}

Per-GPU worker state. Each GPU gets its own model cache, load lock, and health tracking.

Fields

gpu: DiscoveredGpu

model_cache: Arc<Mutex<ModelCache>>

active_generation: Arc<RwLock<Option<ActiveGeneration>>>

model_load_lock: Arc<Mutex<()>>

shared_pool: Arc<Mutex<SharedPool>>

in_flight: AtomicUsize

consecutive_failures: AtomicUsize

poisoned: AtomicBool

Fatal CUDA errors poison a process-owned context permanently. A poisoned worker is quarantined until the server process is restarted.

fatal_cuda_error: Arc<AtomicBool>

Shared process-level signal. A fatal context requires process teardown; supervised servers restart after run_server returns an error.

fatal_cuda_shutdown: Arc<Notify>

degraded_until: RwLock<Option<Instant>>

job_tx: SyncSender<GpuJob>

Implementations

impl GpuWorker

pub fn is_degraded(&self) -> bool

Check if this worker is in a degraded state (3+ consecutive failures, within cooldown).

When the cooldown has expired we clear the failure counter and the degraded_until timestamp so the next single failure doesn’t immediately re-degrade the GPU. Without this lazy reset, a worker that has 3 historical failures followed by a long idle period would flip back to Degraded on the very first post-cooldown failure (because consecutive_failures is still >= 3 from before).

pub fn status(&self) -> GpuWorkerStatus

Build a status snapshot for this worker.