Skip to main content

run_queue_worker

Function run_queue_worker 

Source
pub async fn run_queue_worker(job_rx: Receiver<GenerationJob>, state: AppState)
Expand description

Runs the generation queue worker loop. Processes one job at a time (FIFO), but uses a small bounded lookahead buffer to prefer jobs whose model is already loaded — minimizing model swaps when the queue interleaves models. Exits when the sender half of the channel is dropped (server shutdown).