Dispatch gate shared through AppState, toggled by POST /api/queue/pause
and POST /api/queue/resume. When paused the dispatch loops stop pulling
new jobs off the channel; the job already running on a worker finishes
untouched. Cheap to poll (a single relaxed-ish atomic) so it can sit at the
top of every loop iteration.
Runs the multi-GPU dispatch loop. Routes each generation job to the best
GPU worker based on the placement strategy (model-loaded > idle > evict LRU).
Uses a small lookahead buffer so an interleaved queue ([A, B, A, B])
doesn’t force a sibling worker to swap models when one already has the
right one warm.
Runs the generation queue worker loop. Processes one job at a time (FIFO),
but uses a small bounded lookahead buffer to prefer jobs whose model is
already loaded — minimizing model swaps when the queue interleaves models.
Exits when the sender half of the channel is dropped (server shutdown).