pub async fn run_queue_dispatcher(
job_rx: Receiver<GenerationJob>,
state: AppState,
)Expand description
Runs the multi-GPU dispatch loop. Routes each generation job to the best
GPU worker based on the placement strategy (model-loaded > idle > evict LRU).
Uses a small lookahead buffer so an interleaved queue ([A, B, A, B])
doesn’t force a sibling worker to swap models when one already has the
right one warm.
Exits when the sender half of the channel is dropped (server shutdown).