Skip to main content

dispatch_inference

Function dispatch_inference 

Source
pub fn dispatch_inference(
    agents: Query<'_, '_, (Entity, &AgentState, &ContextWindow, Option<&InferenceConfig>, &StageInference, Option<&InFlightWork>, Option<&StageProgress>), With<ReadyToInfer>>,
    stage: Res<'_, InferenceStage>,
    providers: Res<'_, Providers>,
    par_commands: ParallelCommands<'_, '_>,
)
Expand description

Inference-dispatch system: for every ReadyToInfer agent, resolve its provider and, if a per-model permit is free, build the request, spawn the inference job, and move it to AwaitingInference. If its provider is missing or no slot is free, it stays ReadyToInfer and is retried on a later tick - no blocking, no wasted task.