Expand description
Per-model inference concurrency pools.
A single InferencePools belongs to the world and bounds how many
inference requests are in flight to each model at once - e.g. “at most 3
concurrent requests to anthropic:claude-opus-4-8”, “at most 1 to a local
ollama:gemma”. This is the world-level control the ECS inference-dispatch
system consults before issuing a request: an agent only leaves ReadyToInfer
once a permit for its model is available; otherwise it stays ready and is
retried on a later tick (so “waiting for a slot” costs nothing but data).
Why this matters: a single inference can take up to an hour for very large requests, so a permit may legitimately be held for a very long time - the pool is what keeps us from opening an unbounded number of simultaneous long-lived requests to a provider.
Distinct from a blueprint’s fan-out max_workers (which bounds a stage’s
sub-agent fan width); these pools bound total in-flight inferences per
model across every agent in the world.
Structs§
- Inference
Permit - An RAII permit occupying one slot of a model’s inference pool. Dropping it frees the slot for the next waiting agent.
- Inference
Pool Config - Configuration for the world’s per-model inference concurrency limits.
- Inference
Pools - The world’s live per-model inference pools. Cheap to clone-share behind an
Arc; semaphores are created lazily the first time a model is seen.