Skip to main content

MAX_TOTAL_LLM_WORKERS

Constant MAX_TOTAL_LLM_WORKERS 

Source
pub const MAX_TOTAL_LLM_WORKERS: usize = 64;
Expand description

DEFAULT joint ceiling on max_concurrency × llm_parallelism for one host.

The two knobs are validated independently — --max-concurrency against 2 × nCPUs and --llm-parallelism against 32 — so nothing used to stop their PRODUCT from authorising 2 × nCPUs × 32 in-flight workers, which on a 16-core host is 1024. This constant is the missing joint bound; the per-knob ceilings stay exactly as they are and this one only clamps the product.

Read it through max_total_llm_workers, never directly.