Expand description
Continuous Batching Engine
Iteration-level continuous batching: each step processes a mixed batch of
prefill and decode requests selected by the scheduler. Multiple callers
can submit requests concurrently. An iteration_lock makes publication
atomic with planning, while the single background driver owns device-wave
execution.
Structsยง
- Continuous
Batch Engine - Continuous batching inference engine.
- Sequence
State - State of a running sequence in the continuous batch.