Structsยง
- Continuous
Batch Scheduler - High-throughput continuous batch scheduler executing iteration-level speculative inference. Dynamically pools multiple concurrent requests into unified forward passes.
- Queued
Request - An incoming inference request submitted to the batch scheduler.
- Scheduler
Stats - Real-time operational statistics of the continuous batch scheduler.