Skip to main content

Module scheduler

Module scheduler 

Source

Structsยง

ContinuousBatchScheduler
High-throughput continuous batch scheduler executing iteration-level speculative inference. Dynamically pools multiple concurrent requests into unified forward passes.
QueuedRequest
An incoming inference request submitted to the batch scheduler.
SchedulerStats
Real-time operational statistics of the continuous batch scheduler.