Struct ContinuousBatchScheduler 

pub struct ContinuousBatchScheduler {
    pub max_concurrent: usize,
    pub max_queue_size: usize,
    /* private fields */
}

Continuous-batch scheduler.

Manages request lifecycle from submission through generation to completion, interleaving multiple requests at the token level.

Fields

max_concurrent: usize

Maximum number of requests decoding simultaneously.

max_queue_size: usize

Maximum number of requests that may wait in the queue.

Implementations

impl ContinuousBatchScheduler

pub fn new(max_concurrent: usize, max_queue_size: usize) -> Self

Create a new scheduler.

max_concurrent — at most this many requests decode in parallel. max_queue_size — queue rejects new submissions beyond this count.

pub fn submit( &mut self, prompt_tokens: Vec<u32>, params: SamplingParams, max_tokens: usize, ) -> Result<u64, SchedulerError>

Submit a request with Normal priority.

Returns the assigned request ID, or SchedulerError::QueueFull if the waiting queue is already at capacity.

pub fn submit_with_priority( &mut self, prompt_tokens: Vec<u32>, params: SamplingParams, max_tokens: usize, priority: RequestPriority, ) -> Result<u64, SchedulerError>

Submit a request with an explicit priority.

pub fn step(&mut self, engine: &mut InferenceEngine<'_>)

Advance one iteration of the batch.

1. Promotes waiting requests into the active set until max_concurrent slots are full (or the queue is drained).
2. Steps every active request by generating one token.
3. Moves finished requests to the completed list.

pub fn run_to_completion(&mut self, engine: &mut InferenceEngine<'_>)

Run all pending and active requests to completion, blocking until the scheduler is idle.

pub fn get_result(&self, id: u64) -> Option<&BatchRequest>

Look up a completed (or failed) request by ID.

Returns None if the request is still waiting/decoding or does not exist.

pub fn queue_depth(&self) -> usize

Number of requests currently waiting in the queue.

pub fn active_count(&self) -> usize

Number of requests currently being decoded.

pub fn completed_count(&self) -> usize

Number of completed (or failed) requests.

pub fn is_idle(&self) -> bool

true when there are no waiting or active requests.

pub fn throughput_stats(&self) -> SchedulerStats

Snapshot of current throughput statistics.

pub fn drain_completed(&mut self) -> Vec<BatchRequest>

Remove and return all completed requests, clearing the completed list.