Struct BatchConstructor 

pub struct BatchConstructor {
    pub max_tokens_per_batch: usize,
    pub max_entries: usize,
}

Constructs a continuous batch: pulls pending decode requests first (cheap, one-token forwards), then fills remaining token budget with prefill chunks. The token budget — not the sequence count — is the gating constraint because prefills can be arbitrarily long.

Fields

max_tokens_per_batch: usize

Maximum tokens per forward across all entries.

max_entries: usize

Maximum entries per forward (also bounds memory/scheduler overhead).

Implementations

impl BatchConstructor

pub fn new(max_tokens_per_batch: usize, max_entries: usize) -> Self

pub fn build( &self, decode_queue: &mut VecDeque<BatchEntry>, prefill_queue: &mut VecDeque<BatchEntry>, ) -> Vec<BatchEntry>

Build the next batch. Walks decode_queue first (one token each, cheap), then fills remaining token budget by chunking from prefill_queue. Sequences that didn’t fit stay in their queues for the next call.