Struct UnifiedBatch 

pub struct UnifiedBatch {
    pub items: Vec<UnifiedBatchItem>,
}

A mixed-batch forward request: any combination of in-progress prefill chunks and decode steps. See UnifiedBatchItem for the per-item semantics. The producer (engine) groups all sequences active in this iter into a single batch; the consumer (model) runs one forward and returns per-item logits (only for items with is_final_chunk = true, in the order they appear in items).

Fields

items: Vec<UnifiedBatchItem>

Implementations

impl UnifiedBatch

pub fn new() -> Self

pub fn total_q_tokens(&self) -> usize

Total query tokens across all items — corresponds to the M dim of the model’s per-layer GEMMs in the unified forward.

pub fn num_sampled_items(&self) -> usize

Number of items that will produce a logits vector (decode items always; prefill items only on their final chunk).

Trait Implementations

impl Clone for UnifiedBatch

fn clone(&self) -> UnifiedBatch

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for UnifiedBatch

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for UnifiedBatch

fn default() -> UnifiedBatch

Returns the “default value” for a type.