Struct BatchHint 

pub struct BatchHint {
    pub max_batch_size: usize,
    pub max_tokens: usize,
    pub target_latency_ms: Option<u64>,
    pub available_memory: Option<u64>,
    pub resource_constraints: ResourceConstraints,
}

Batch hint for scheduler optimization

Fields

max_batch_size: usize

Maximum batch size

max_tokens: usize

Maximum total tokens in batch

target_latency_ms: Option<u64>

Target latency for batch formation

available_memory: Option<u64>

Available memory for batch

resource_constraints: ResourceConstraints

Resource constraints

Implementations

impl BatchHint

pub fn simple(max_batch_size: usize) -> Self

Create simple batch hint with size limit

Trait Implementations

impl Clone for BatchHint

fn clone(&self) -> BatchHint

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for BatchHint

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.