Struct InferenceRequest 

pub struct InferenceRequest {
    pub request_id: RequestId,
    pub sequence_length: usize,
    pub max_new_tokens: usize,
    pub priority: Priority,
    pub arrival_time_ns: u64,
    pub deadline_ns: Option<u64>,
}

An incoming inference request.

Fields

request_id: RequestId

Unique request identifier.

sequence_length: usize

Number of input (prompt) tokens.

max_new_tokens: usize

Maximum number of tokens to generate.

priority: Priority

Scheduling priority.

arrival_time_ns: u64

Monotonic arrival timestamp in nanoseconds.

deadline_ns: Option<u64>

Optional hard deadline in nanoseconds (absolute).

Trait Implementations

impl Clone for InferenceRequest

fn clone(&self) -> InferenceRequest

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for InferenceRequest

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.