Struct BatchRequest 

pub struct BatchRequest {
    pub id: u64,
    pub prompt_tokens: Vec<u32>,
    pub params: SamplingParams,
    pub max_tokens: usize,
    pub priority: RequestPriority,
    pub state: RequestState,
    pub generated_tokens: Vec<u32>,
    pub created_at: Instant,
    pub started_at: Option<Instant>,
    pub completed_at: Option<Instant>,
}

A single inference request managed by the continuous-batch scheduler.

Fields

id: u64

Unique request identifier returned by ContinuousBatchScheduler::submit.

prompt_tokens: Vec<u32>

Tokenised prompt.

params: SamplingParams

Sampling parameters for this request.

max_tokens: usize

Maximum number of tokens to generate.

priority: RequestPriority

Scheduling priority.

state: RequestState

Current lifecycle state.

generated_tokens: Vec<u32>

Tokens generated so far (not including the prompt).

created_at: Instant

Wall-clock time at which the request was submitted.

started_at: Option<Instant>

Wall-clock time at which the first token was generated (prefill complete).

completed_at: Option<Instant>

Wall-clock time at which generation finished.

Implementations

impl BatchRequest

pub fn new( id: u64, prompt_tokens: Vec<u32>, params: SamplingParams, max_tokens: usize, ) -> Self

Create a new request with Normal priority.

pub fn with_priority(self, priority: RequestPriority) -> Self

Override the priority, returning self for builder-style chaining.

pub fn time_to_first_token(&self) -> Option<Duration>

Elapsed time from submission to first generated token.

Returns None if the first token has not yet been produced.

pub fn total_latency(&self) -> Option<Duration>

Elapsed time from submission to completion.

Returns None if the request has not yet completed.

pub fn tokens_generated(&self) -> usize

Number of tokens generated so far.

pub fn is_finished(&self) -> bool

true when the request is in RequestState::Completed or RequestState::Failed.