Struct InferenceQueue 

pub struct InferenceQueue { /* private fields */ }

High-level inference request queue wrapping BoundedQueue<InferenceWorkItem>.

Each call to InferenceQueue::submit returns a std::sync::mpsc::Receiver through which the caller can retrieve the generated token IDs once inference completes. Returns None immediately if the queue is full (backpressure).

Implementations

impl InferenceQueue

pub fn new(capacity: usize) -> Self

Create a new inference queue with the given maximum pending-request capacity.

pub fn submit( &self, prompt_tokens: Vec<u32>, max_tokens: usize, params: SamplingParams, ) -> Option<Receiver<Vec<u32>>>

Submit an inference request.

Returns a Receiver that will yield the generated token IDs once the request is processed, or None if the queue is full.

pub fn queue_depth(&self) -> usize

Number of pending requests currently in the queue.

pub fn is_full(&self) -> bool

true if the queue is at capacity.

pub fn stats(&self) -> QueueStats

Take a statistics snapshot.