Struct EngineLease 

pub struct EngineLease { /* private fields */ }

An exclusive lease on one engine from an EnginePool.

Derefs (mutably) to the borrowed InferenceEngine, so callers invoke the usual generate* methods directly. On drop, the engine is returned to the pool and the semaphore permit is released — in that order, so a waiter is guaranteed to find an idle engine the instant its permit is granted.

The engine is stored in ManuallyDrop so Drop can move it back into the pool without an Option/unwrap dance: this is the whole reason the guard is panic-free.

Methods from Deref<Target = InferenceEngine<'static>>

pub fn set_metrics(&mut self, metrics: Arc<InferenceMetrics>)

Attach shared metrics to this engine for recording inference telemetry.

pub fn set_rate_aggregator(&mut self, aggregator: Arc<RequestRateAggregator>)

Attach a workload-level RequestRateAggregator to this engine.

Once attached, every call to InferenceEngine::generate_tracked (or InferenceEngine::generate_with_request_id) will push its per-request RequestRateSnapshot into the aggregator on completion. The aggregator is reference-counted, so the same instance can be shared with the Prometheus metrics layer or the admin endpoints.

pub fn rate_aggregator(&self) -> Option<&Arc<RequestRateAggregator>>

Read-only access to the attached rate aggregator, if any.

pub fn model(&self) -> &BonsaiModel<'a>

Get a reference to the model.

pub fn model_token_embd(&self) -> Arc<[f32]>

Cheaply clone a handle to this engine’s shared token-embedding table.

Thin delegate to BonsaiModel::shared_token_embd. The engine pool calls this on replica #1 to extract the one shared Arc<[f32]>, which it then hands to InferenceEngine::from_gguf_static_with_embd when building replicas 2..N — so every replica’s token_embd is a clone of the same allocation (one ~1.16 GiB table for the 1.7B, not N).

pub fn model_mut(&mut self) -> &mut BonsaiModel<'a>

Get a mutable reference to the model.

Used by the prefix-cache integration to inject restored KV blocks before running the abbreviated prefill.

pub fn kernel(&self) -> &KernelDispatcher

Get a reference to the kernel dispatcher.

pub fn kernel_tier(&self) -> KernelTier

Kernel tier this engine dispatches to.

Feature-agnostic convenience used by the engine pool to decide how many replicas may safely run in parallel (GPU tiers funnel through a process-global singleton and are pinned to a single replica).

pub fn prefill_from_pos( &mut self, prompt_tokens: &[u32], pos_start: usize, ) -> RuntimeResult<Vec<f32>>

Run prefill at a given KV-cache offset.

Unlike InferenceEngine::generate, this does not reset the model’s KV cache before execution: callers (e.g. the prefix-cache engine) are expected to have prepared the cache state explicitly.

Increments the prefill_token_count counter by prompt_tokens.len() on success.

pub fn decode_step(&mut self, token: u32, pos: usize) -> RuntimeResult<Vec<f32>>

Forward one token at the given absolute position.

pub fn sample(&mut self, logits: &[f32]) -> RuntimeResult<u32>

Sample one token from logits using the engine’s current sampler.

pub fn prefill_token_count(&self) -> u64

Cumulative number of tokens that have been processed by InferenceEngine::prefill_from_pos over this engine’s lifetime.

pub fn reset(&mut self)

Reset the model state for a new conversation.

pub fn stats(&self) -> &Arc<EngineStats>

Get a shared reference to the engine statistics.

pub fn active_sessions(&self) -> usize

Number of currently active sessions (tracked via stats).

pub fn session_count(&self) -> u64

Total number of completed requests (tracked via stats).

pub fn batch_generate( &mut self, prompts: &[Vec<u32>], max_tokens: usize, ) -> Vec<RuntimeResult<BatchResult>>

Process a batch of prompts, delegating to batch_engine::batch_generate.

Resets the engine state between each prompt. Returns one result per prompt.

pub fn generate( &mut self, prompt_tokens: &[u32], max_tokens: usize, ) -> RuntimeResult<Vec<u32>>

Generate tokens from a prompt.

Runs prefill (process the entire prompt), then decodes token by token until max_tokens or EOS is reached. Returns the generated token IDs (not including the prompt).

pub fn generate_tracked( &mut self, prompt_tokens: &[u32], max_tokens: usize, tracker: &mut RequestRateTracker, ) -> RuntimeResult<Vec<u32>>

Generate tokens from a prompt while populating a RequestRateTracker.

Behaves identically to InferenceEngine::generate but additionally:

• records record_admission() immediately on entry,
• records record_first_token() for the first sampled token,
• records record_token() for every subsequent sampled token,
• on success, pushes the resulting RequestRateSnapshot into the engine’s attached RequestRateAggregator (if any).

The tracker is borrowed mutably so callers can inspect intermediate state via RequestRateTracker::snapshot after the call returns.

pub fn generate_with_request_id( &mut self, request_id: RequestId, prompt_tokens: &[u32], max_tokens: usize, ) -> RuntimeResult<(Vec<u32>, RequestRateTracker)>

Generate tokens from a prompt with a RequestId tagging the surrounding tracing span and an internally-managed RequestRateTracker.

Returns both the generated tokens and the final tracker so callers can extract per-request metrics (e.g. queue-wait, p95 inter-token latency) for client-side observability.

pub fn generate_with_seed( &mut self, prompt_tokens: &[u32], max_tokens: usize, seed: u64, params: &SamplingParams, ) -> RuntimeResult<Vec<u32>>

Generate tokens from a prompt using a specific seed for this run.

Temporarily overrides the sampler seed for deterministic multi-completion generation (n > 1). The sampler state is replaced for the duration of this call and then restored.

pub fn generate_with_params( &mut self, prompt_tokens: &[u32], max_tokens: usize, params: &SamplingParams, ) -> RuntimeResult<Vec<u32>>

Generate tokens from a prompt using caller-supplied sampling parameters for the duration of this call only.

Swaps in params (temperature, top-k, top-p, repetition penalty) on the engine’s existing sampler, runs InferenceEngine::generate, then restores the previous parameters. Crucially, the sampler’s PRNG state is not reset — only the parameters change — so the RNG sequence for the next request is identical to what it would have been had this call used the engine’s default parameters. This makes the default-parameter case bit-identical to calling generate directly.

pub fn generate_streaming( &mut self, prompt_tokens: &[u32], max_tokens: usize, tx: &UnboundedSender<u32>, ) -> RuntimeResult<usize>

Generate tokens one at a time, sending each through the channel. Returns the total count of generated tokens.

Not available on WASM targets (tokio channels not supported on wasm32-unknown-unknown).

pub fn generate_streaming_with_params( &mut self, prompt_tokens: &[u32], max_tokens: usize, params: &SamplingParams, tx: &UnboundedSender<u32>, ) -> RuntimeResult<usize>

Streaming generation using caller-supplied sampling parameters for the duration of this call only.

Swaps in params on the engine’s existing sampler, runs InferenceEngine::generate_streaming, then restores the previous parameters. As with InferenceEngine::generate_with_params, the sampler’s PRNG state is preserved (only the parameters change), so the default-parameter case is bit-identical to calling generate_streaming directly.

pub fn generate_streaming_sync( &mut self, prompt_tokens: &[u32], max_tokens: usize, tx: &Sender<u32>, ) -> RuntimeResult<usize>

Streaming generation using a synchronous std::sync::mpsc::Sender.

Each generated token is sent through the channel immediately, allowing the consumer to print tokens as they arrive without requiring a tokio runtime.

Trait Implementations

impl Deref for EngineLease

type Target = InferenceEngine<'static>

The resulting type after dereferencing.

fn deref(&self) -> &Self::Target

Dereferences the value.

impl DerefMut for EngineLease

fn deref_mut(&mut self) -> &mut Self::Target

Mutably dereferences the value.

impl Drop for EngineLease

fn drop(&mut self)

Executes the destructor for this type.

fn pin_drop(self: Pin<&mut Self>)

🔬This is a nightly-only experimental API. (pin_ergonomics)

Execute the destructor for this type, but different to Drop::drop, it requires self to be pinned.