Struct ModelWarmup 

pub struct ModelWarmup {
    pub num_warmup_tokens: usize,
    pub warmup_prompt: String,
}

Runs a small number of dummy inference passes on a freshly-initialised InferenceEngine to prime internal allocation caches and JIT paths before the first real request arrives.

Fields

num_warmup_tokens: usize

Number of tokens to generate during the warmup pass.

warmup_prompt: String

Prompt text fed to the engine during warmup.

Implementations

impl ModelWarmup

pub fn new() -> Self

Create a warmup helper with sensible defaults (32 tokens, generic prompt).

pub fn with_tokens(self, n: usize) -> Self

Override the number of warmup tokens.

pub fn with_prompt(self, p: &str) -> Self

Override the warmup prompt text.

pub fn run( &self, engine: &mut InferenceEngine<'_>, params: &SamplingParams, ) -> u64

Execute the warmup passes on engine using params.

Generates up to ModelWarmup::num_warmup_tokens tokens from a small synthetic token sequence and discards the output. Returns the elapsed wall-clock time in milliseconds.

Errors from the engine are logged as warnings but do not propagate — warmup failure is non-fatal.

pub fn needs_warmup(_engine: &InferenceEngine<'_>) -> bool

Heuristic: should this engine be warmed up?

The current implementation always returns true — callers are responsible for deciding when to apply warmup (e.g. once after initial load, or after a cache miss).

Trait Implementations

impl Default for ModelWarmup

fn default() -> Self

Returns the “default value” for a type.