Struct InferenceStats 

pub struct InferenceStats {
    pub tokens_generated: usize,
    pub cache_hits: usize,
    pub total_attention_ops: usize,
    pub avg_cache_len: f64,
    pub peak_memory_bytes: usize,
}

Tracks statistics for autoregressive token generation with KV-cache.

Fields

tokens_generated: usize

Total tokens generated so far.

cache_hits: usize

Number of cache reads that avoided recomputation.

total_attention_ops: usize

Total number of attention operations executed.

avg_cache_len: f64

Running average of cache sequence length across recorded steps.

peak_memory_bytes: usize

Peak memory usage in bytes observed across all steps.

Implementations

impl InferenceStats

pub fn new() -> Self

Create a zeroed InferenceStats.

pub fn record_step(&mut self, cache: &KvCache)

Record a single generation step given the current cache state.

pub fn summary(&self) -> String

Return a human-readable summary string.

Trait Implementations

impl Clone for InferenceStats

fn clone(&self) -> InferenceStats

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for InferenceStats

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for InferenceStats

fn default() -> Self

Returns the “default value” for a type.