Struct EngineMetrics 

pub struct EngineMetrics {
    pub tokens_generated: AtomicU64,
    pub tokens_prefilled: AtomicU64,
    pub kv_cache_hits: AtomicU64,
    pub kv_cache_misses: AtomicU64,
    pub decode_nanos: AtomicU64,
    pub prefill_nanos: AtomicU64,
    pub requests_started: AtomicU64,
    pub requests_completed: AtomicU64,
}

Thread-safe metrics counters for an inference engine.

Fields

tokens_generated: AtomicU64

Total tokens generated (decode phase).

tokens_prefilled: AtomicU64

Total tokens processed in prefill.

kv_cache_hits: AtomicU64

Total KV cache hits (prefix/page cache returns a cached slot).

kv_cache_misses: AtomicU64

Total KV cache misses.

decode_nanos: AtomicU64

Total decode time in nanoseconds.

prefill_nanos: AtomicU64

Total prefill time in nanoseconds.

requests_started: AtomicU64

Number of requests started.

requests_completed: AtomicU64

Number of requests completed.

Implementations

impl EngineMetrics

pub fn new() -> Arc<Self>

Create an Arc-wrapped, zero-initialised metrics instance.

pub fn record_decode_token(&self, elapsed: Duration)

Record a single decode token and the time taken to produce it.

pub fn record_prefill(&self, n_tokens: u64, elapsed: Duration)

Record a prefill phase that processed n_tokens prompt tokens.

pub fn record_kv_hit(&self)

Record a KV-cache hit (prefix or page reuse).

pub fn record_kv_miss(&self)

Record a KV-cache miss (full prefill required).

pub fn record_request_start(&self)

Record that a new inference request has started.

pub fn record_request_complete(&self)

Record that an inference request has completed.

pub fn throughput(&self) -> (f64, f64)

Returns (decode_tokens_per_sec, prefill_tokens_per_sec).

Both values are 0.0 if no time has been recorded for that phase.

pub fn kv_cache_hit_rate(&self) -> f64

Returns KV cache hit rate in the range [0.0, 1.0].

Returns 0.0 when no lookups have been recorded.

pub fn snapshot(&self) -> MetricsSnapshot

Returns a point-in-time snapshot of all counters.

pub fn reset(&self)

Reset all counters to zero.

Trait Implementations

impl Debug for EngineMetrics

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for EngineMetrics

fn default() -> EngineMetrics

Returns the “default value” for a type.