Struct InferenceMetrics 

pub struct InferenceMetrics {

    pub tokens_generated_total: Counter,
    pub requests_total: Counter,
    pub errors_total: Counter,
    pub prompt_tokens_total: Counter,
    pub prefill_duration_seconds: Histogram,
    pub decode_token_duration_seconds: Histogram,
    pub request_duration_seconds: Histogram,
    pub tokens_per_second: Histogram,
    pub active_requests: Gauge,
    pub kv_cache_utilization: Gauge,
    pub model_memory_bytes: Gauge,
    pub request_tokens_per_second: Gauge,
    pub inter_token_latency_p50_seconds: Gauge,
    pub inter_token_latency_p95_seconds: Gauge,
    pub queue_wait_seconds: Gauge,
    pub kv_cache_compression_level: Gauge,
}

All Pictor inference metrics collected in one place.

Thread-safe — can be shared via Arc<InferenceMetrics> across handlers and the inference engine.

Example

use pictor_runtime::metrics::InferenceMetrics;

let metrics = InferenceMetrics::new();
metrics.requests_total.inc_by(5);
metrics.tokens_generated_total.inc_by(100);
metrics.active_requests.set(2.0);
metrics.request_duration_seconds.observe(0.42);

assert_eq!(metrics.requests_total.get(), 5);
assert_eq!(metrics.tokens_generated_total.get(), 100);

let prom = metrics.render_prometheus();
assert!(prom.contains("pictor_requests_total 5"));

Fields

tokens_generated_total: Counter

Total number of tokens generated across all requests.

requests_total: Counter

Total number of requests received.

errors_total: Counter

Total number of errors.

prompt_tokens_total: Counter

Total number of prompt tokens processed.

prefill_duration_seconds: Histogram

Duration of the prefill (prompt processing) phase.

decode_token_duration_seconds: Histogram

Duration of each individual decode step.

request_duration_seconds: Histogram

Total end-to-end request duration.

tokens_per_second: Histogram

Observed tokens-per-second rate.

active_requests: Gauge

Number of currently active (in-flight) requests.

kv_cache_utilization: Gauge

KV cache utilization ratio (0.0 – 1.0).

model_memory_bytes: Gauge

Total model memory usage in bytes.

request_tokens_per_second: Gauge

Smoothed average tokens-per-second across recent requests.

inter_token_latency_p50_seconds: Gauge

p50 inter-token latency across recent requests, in seconds.

inter_token_latency_p95_seconds: Gauge

p95 inter-token latency across recent requests, in seconds.

queue_wait_seconds: Gauge

Mean queue-wait time (admission → first token) in seconds.

kv_cache_compression_level: Gauge

Effective tier of the runtime KV-cache compression policy: 0 = FP16, 1 = Q8, 2 = Q4.

Implementations

impl InferenceMetrics

pub fn update_memory_from_rss(&self)

Update the model_memory_bytes gauge from the current process RSS.

This is a best-effort call: on platforms where RSS is unavailable it records 0.0, which is still a valid (though unhelpful) gauge value.

pub fn new() -> Self

Create a new set of inference metrics with default buckets.

pub fn update_request_rate(&self, snap: &AggregateRateSnapshot)

Update the per-request rate gauges from a crate::request_metrics::AggregateRateSnapshot.

pub fn update_kv_cache_level(&self, level: KvCacheLevel)

Update the KV-cache compression-level gauge from a crate::kv_cache_policy::KvCacheLevel.

pub fn render_prometheus(&self) -> String

Render all metrics in Prometheus text exposition format.

Trait Implementations

impl Default for InferenceMetrics

fn default() -> Self

Returns the “default value” for a type.