Struct RequestStreamMetrics 

pub struct RequestStreamMetrics { /* private fields */ }

Per-request streaming metrics collector.

Call record_first_token on the first generated token, record_token for each subsequent token, and finish when generation is complete.

Implementations

impl RequestStreamMetrics

pub fn new() -> Self

Create a new metrics collector with zero prompt tokens.

pub fn new_with_prompt_tokens(prompt_tokens: usize) -> Self

Create a new metrics collector that records the prompt token count.

pub fn record_first_token(&mut self)

Record the arrival of the first generated token.

Calling this multiple times is safe — only the first call is recorded.

pub fn record_token(&mut self)

Record the arrival of a subsequent generated token (not the first).

If record_first_token has not been called yet, this call is treated as the first token.

pub fn finish(&mut self)

Mark the end of the generation pass.

This records a final timestamp used for end-to-end latency. It is safe to call even before any tokens have been recorded.

pub fn ttft(&self) -> Option<Duration>

Time to first token (TTFT).

Returns None if record_first_token has not been called.

pub fn median_tbt(&self) -> Option<Duration>

Median inter-token latency.

Returns None when fewer than two tokens have been recorded (i.e. there are no TBT samples).

pub fn p99_tbt(&self) -> Option<Duration>

P99 inter-token latency.

Returns None when there are no TBT samples.

pub fn mean_tbt(&self) -> Option<Duration>

Mean (arithmetic average) inter-token latency.

Returns None when there are no TBT samples.

pub fn tokens_per_second(&self) -> Option<f64>

Generation throughput in tokens per second.

Computed over the decode window (from first token to last token) so that TTFT does not distort the throughput figure.

Returns None if the decode window cannot be determined.

pub fn e2e_latency(&self) -> Option<Duration>

End-to-end request latency (from request start to generation finish).

Returns None if finish has not been called.

pub fn completion_tokens(&self) -> usize

Total number of completion tokens generated so far.

pub fn snapshot(&self) -> StreamMetricsSnapshot

Take a point-in-time snapshot of the current metrics.

Trait Implementations

impl Default for RequestStreamMetrics

fn default() -> Self

Returns the “default value” for a type.