Struct Sample 

pub struct Sample {
    pub completion_tokens: i64,
    pub prompt_tokens: Option<i64>,
    pub ttft_ms: i64,
    pub decode_ms: i64,
    pub prompt_ms: Option<i64>,
    pub estimated_tokens: bool,
    pub source: TimingSource,
}

One run of one model.

Fields

completion_tokens: i64

Tokens generated, reported or estimated.

prompt_tokens: Option<i64>

Tokens of prompt, when the backend counted them.

ttft_ms: i64

Time to the first token. Always wall clock: it is what a caller waits.

decode_ms: i64

Time spent generating, the prompt excluded.

prompt_ms: Option<i64>

Time spent on the prompt, when the backend reports the phases apart.

estimated_tokens: bool

Whether the token count was counted from the text rather than reported.

source: TimingSource

Where decode_ms came from.

Implementations

impl Sample

pub fn new( stats: Option<&GenerationStats>, generated: &str, wall: WallClock, ) -> Self

Fold one run’s reported stats and wall-clock marks into a sample. generated is the text the run produced, which stands in for a token count the runtime did not report.

pub fn tokens_per_second(&self) -> Option<f64>

Tokens generated a second, when the run took any measurable time.

A wall-clock run is timed from the first token to the last, so the tokens that span is worth are all of them but the first; a backend times its own decode against every token it produced. Counting each the way it was measured is what lets the two sit in one ranked column.

pub fn prompt_tokens_per_second(&self) -> Option<f64>

Prompt tokens processed a second, only where the backend timed the prompt phase; there is nothing to divide otherwise.

Trait Implementations

impl Clone for Sample

fn clone(&self) -> Sample

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Sample

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for Sample

fn eq(&self, other: &Sample) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl StructuralPartialEq for Sample