Struct TokenBudget 

pub struct TokenBudget {
    pub us_per_token: f64,
    pub tokens_per_sec: f64,
    pub batch_size: usize,
}

Performance budget expressed in token terms. Aligns compute costs with LLM inference metrics.

Fields

us_per_token: f64

Latency budget per token (microseconds)

tokens_per_sec: f64

Throughput target (tokens/second)

batch_size: usize

Batch size for amortization

Implementations

impl TokenBudget

pub fn from_latency(us_per_token: f64) -> Self

Create budget from latency target. 50µs/token = 20,000 tokens/sec

pub fn from_throughput(tokens_per_sec: f64) -> Self

Create budget from throughput target. 20,000 tokens/sec = 50µs/token

pub fn with_batch_size(self, batch_size: usize) -> Self

Set batch size for amortization.

pub fn is_met(&self, actual_us_per_token: f64) -> bool

Check if actual performance meets budget.

pub fn utilization(&self, actual_us_per_token: f64) -> f64

Calculate budget utilization (0.0 = unused, 1.0 = exactly at budget, >1.0 = over budget).

Trait Implementations

impl Clone for TokenBudget

fn clone(&self) -> TokenBudget

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenBudget

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for TokenBudget

fn default() -> Self

Returns the “default value” for a type.

impl Copy for TokenBudget