Enum PerfModel 

pub enum PerfModel {
    Polynomial,
    Interpolated {
        prefill_interp: Arc<dyn PrefillInterpolator>,
        decode_interp: Arc<dyn DecodeInterpolator>,
    },
}

Performance model for predicting prefill and decode timing

Variants

Polynomial

Default polynomial-based model using hardcoded formulas

Interpolated

Interpolation-based model using profiler data Interpolators are built once and stored as trait objects

Fields

prefill_interp: Arc<dyn PrefillInterpolator>

decode_interp: Arc<dyn DecodeInterpolator>

Implementations

impl PerfModel

pub fn from_npz(path: &Path) -> Result<Self>

Load performance model from NPZ file

Expected arrays in NPZ file:

• prefill_isl: 1D array of input sequence lengths
• prefill_ttft_ms: 1D array of time to first token in milliseconds
• decode_active_kv_tokens: 1D array of active KV token counts
• decode_context_length: 1D array of context lengths
• decode_itl: 2D array of inter-token latencies in milliseconds

pub fn predict_prefill_time(&self, new_tokens: usize) -> f64

Predict prefill time in milliseconds given the number of new tokens

pub fn predict_decode_time( &self, active_kv_tokens: usize, context_length: usize, ) -> f64

Predict decode time in milliseconds given active KV tokens and context length

For the Polynomial variant, this computes active percentage as active_kv_tokens / 16384. For the Interpolated variant, this performs 2D bilinear interpolation.

Trait Implementations

impl Clone for PerfModel

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for PerfModel

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for PerfModel

fn default() -> PerfModel

Returns the “default value” for a type.