Enum PerfModel 

pub enum PerfModel {
    Polynomial,
    Interpolated {
        prefill_interp: Arc<dyn PrefillInterpolator>,
        decode_interp: Arc<dyn DecodeInterpolator>,
    },
    Aiconfigurator {
        callback: Arc<dyn AicCallback>,
    },
}

Performance model for predicting prefill and decode timing

Variants

Polynomial

Default polynomial-based model using hardcoded formulas

Interpolated

Interpolation-based model using profiler data Decode axes: (active_kv_tokens, context_length)

Fields

prefill_interp: Arc<dyn PrefillInterpolator>

decode_interp: Arc<dyn DecodeInterpolator>

Aiconfigurator

AI Configurator SDK calls through the configured callback. Passes the reduced prefill inputs (batch_size, effective_isl, prefix).

Fields

callback: Arc<dyn AicCallback>

Implementations

impl PerfModel

pub fn from_npz(path: &Path) -> Result<Self>

Load performance model from NPZ file

Expected arrays in NPZ file:

• prefill_isl: 1D array of input sequence lengths
• prefill_ttft_ms: 1D array of time to first token in milliseconds
• decode_active_kv_tokens: 1D array of active KV token counts
• decode_context_length: 1D array of context lengths
• decode_itl: 2D array of inter-token latencies in milliseconds

pub fn from_aic_callback(callback: Arc<dyn AicCallback>) -> Self

Create an Aiconfigurator perf model from a callback.

pub fn predict_prefill_time( &self, batch_size: usize, isl: usize, prefix: usize, ) -> Result<f64>

Predict prefill time in milliseconds.

Callers always pass all parameters; each variant uses what it needs:

• Polynomial/Interpolated: uses total new tokens across the batch (batch_size * (isl - prefix)), modeling GPU processing total tokens in parallel
• Aiconfigurator: passes (batch_size, isl - prefix, prefix) to the AIC SDK

pub fn predict_decode_time( &self, batch_size: usize, active_kv_tokens: usize, context_length: usize, total_kv_tokens: usize, ) -> Result<f64>

Predict decode time in milliseconds.

Callers always pass all parameters; each variant uses what it needs:

• Polynomial: uses (active_kv_tokens, total_kv_tokens) as utilization
• Interpolated: uses (active_kv_tokens, context_length)
• Aiconfigurator: uses (batch_size, context_length)

Trait Implementations

impl Clone for PerfModel

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for PerfModel

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for PerfModel

fn default() -> PerfModel

Returns the “default value” for a type.