pub struct ForwardPassPerfModel { /* private fields */ }Expand description
Forward-pass-level performance model with optional online tuning.
This API intentionally stays at AIC’s forward-pass abstraction. It does not model TTFT, ITL, SLA, engine capacity, queueing policy, or Dynamo engine limits. Callers pass FPMs for one engine iteration and receive one forward-pass latency estimate in milliseconds.
The prefill/decode/mixed workload kind is inferred from each iteration’s
scheduled_requests fields; it is not chosen at construction:
- prefill: scheduled prefill tokens and no scheduled decode work, using
[sum_prefill_tokens] - decode: scheduled decode work and no scheduled prefill tokens, using
[num_decode_requests, sum_decode_kv_tokens] - mixed/agg: both scheduled prefill and decode work, using
[sum_prefill_tokens, sum_decode_kv_tokens] - empty: no scheduled prefill or decode work, estimates
0.0and is not used for tuning
Native correction grids use fixed constructor-time ranges from
ForwardPassPerfOptions: max_num_tokens bounds sum_prefill_tokens,
max_batch_size bounds num_decode_requests, and max_kv_tokens bounds
sum_decode_kv_tokens. min_faster_correction_factor and
max_slower_correction_factor place independent absolute bounds on learned
native correction factors in each direction, defaulting to 0.5 and 2.0.
Callers may explicitly disable either bound.
Queued request fields are accepted for FPM schema parity but ignored by this
forward-pass-level model. estimate_forward_pass_time_ms treats FPM as a
workload descriptor: it uses scheduled workload fields and ignores
wall_time. tune_with_fpms treats FPM as observed telemetry: it uses the
same scheduled workload fields as features and uses positive wall_time as
the observation target. For attention-DP configurations, the input for one
iteration is one FPM per attention-DP rank; tuning merges that list into one
observation by taking max-rank load features and max nonzero wall_time.
Implementations§
Source§impl ForwardPassPerfModel
impl ForwardPassPerfModel
Sourcepub fn from_regression(
options: ForwardPassPerfOptions,
) -> Result<Self, AicError>
pub fn from_regression( options: ForwardPassPerfOptions, ) -> Result<Self, AicError>
API:
ForwardPassPerfModel::from_regression(options) -> Result<Self, AicError>
Description: create a regression-only forward-pass model.
This mode is for native-AIC-unsupported models. It returns None from
estimate_forward_pass_time_ms for non-empty iterations until the
inferred workload kind has at least options.min_observations tuning samples.
Correction factor getters always return None in this mode.
Sourcepub fn estimate_forward_pass_time_ms(
&self,
metrics_by_rank: &[ForwardPassMetrics],
) -> Result<Option<f64>, AicError>
pub fn estimate_forward_pass_time_ms( &self, metrics_by_rank: &[ForwardPassMetrics], ) -> Result<Option<f64>, AicError>
API:
model.estimate_forward_pass_time_ms(metrics_by_rank) -> Result<Option<f64>, AicError>
Description: estimate one forward-pass iteration in milliseconds.
metrics_by_rank must contain the FPMs for a single engine iteration,
one entry per attention-DP rank. Single-rank callers pass a one-element
slice. The inferred workload kind uses only scheduled_requests as described on
ForwardPassPerfModel; queued fields and wall_time are ignored for
estimation.
Native models return an AIC estimate immediately, multiplied by the
correction factor for the matching workload region. Correction factors
default to 1.0 for inferred workload kinds with fewer than
min_observations total samples, empty regions, and queries outside the
configured correction-grid workload ranges in
ForwardPassPerfOptions. Regression models return Ok(None) until the
matching inferred workload kind has enough tuning samples. Empty
scheduled work returns Ok(Some(0.0)).
Pure Rust over the Engine — no Python re-entry.
Sourcepub fn tune_with_fpms(
&mut self,
iterations: &[Vec<ForwardPassMetrics>],
) -> Result<(), AicError>
pub fn tune_with_fpms( &mut self, iterations: &[Vec<ForwardPassMetrics>], ) -> Result<(), AicError>
API:
model.tune_with_fpms(iterations) -> Result<(), AicError>
Description: tune the model from observed FPM iterations.
The outer slice is a list of observed iterations. Each inner slice is
the per-attention-DP-rank FPM list for one iteration:
[[iter0_rank0, iter0_rank1], [iter1_rank0, iter1_rank1]].
Single-rank callers still use one FPM per inner slice.
For each non-empty iteration, this method infers the workload kind from
scheduled request fields, takes max-rank load features, and uses the max
finite positive wall_time across ranks as the observed latency target
in milliseconds. Iterations with no scheduled work or no positive
wall_time are ignored. Native models update the matching region’s
median observed_ms / native_ms correction factor, with each ratio
bounded by min_faster_correction_factor and
max_slower_correction_factor when configured. Regions are used only
after their inferred workload kind has min_observations total samples;
empty regions keep the default factor 1.0. Observations outside the
configured correction-grid workload ranges are ignored by native
correction models. Regression models learn a workload-specific linear
fit.
Pure Rust over the Engine — no Python re-entry.
Sourcepub fn diagnostics(&self) -> ForwardPassPerfDiagnostics
pub fn diagnostics(&self) -> ForwardPassPerfDiagnostics
API:
model.diagnostics() -> ForwardPassPerfDiagnostics
Description: return the current backend, readiness, retained sample count, and fallback warning.
Sourcepub fn min_correction_factor(&self) -> Option<f64>
pub fn min_correction_factor(&self) -> Option<f64>
API:
model.min_correction_factor() -> Option<f64>
Description: return the smallest ready native correction factor across all workload kinds.
Returns None before any native correction workload kind has enough samples.
Regression-only models also return None.
Sourcepub fn max_correction_factor(&self) -> Option<f64>
pub fn max_correction_factor(&self) -> Option<f64>
API:
model.max_correction_factor() -> Option<f64>
Description: return the largest ready native correction factor across all workload kinds.
Returns None before any native correction workload kind has enough samples.
Regression-only models also return None.
Sourcepub fn avg_correction_factor(&self) -> Option<f64>
pub fn avg_correction_factor(&self) -> Option<f64>
API:
model.avg_correction_factor() -> Option<f64>
Description: return the arithmetic mean of ready native correction factors across all workload kinds.
Returns None before any native correction workload kind has enough samples.
Regression-only models also return None.
Sourcepub fn options(&self) -> &ForwardPassPerfOptions
pub fn options(&self) -> &ForwardPassPerfOptions
API:
model.options() -> &ForwardPassPerfOptions
Description: return the immutable tuning options used by this model.
Trait Implementations§
Source§impl Clone for ForwardPassPerfModel
impl Clone for ForwardPassPerfModel
Source§fn clone(&self) -> ForwardPassPerfModel
fn clone(&self) -> ForwardPassPerfModel
1.0.0 (const: unstable) · Source§fn clone_from(&mut self, source: &Self)
fn clone_from(&mut self, source: &Self)
source. Read more