Struct ModelCost 

pub struct ModelCost {
    pub n_layers: usize,
    pub per_layer_bytes: u64,
    pub per_layer_expert_bytes: u64,
    pub per_layer_expert_active_bytes: u64,
    pub embed_bytes: u64,
    pub head_bytes: u64,
    pub per_layer_flops: f64,
    pub kv: KvProfile,
    pub params: u64,
    pub hidden_size: usize,
}

Resident-memory + compute cost model for a layer-stack model.

Bytes are the ACTUAL per-node footprint (packed weights as stored), so the planner’s budgets mean something. Build it with ModelCost::from_tensor_index rather than by hand.

Fields

n_layers: usize

per_layer_bytes: u64

Average resident bytes per layer, excluding routed experts.

per_layer_expert_bytes: u64

Average routed-expert bytes per layer. These may live on disk.

per_layer_expert_active_bytes: u64

Expert bytes actually read per token — per_layer_expert_bytes × expert_active_fraction. Drives the paging-bandwidth estimate.

embed_bytes: u64

Extra bytes on the FIRST stage (token embedding).

head_bytes: u64

Extra bytes on the LAST stage (final norm + LM head).

per_layer_flops: f64

Relative FLOPs per layer (throughput policy; 1.0 = uniform).

kv: KvProfile

What the attention cache costs, and whether that is known.

ModelCost::from_tensor_index infers it from tensor shapes where it can (see CostOptions::kv); otherwise it is KvSource::Unknown and the planner refuses to guess.

params: u64

Total parameters, used to re-scale the model between precisions.

hidden_size: usize

Model width, read from a rank-1 norm inside a block. 0 when unknown.

Worth carrying rather than re-deriving: the worst-case KV bound needs it, and inferring it from byte counts (sqrt(per_layer_bytes / 12)) is badly wrong for a quantized MoE, whose per-layer bytes are dominated by expert banks at ~2 bits per weight.

Implementations

impl ModelCost

pub fn from_gguf(paths: &[PathBuf], opts: &CostOptions) -> Result<Self>

Derive the cost model straight from GGUF shards.

impl ModelCost

pub fn at_bits(&self, target_bits: f64) -> Option<Self>

The same model at a different weight precision.

Scales every byte figure by target_bits / current_bits, where the current width is inferred from the checkpoint (bytes × 8 / params). Approximate by construction — it assumes the whole model re-quantizes uniformly — but it is the right shape for answering “would this fit one step down?”, which is the question a precision ladder asks.

Returns None when the parameter count is unknown (nothing to scale by).

pub fn kv_bytes(&self, layers: u64, context: usize) -> u64

KV bytes one node holds for layers at context tokens.

pub fn total_bytes(&self) -> u64

Total bytes across the whole model, experts included.

pub fn total_resident_bytes(&self) -> u64

Bytes that must be RESIDENT for the whole model (experts excluded).

pub fn is_expert_dominated(&self) -> bool

True when routed experts dominate — the regime where disk and IO decide placement rather than RAM.

pub fn from_tensor_index( entries: &[TensorEntry], opts: &CostOptions, ) -> Result<Self>

Derive the cost model from a checkpoint’s tensor index.

Layer count is max(block index) + 1; per-layer figures are averaged over the blocks actually present, so a model with a trailing MTP block or an uneven layer stack still gets a usable mean.

pub fn summary(&self) -> String

One-line summary for logs.

Trait Implementations

impl Clone for ModelCost

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for ModelCost

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for ModelCost

fn default() -> Self

Returns the “default value” for a type.