Struct PlacementPlan 

pub struct PlacementPlan {
    pub default_placement: ExpertPlacement,
    pub overrides: HashMap<usize, ExpertPlacement>,
}

A CPU/GPU placement plan for a layer’s experts.

Fields

default_placement: ExpertPlacement

overrides: HashMap<usize, ExpertPlacement>

Implementations

impl PlacementPlan

pub fn all_cpu(n_experts: usize) -> Self

pub fn hot_experts_on_gpu(n_experts: usize, n_gpu_resident: usize) -> Self

Index-based placeholder: puts the first n_gpu_resident experts on GPU regardless of their actual size or how often they’re activated. Kept only as a trivial fallback for callers with no real budget/hotness data at all (e.g. ferrox smoke’s synthetic-weight demo); real deployments should use from_budget instead, which places by measured VRAM budget and observed expert hotness rather than by index.

pub fn from_budget( expert_bytes: &[usize], activation_counts: Option<&[u64]>, vram_budget_bytes: u64, ) -> Self

Builds a placement plan from a real VRAM budget and each expert’s actual resident byte size (e.g. summed WeightMatrix::resident_bytes() across an expert’s gate/up/down matrices), following ik_llama.cpp’s --cpu-moe/--override-tensor pattern of deciding CPU-vs-GPU per tensor rather than by a fixed index cutoff.

activation_counts, if given (one count per expert, e.g. accumulated from RoutingDecision::expert_ids over a real or representative workload), places the most frequently activated experts on GPU first – the actual point of expert offload, since keeping a rarely-used expert resident in VRAM wastes the budget a hot expert could have used instead. Without observed counts, falls back to a documented, deterministic policy (index order) rather than guessing at hotness.

Greedy, not globally optimal (a smaller-but-colder expert can still be skipped in favor of trying the next candidate once a larger higher-priority expert doesn’t fit) – optimal knapsack packing is not worth the complexity here, and greedy-by-priority is the same approach real offload tooling uses.

pub fn plan_layers_against_global_budget( expert_bytes_per_layer: &[Vec<usize>], activation_counts_per_layer: Option<&[Vec<u64>]>, vram_budget_bytes: u64, ) -> ResidencyPlan

A device-placement plan for EVERY layer’s routed experts against ONE shared VRAM budget – the fix for the real accounting bug where each layer independently called from_budget with the full budget, so a model with N layers would plan N x the configured bytes of GPU residency. All (layer, expert) candidates compete in one global priority order (hottest first, ties broken by layer then expert index for determinism), and a candidate is only placed on the device if the global running total still fits.

pub fn placement_for(&self, expert_id: usize) -> ExpertPlacement

Trait Implementations

impl Clone for PlacementPlan

fn clone(&self) -> PlacementPlan

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for PlacementPlan

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.