Skip to main content

max_replicas

Function max_replicas 

Source
pub fn max_replicas(model_bytes: u64, worker_vram: &[u64]) -> usize
Expand description

Upper bound on replicas the pool could form for this model (VRAM-limited): ⌊ΣVRAM / model⌋. The device-count/layer cap can lower the achievable count below this; plan_replicas with ReplicaPolicy::Throughput returns the actual achievable number.