Struct HardwareConfig 

pub struct HardwareConfig {
    pub name: String,
    pub compute_flops: f64,
    pub memory_bandwidth: f64,
    pub memory_capacity: u64,
    pub kv_cache_capacity: u64,
    pub gpu_memory_utilization: f64,
    pub bytes_per_param: u32,
    pub compute_bound_threshold: u32,
}

Fields

name: String

Accelerator name (e.g., “H100”, “A100”)

compute_flops: f64

Compute capacity in FLOPS (for specific precision, e.g., bf16)

memory_bandwidth: f64

Memory bandwidth in bytes/sec

memory_capacity: u64

Total memory capacity in bytes

kv_cache_capacity: u64

KV cache capacity in bytes (subset of memory_capacity) If not specified, calculated from gpu_memory_utilization

gpu_memory_utilization: f64

Fraction of GPU memory to use (vLLM default: 0.9) Used to calculate kv_cache_capacity if not explicitly set

bytes_per_param: u32

Number of bytes per parameter (1 for fp8, 2 for bf16)

compute_bound_threshold: u32

Compute-bound threshold (derived from flops/bandwidth ratio) This is calculated: bytes_per_param * compute_flops / memory_bandwidth

Implementations

impl HardwareConfig

pub fn compute_threshold(&mut self)

Calculate and set the compute-bound threshold

pub fn compute_kv_cache_capacity(&mut self, model_size_bytes: u64)

Calculate KV cache capacity if not explicitly set Formula: (memory_capacity * gpu_memory_utilization) - model_size This matches vLLM’s behavior: requested_memory - non_kv_cache_memory

pub fn with_threshold(self) -> Self

Initialize with threshold pre-computed

Trait Implementations

impl Clone for HardwareConfig

fn clone(&self) -> HardwareConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for HardwareConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for HardwareConfig

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.