Struct BandwidthProfile 

pub struct BandwidthProfile {
    pub version: Option<u32>,
    pub threshold: Option<f64>,
    pub gpu: ProfileGpu,
    pub dtypes: BTreeMap<String, MoeBackend>,
    pub dtype_kernels: BTreeMap<String, KernelBandwidths>,
    pub workloads: BTreeMap<String, Workload>,
}

A measured bandwidth profile for one machine.

These numbers are hardware facts, so a profile is keyed to the card it was taken on. A profile whose GPU name disagrees with the card in front of you is not “close enough”: applying another machine’s split is worse than having no split at all, so the lookups below refuse it rather than approximate.

Fields

version: Option<u32>

threshold: Option<f64>

gpu: ProfileGpu

dtypes: BTreeMap<String, MoeBackend>

Per-format verdicts, the authoritative source.

dtype_kernels: BTreeMap<String, KernelBandwidths>

Per-format measured bandwidths.

workloads: BTreeMap<String, Workload>

Per-model detail, consulted when the per-format entry is absent.

Implementations

impl BandwidthProfile

pub fn matches_gpu(&self, gpu_name: Option<&str>) -> bool

Whether this profile describes the card in front of us.

A profile with no recorded GPU name is accepted (an older benchmark), because refusing it would silently discard a measurement the user did take. A profile with a different recorded name is refused.

pub fn backend_for(&self, format: &str) -> Option<MoeBackend>

The backend this profile recommends for format.

The per-format verdict wins. Failing that, the per-model entries for the same format are aggregated conservatively: hybrid only if every model that was measured picked hybrid, because one model that does not benefit is evidence the machine is on the wrong side of the line.

pub fn fetch_fraction_for(&self, format: &str) -> Option<f64>

The fetch fraction this profile implies for format.

pub fn policy_for(&self, format: &str) -> QStarPolicy

The policy to serve format with, given this profile.

A profile that measured nothing usable for this format yields the unbenchmarked default: a fixed cap of one fetch per layer per step.

Trait Implementations

impl Clone for BandwidthProfile

fn clone(&self) -> BandwidthProfile

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for BandwidthProfile

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for BandwidthProfile

fn default() -> BandwidthProfile

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for BandwidthProfile

where BandwidthProfile: Default,

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl PartialEq for BandwidthProfile

fn eq(&self, other: &BandwidthProfile) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl Serialize for BandwidthProfile

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl StructuralPartialEq for BandwidthProfile