Struct ModelTargets 

pub struct ModelTargets {
    pub targets: HashMap<String, Vec<InferenceTarget>>,
    /* private fields */
}

Per-model routing table.

Fields

targets: HashMap<String, Vec<InferenceTarget>>

model_name -> list of inference targets.

Implementations

impl ModelTargets

pub fn get(&self, model: &str) -> InferenceTarget

Get target for a specific model. Round-robins across multiple hosts.

pub fn candidates(&self, model: &str) -> Vec<InferenceTarget>

All candidate targets for a model, preserving their current order.

pub fn pick_from(&self, candidates: &[InferenceTarget]) -> InferenceTarget

Round-robin pick from a caller-supplied candidate slice.

pub fn pick_sticky_from( candidates: &[InferenceTarget], sticky_key: u64, ) -> InferenceTarget

Sticky pick from a caller-supplied candidate slice.

Trait Implementations

impl Clone for ModelTargets

fn clone(&self) -> ModelTargets

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for ModelTargets

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl Default for ModelTargets

fn default() -> ModelTargets

Returns the “default value” for a type.