Enum RoutingWorkload 

pub enum RoutingWorkload {
    Interactive,
    Batch,
    Background,
    LocalPreferred,
}

How latency-sensitive this generation request is.

Variants

Interactive

User-facing, interactive request where latency matters.

Batch

Batch job where latency matters somewhat, but quality/cost matter more.

Background

Background or offline work where latency is a weak concern.

LocalPreferred

Caller explicitly prefers on-device models. Distinct from Background (which is “this is a background job, latency barely matters”). The caller may be doing latency-sensitive interactive work but wants the privacy / cost / offline properties of local inference. Same local_bonus as Background plus a slightly more quality-aware weight profile — the caller chose local for a reason, not because the work is throwaway.

Implementations

impl RoutingWorkload

pub fn is_latency_sensitive(self) -> bool

pub fn weights(self) -> (f64, f64, f64)

pub fn local_bonus(self) -> f64

Trait Implementations

impl Clone for RoutingWorkload

fn clone(&self) -> RoutingWorkload

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for RoutingWorkload

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for RoutingWorkload

fn default() -> RoutingWorkload

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for RoutingWorkload

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl PartialEq for RoutingWorkload

fn eq(&self, other: &RoutingWorkload) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Serialize for RoutingWorkload

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl Copy for RoutingWorkload

impl Eq for RoutingWorkload

impl StructuralPartialEq for RoutingWorkload