Struct InferenceSpec 

pub struct InferenceSpec {
    pub role: String,
    pub model: Option<String>,
    pub runtime: InferFlavor,
    pub serve: ServeArgs,
    pub memory: MemoryBudget,
    pub excludes: Vec<String>,
    pub latency_class: LatencyClass,
    pub sticky: bool,
    pub api: String,
}

An inference endpoint to adopt or run.

Fields

role: String

Stable role name consumers use (coder-daily).

model: Option<String>

Model registry name or package id. Required to manage an instance; optional when adopting (then only the server is observed).

runtime: InferFlavor

Server software.

serve: ServeArgs

Serve arguments.

memory: MemoryBudget

Expected memory.

excludes: Vec<String>

Co-residency exclusions (role:planner).

latency_class: LatencyClass

Latency class.

sticky: bool

No automatic migration (warm KV).

api: String

API dialect.

Implementations

impl InferenceSpec

pub fn new(role: &str, runtime: InferFlavor) -> Self

A spec with defaults for everything but its identity.

pub fn bind_ip(&self) -> Result<IpAddr, RuntimeError>

The loopback address the server is reached at.

pub fn port(&self) -> Result<u16, RuntimeError>

Port in force.

pub fn base_url(&self) -> Result<String, RuntimeError>

http://<ip>:<port> of the server.

pub fn validate(&self) -> Result<(), RuntimeError>

Boundary validation.

Trait Implementations

impl Clone for InferenceSpec

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for InferenceSpec

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for InferenceSpec

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Eq for InferenceSpec

impl PartialEq for InferenceSpec

fn eq(&self, other: &Self) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl Serialize for InferenceSpec

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl StructuralPartialEq for InferenceSpec