Skip to main content

ModelRunner

sim_lib_agent

Trait ModelRunner 

Source 
pub trait ModelRunner: Send + Sync {
    // Required methods
    fn card(&self) -> ModelCard;
    fn infer(
        &self,
        cx: &mut Cx,
        request: ModelRequest,
    ) -> Result<ModelResponse, Error>;

    // Provided methods
    fn infer_stream(
        &self,
        cx: &mut Cx,
        request: ModelRequest,
        sink: &mut dyn ModelEventSink,
    ) -> Result<ModelResponse, Error> { ... }
    fn bid(&self, _request: &ModelRequest) -> Result<ModelBid, Error> { ... }
}
Expand description

Executable contract for a model backend or model-like runtime surface.

Required Methods§

Source

fn card(&self) -> ModelCard

Returns metadata describing the runner’s advertised model surface.

Source

fn infer( &self, cx: &mut Cx, request: ModelRequest, ) -> Result<ModelResponse, Error>

Executes one non-streaming inference request.

Provided Methods§

Source

fn infer_stream( &self, cx: &mut Cx, request: ModelRequest, sink: &mut dyn ModelEventSink, ) -> Result<ModelResponse, Error>

Executes one streaming inference request.

The default implementation falls back to Self::infer and emits a synthesized final event.

Source

fn bid(&self, _request: &ModelRequest) -> Result<ModelBid, Error>

Returns an availability and score hint for market-style routing.

Dyn Compatibility§

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors§