Struct ModelRuntime 

pub struct ModelRuntime<B: BackendProvider> { /* private fields */ }

A prepared model, its selected backend, and its backend-owned session.

This is the canonical client-side execution owner. Keeping the backend and session together makes backend selection a whole-model decision and makes it impossible to submit a session through a different backend instance. Backend-owned executable, cache, tensor, and completion types remain associated types and never enter the portable API.

Implementations

impl<B: BackendProvider> ModelRuntime<B>

pub fn prepare(backend: B, config: B::ModelConfig) -> Result<Self, B::Error>

Prepares config and creates its sole execution session.

pub fn from_prepared( backend: B, model: PreparedModel<B::Model>, ) -> Result<Self, B::Error>

Creates the sole execution session for an already prepared model.

pub const fn backend(&self) -> &B

Returns the selected backend.

pub const fn session(&self) -> &B::Session

Returns the backend-owned session for optional backend capabilities.

pub fn session_mut(&mut self) -> &mut B::Session

Returns the backend-owned session for optional backend capabilities.

pub fn parts_mut(&mut self) -> (&B, &mut B::Session)

Borrows the selected backend and its mutable session together.

pub fn capabilities(&self) -> SessionCapabilities

Reports capabilities of the exact prepared model session.

pub fn prefill( &mut self, input: <B::Session as BackendSession<B>>::PrefillInput, ) -> Result<SessionSubmission<B>, B::Error>

Submits prompt prefill through the selected backend and session.

pub fn decode( &mut self, input: <B::Session as BackendSession<B>>::DecodeInput, ) -> Result<SessionSubmission<B>, B::Error>

Submits cached decode through the selected backend and session.

pub fn observe_output( &self, output: &<B::Session as BackendSession<B>>::Output, ) -> Result<ObservationSet, B::Error>

Materializes portable observations from an already completed output.

impl<B> ModelRuntime<B>

where B: BackendProvider, B::Session: InspectableBackendSession<B>,

pub fn inspect_prefill( &mut self, input: <B::Session as BackendSession<B>>::PrefillInput, request: &ObservationRequest, ) -> Result<InspectedOutput<<B::Session as BackendSession<B>>::Output>, B::Error>

Executes a completed, explicitly instrumented prefill operation.

pub fn inspect_decode( &mut self, input: <B::Session as BackendSession<B>>::DecodeInput, request: &ObservationRequest, ) -> Result<InspectedOutput<<B::Session as BackendSession<B>>::Output>, B::Error>

Executes a completed, explicitly instrumented decode operation.

impl<B: ModelLoadingBackend> ModelRuntime<B>

pub fn load( backend: B, artifact: impl AsRef<Path>, options: B::LoadOptions, ) -> Result<Self, ModelLoadError<B::Error>>

Loads an artifact and creates its sole session on backend.