Struct NativeRuntime 

pub struct NativeRuntime { /* private fields */ }

Native runtime for local LLM inference.

Uses mistral.rs for high-performance inference on GGUF models. Supports CPU and GPU (Metal on macOS, CUDA on Linux) acceleration.

Example

use spn_native::inference::NativeRuntime;
use spn_core::LoadConfig;

let mut runtime = NativeRuntime::new()?;
runtime.load("model.gguf".into(), LoadConfig::default()).await?;
let response = runtime.infer("Hello!", Default::default()).await?;

Implementations

impl NativeRuntime

pub fn new() -> Self

Create a new native runtime.

The runtime is created without a model loaded. Call load() to load a model before running inference.

pub fn model_path(&self) -> Option<&PathBuf>

Get the path to the currently loaded model.

pub fn config(&self) -> Option<&LoadConfig>

Get the load configuration for the current model.

Trait Implementations

impl Clone for NativeRuntime

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for NativeRuntime

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for NativeRuntime

fn default() -> Self

Returns the “default value” for a type.

impl InferenceBackend for NativeRuntime

Available on non-crate feature inference only.

async fn load( &mut self, _model_path: PathBuf, _config: LoadConfig, ) -> Result<(), NativeError>

Load a model from disk.

async fn unload(&mut self) -> Result<(), NativeError>

Unload the model from memory.

fn is_loaded(&self) -> bool

Check if a model is currently loaded.

fn model_info(&self) -> Option<&ModelInfo>

Get metadata about the loaded model.

async fn infer( &self, _prompt: &str, _options: ChatOptions, ) -> Result<ChatResponse, NativeError>

Generate a response (non-streaming).

async fn infer_stream( &self, _prompt: &str, _options: ChatOptions, ) -> Result<impl Stream<Item = Result<String, NativeError>> + Send, NativeError>

Generate a response (streaming).