Struct Local 

pub struct Local { /* private fields */ }

Local LLM provider wrapping a mistralrs Model.

Implementations

impl Local

pub fn from_model(model: Model) -> Self

Construct from a pre-built mistralrs Model.

pub async fn from_text( model_id: &str, isq: Option<IsqType>, chat_template: Option<&str>, ) -> Result<Self>

Build using TextModelBuilder.

Standard text models from HuggingFace.

pub async fn from_gguf( model_id: &str, chat_template: Option<&str>, ) -> Result<Self>

Build using GgufModelBuilder.

GGUF quantized models from HuggingFace. The model_id is the HF repo ID; mistralrs auto-discovers GGUF files in the repo.

pub fn model(&self) -> &Model

Access the inner mistralrs Model.

pub fn context_length(&self, model: &str) -> Option<usize>

Query the context length for a given model ID.

Returns None if the model doesn’t report a sequence length.

pub async fn from_vision( model_id: &str, isq: Option<IsqType>, chat_template: Option<&str>, ) -> Result<Self>

Build using VisionModelBuilder.

Vision-language models from HuggingFace.

Trait Implementations

impl Clone for Local

fn clone(&self) -> Local

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Model for Local

async fn send(&self, request: &Request) -> Result<Response>

Send a chat completion request.

fn stream( &self, request: Request, ) -> impl Stream<Item = Result<StreamChunk>> + Send

Stream a chat completion response.

fn context_limit(&self, model: &str) -> usize

Resolve the context limit for a model name.

fn active_model(&self) -> CompactString

Get the active/default model name.