Struct LocalLlmProvider 

pub struct LocalLlmProvider { /* private fields */ }

Wraps a CandleEngine behind the LlmProvider trait using a byte-level tokenizer. A production build would swap in a HuggingFace tokenizer loaded from the model file.

Implementations

impl LocalLlmProvider

pub fn from_path( path: impl AsRef<Path>, eos: Token, permit: LoadPermit, ) -> Result<Self>

Load from a consumer-supplied GGUF file.

pub fn toy( vocab: usize, dim: usize, eos: Token, permit: LoadPermit, ) -> Result<Self>

Build a toy provider for testing.

Trait Implementations

impl LlmProvider for LocalLlmProvider

fn chat(&self, req: &ChatRequest) -> Result<ChatResponse>

Blocking, non-streaming chat completion.

fn chat_stream( &self, req: &ChatRequest, on_token: &mut dyn FnMut(ChatToken), ) -> Result<()>

Streaming chat: calls on_token for each fragment as it arrives. Returns when generation is complete or on error. The final call will have ChatToken::is_final == true.