Struct LlamaContext 

pub struct LlamaContext {
    pub backend: Arc<LlamaLib>,
    pub handle: *mut llama_context,
}

Inference context attached to a model

Fields

backend: Arc<LlamaLib>

handle: *mut llama_context

Implementations

impl LlamaContext

pub fn new( model: &LlamaModel, params: llama_context_params, ) -> Result<Self, LlamaError>

pub fn default_params(model: &LlamaModel) -> llama_context_params

pub fn decode(&mut self, batch: &LlamaBatch) -> Result<(), LlamaError>

pub fn kv_cache_clear(&mut self)

Clear the KV cache for this context. Resets all cached key/value state, allowing the context to be reused for a fresh generation without reallocating.

Trait Implementations

impl Drop for LlamaContext

fn drop(&mut self)

Executes the destructor for this type.