Struct InferenceEngine 

pub struct InferenceEngine {
    pub backend: Arc<LlamaBackend>,
    pub model: Arc<LlamaModel>,
    pub config: InferenceConfig,
}

Shared inference engine that holds the backend + model in Arcs.

This is the resource-heavy object: it loads the DLL and the model weights. Multiple agents can share the same InferenceEngine — each agent just creates its own lightweight LlamaContext.

Example

use llama_cpp_v3_agent_sdk::inference::{InferenceEngine, InferenceConfig};
use llama_cpp_v3::backend::Backend;
use std::sync::Arc;

let engine = Arc::new(InferenceEngine::load(InferenceConfig {
    backend: Backend::Vulkan,
    model_path: "model.gguf".into(),
    n_gpu_layers: 99,
    ..Default::default()
}).expect("Failed to load model"));

// Share with multiple agents:
let agent1 = llama_cpp_v3_agent_sdk::AgentBuilder::new()
    .engine(engine.clone())
    .build().unwrap();
let agent2 = llama_cpp_v3_agent_sdk::AgentBuilder::new()
    .engine(engine.clone())
    .system_prompt("You are a different agent.")
    .build().unwrap();

Fields

backend: Arc<LlamaBackend>

model: Arc<LlamaModel>

config: InferenceConfig

Implementations

impl InferenceEngine

pub fn load(config: InferenceConfig) -> Result<Self, AgentError>

Load a model from the given configuration.

This performs the expensive operations (DLL loading, model weight loading) exactly once. The returned engine can be wrapped in Arc and shared.

pub fn create_context( &self, n_ctx_override: Option<u32>, ) -> Result<LlamaContext, AgentError>

Create a new LlamaContext from this engine.

Each agent should have its own context (it holds the KV cache). The n_ctx override lets callers use a different context size than the engine default.

pub fn model(&self) -> &LlamaModel

Access the raw LlamaModel.

pub fn backend(&self) -> &LlamaBackend

Access the raw LlamaBackend.

pub fn lib(&self) -> Arc<LlamaLib>

Get the Arc<LlamaLib> for creating samplers and batches.

Trait Implementations

impl Clone for InferenceEngine

fn clone(&self) -> InferenceEngine

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.