Struct Engine 

pub struct Engine { /* private fields */ }

High-level inference engine that wraps model loading, tokenization, and generation.

Engine is Send + Sync safe for the immutable model and config, but the mutable inference context and sampler are created per-call via internal state.

Implementations

impl Engine

pub fn load(config: EngineConfig) -> Result<Self, EngineError>

Load a model and create an inference engine.

This opens the model file (GGUF or ONNX), loads the tokenizer and model weights, and selects the appropriate backend (CPU or GPU).

Format is auto-detected by file extension:

• .gguf – GGUF format (default)
• .onnx – ONNX format (requires onnx feature, companion config.json + tokenizer.json)

pub fn select_gpu_backend(model: &LlamaModel) -> Arc<dyn Backend>

Select the best available GPU backend.

Priority: CUDA > Metal > DX12 > Vulkan > CPU fallback.

pub fn model_config(&self) -> &ModelConfig

Get the model configuration.

pub fn chat_template(&self) -> &ChatTemplate

Get the detected chat template.

pub fn gguf(&self) -> Option<&GgufFile>

Get the GGUF file metadata (None for ONNX-loaded models).

pub fn tokenizer(&self) -> &Tokenizer

Get the tokenizer.

pub fn engine_config(&self) -> &EngineConfig

Get the engine configuration.

pub fn model(&self) -> &dyn Model

Get the underlying model (for advanced usage like perplexity computation).

pub fn backend(&self) -> &Arc<dyn Backend>

Get the backend.

pub fn add_bos(&self) -> bool

Whether to add a BOS token when encoding prompts.

pub fn create_inference_context(&self) -> InferenceContext

Generate text from a prompt.

The prompt is automatically wrapped with the detected chat template unless it already contains chat formatting tokens.

Create an InferenceContext respecting the configured KV cache type.

pub fn generate( &self, prompt: &str, max_tokens: usize, ) -> Result<String, EngineError>

Returns the generated text (not including the prompt).

pub fn generate_streaming( &self, prompt: &str, max_tokens: usize, ) -> GenerationStream<'_>

Generate text from a prompt, yielding tokens as they are produced.

Each item in the returned iterator is a Result<String, EngineError> containing the decoded text of one or more tokens.

pub fn embed(&self, text: &str) -> Result<Vec<f32>, EngineError>

Extract embeddings from text using the model.