Struct EngineConfig 

pub struct EngineConfig {
    pub model_path: String,
    pub tokenizer_path: Option<String>,
    pub temperature: f32,
    pub top_k: usize,
    pub top_p: f32,
    pub repeat_penalty: f32,
    pub max_tokens: usize,
    pub seed: Option<u64>,
    pub use_gpu: bool,
    pub max_context_len: Option<usize>,
    pub kv_cache_type: KVCacheType,
}

Configuration for creating an Engine.

Can be constructed manually, from a Config TOML file, or with Default::default().

Fields

model_path: String

Path to the model file (GGUF or ONNX).

tokenizer_path: Option<String>

Optional path to a tokenizer file.

For ONNX models, this defaults to tokenizer.json in the same directory. Can also point to a GGUF file to extract just the tokenizer.

temperature: f32

Temperature for sampling (0.0 = greedy, higher = more random).

top_k: usize

Top-K sampling: only consider the K most likely tokens (0 = disabled).

top_p: f32

Top-P (nucleus) sampling: only consider tokens with cumulative probability <= p.

repeat_penalty: f32

Repetition penalty (1.0 = no penalty).

max_tokens: usize

Default maximum tokens to generate.

seed: Option<u64>

Random seed for reproducible generation (None = random).

use_gpu: bool

Use GPU acceleration (requires cuda feature).

max_context_len: Option<usize>

Maximum context length override.

If set, caps the model’s context length (and thus KV cache size). The model’s native max_seq_len from GGUF metadata can be very large (e.g. 32768) which may exhaust GPU memory for the KV cache alone. Set this to a smaller value (e.g. 2048) to reduce VRAM usage. If None or 0, uses the model’s native max_seq_len.

kv_cache_type: KVCacheType

KV cache type for memory-efficient inference.

Implementations

impl EngineConfig

pub fn from_config_file(path: impl AsRef<Path>) -> Result<Self, ConfigError>

Load an EngineConfig from a Config TOML file.

This is a convenience method that loads the full config and extracts the engine-relevant sections.

pub fn from_config( config_path: Option<impl AsRef<Path>>, ) -> Result<Self, ConfigError>

Load an EngineConfig using the full config precedence chain.

Searches default config file locations, then applies environment variable overrides.

Trait Implementations

impl Clone for EngineConfig

fn clone(&self) -> EngineConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for EngineConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for EngineConfig

fn default() -> Self

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for EngineConfig

where EngineConfig: Default,

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Serialize for EngineConfig

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.