Struct EngineConfig 

pub struct EngineConfig {
    pub model_path: String,
    pub tokenizer_path: Option<String>,
    pub context_size: Option<usize>,
    pub num_threads: usize,
    pub sampler: SamplerConfig,
    pub prefill_chunk_size: usize,
    pub offload_policy: OffloadPolicy,
}

Configuration for the inference engine.

Fields

model_path: String

Path to the GGUF model file.

tokenizer_path: Option<String>

Path to the tokenizer JSON file (if not embedded in GGUF).

context_size: Option<usize>

Context size override (None = use model default).

num_threads: usize

Number of threads for parallel computation.

sampler: SamplerConfig

Sampling configuration.

prefill_chunk_size: usize

Prefill chunk size: how many prompt tokens to process per forward call.

Set to 0 or usize::MAX to process the entire prompt in one batch. Smaller values reduce peak memory usage for long prompts at the cost of slightly higher overhead from multiple forward calls. Default: 512.

offload_policy: OffloadPolicy

CPU/disk offload policy.

Controls which model weights are kept resident in RAM and which are evicted to disk and reloaded on demand.

Default: OffloadPolicy::None — all weights remain in RAM, matching classic llama.cpp behaviour.

Implementations

impl EngineConfig

pub fn with_offload(self, policy: OffloadPolicy) -> Self

Set the CPU/disk offload policy, consuming self and returning the updated config (builder pattern).

Trait Implementations

impl Clone for EngineConfig

fn clone(&self) -> EngineConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for EngineConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for EngineConfig

fn default() -> Self

Returns the “default value” for a type.