Struct ModelParameters

pub struct ModelParameters {
    pub prefer_mmap: bool,
    pub n_context_tokens: usize,
    pub inference_parameters: InferenceParameters,
}

Parameters for tuning model instances

Fields

prefer_mmap: bool

For GGML formats that support it, mmap is the default. Although mmap typically improves performance, setting this value to false may be preferred in resource-constrained environments.

n_context_tokens: usize

The context size (“memory”) the model should use when evaluating a prompt. A larger context consumes more resources, but produces more consistent and coherent responses.

inference_parameters: InferenceParameters

Default InferenceParameters to use when evaluating a prompt with this model.

Trait Implementations

impl Default for ModelParameters

fn default() -> Self

Returns the “default value” for a type.