Struct llm_base::model::ModelParameters

pub struct ModelParameters {
    pub prefer_mmap: bool,
    pub n_context_tokens: usize,
    pub inference_params: InferenceParameters,
    pub inference_prompt_params: InferenceWithPromptParameters,
}

Parameters for tuning model instances

Fields

prefer_mmap: bool

For GGML formats that support it, mmap is the default. Although mmap typically improves performance, setting this value to false may be preferred in resource-constrained environments.

n_context_tokens: usize

The context size (“memory”) the model should use when evaluating a prompt. A larger context consumes more resources, but produces more consistent and coherent responses.

inference_params: InferenceParameters

Default InferenceParameters to use when evaluating a prompt with this model.

inference_prompt_params: InferenceWithPromptParameters

Default InferenceWithPromptParameters to use when evaluating a prompt with this model.

Trait Implementations

impl Default for ModelParameters

fn default() -> Self

Returns the “default value” for a type.