Struct FitParams 

pub struct FitParams {
    pub model_params: LlamaModelParams,
    pub context_params: LlamaContextParams,
    pub margins: Vec<usize>,
    pub n_ctx_min: u32,
    pub log_level: ggml_log_level,
}

Input to fit_params.

Defaults mirror upstream common_params: unset n_ctx (0) so context size can be reduced, default model params so n_gpu_layers may be adjusted, and 1 GiB per-device memory margins.

Fields

model_params: LlamaModelParams

Starting model parameters. Only fields still at their defaults are modified.

context_params: LlamaContextParams

Starting context parameters. Set n_ctx to 0 via LlamaContextParams::with_n_ctx(None) to let fitting pick a context size.

margins: Vec<usize>

Minimum free memory to leave on each device, in bytes (one entry per device).

n_ctx_min: u32

Minimum context size when fitting must reduce n_ctx.

log_level: ggml_log_level

Minimum log level printed during fitting.

Implementations

impl FitParams

pub fn with_model_params(self, model_params: LlamaModelParams) -> Self

Override starting model parameters.

pub fn with_context_params(self, context_params: LlamaContextParams) -> Self

Override starting context parameters.

pub fn with_margins(self, margins: Vec<usize>) -> Self

Per-device memory margins in bytes (length must be at least max_devices()).

pub fn with_n_ctx_min(self, n_ctx_min: u32) -> Self

Minimum context size when fitting reduces memory by shrinking n_ctx.

pub fn with_log_level(self, log_level: ggml_log_level) -> Self

Minimum log level emitted while fitting.

Trait Implementations

impl Debug for FitParams

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for FitParams

fn default() -> Self

Returns the “default value” for a type.