Struct InferenceParameters

pub struct InferenceParameters {
    pub n_threads: usize,
    pub n_batch: usize,
    pub top_k: usize,
    pub top_p: f32,
    pub repeat_penalty: f32,
    pub temperature: f32,
    pub bias_tokens: TokenBias,
    pub repetition_penalty_last_n: usize,
}

The parameters for text generation.

This needs to be provided during all inference calls, but can be changed between calls.

Fields

n_threads: usize

The number of threads to use.

n_batch: usize

Controls batch/chunk size for prompt ingestion in InferenceSession::feed_prompt.

top_k: usize

The top K words by score are kept during sampling.

top_p: f32

The cumulative probability after which no more words are kept for sampling.

repeat_penalty: f32

The penalty for repeating tokens. Higher values make the generation less likely to get into a loop, but may harm results when repetitive outputs are desired.

temperature: f32

Temperature (randomness) used for sampling. A higher number is more random.

bias_tokens: TokenBias

A list of tokens to bias against in the process of generation.

repetition_penalty_last_n: usize

The number of tokens to consider for the repetition penalty.

Trait Implementations

impl Clone for InferenceParameters

fn clone(&self) -> InferenceParameters

Returns a duplicate of the value.

const fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for InferenceParameters

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for InferenceParameters

fn default() -> Self

Returns the “default value” for a type.

impl PartialEq for InferenceParameters

fn eq(&self, other: &InferenceParameters) -> bool

Tests for self and other values to be equal, and is used by ==.

const fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl StructuralPartialEq for InferenceParameters