Struct llm_base::InferenceParameters

pub struct InferenceParameters {
    pub n_threads: usize,
    pub n_batch: usize,
    pub top_k: usize,
    pub top_p: f32,
    pub repeat_penalty: f32,
    pub temperature: f32,
    pub bias_tokens: TokenBias,
}

The parameters that drive text generation.

Fields

n_threads: usize

The number of threads to use.

n_batch: usize

Controls batch/chunk size for prompt ingestion in InferenceSession::feed_prompt.

top_k: usize

The top K words by score are kept during sampling.

top_p: f32

The cumulative probability after which no more words are kept for sampling.

repeat_penalty: f32

The penalty for repeating tokens. Higher values make the generation less likely to get into a loop, but may harm results when repetitive outputs are desired.

temperature: f32

Temperature (randomness) used for sampling. A higher number is more random.

bias_tokens: TokenBias

A list of tokens to bias against in the process of generation.

Trait Implementations

impl Clone for InferenceParameters

fn clone(&self) -> InferenceParameters

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for InferenceParameters

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for InferenceParameters

fn default() -> Self

Returns the “default value” for a type.

impl PartialEq<InferenceParameters> for InferenceParameters

fn eq(&self, other: &InferenceParameters) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

This method tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl StructuralPartialEq for InferenceParameters