Struct Sampling

pub struct Sampling {
    pub temperature: Option<f64>,
    pub top_k: Option<u32>,
    pub top_p: Option<f64>,
    pub frequency_penalty: Option<f64>,
    pub presence_penalty: Option<f64>,
}

Sampling controls how the tokens (“words”) are selected for the completion.

Fields

temperature: Option<f64>

A temperature encourages the model to produce less probable outputs (“be more creative”). Values are expected to be between 0 and 1. Try high values for a more random (“creative”) response.

top_k: Option<u32>

Introduces random sampling for generated tokens by randomly selecting the next token from the k most likely options. A value larger than 1 encourages the model to be more creative. Set to 0 to get the same behaviour as None.

top_p: Option<f64>

Introduces random sampling for generated tokens by randomly selecting the next token from the smallest possible set of tokens whose cumulative probability exceeds the probability top_p. Set to 0 to get the same behaviour as None.

frequency_penalty: Option<f64>

When specified, this number will decrease (or increase) the likelihood of repeating tokens that were mentioned prior in the completion. The penalty is cumulative. The more a token is mentioned in the completion, the more its probability will decrease. A negative value will increase the likelihood of repeating tokens.

presence_penalty: Option<f64>

The presence penalty reduces the likelihood of generating tokens that are already present in the generated text (repetition_penalties_include_completion=true) respectively the prompt (repetition_penalties_include_prompt=true). Presence penalty is independent of the number of occurrences. Increase the value to reduce the likelihood of repeating text. An operation like the following is applied:

logits[t] -> logits[t] - 1 * penalty

where logits[t] is the logits for any given token. Note that the formula is independent of the number of times that a token appears.

Implementations

impl Sampling

pub const MOST_LIKELY: Self

Always chooses the token most likely to come next. Choose this if you do want close to deterministic behaviour and do not want to apply any penalties to avoid repetitions.

Trait Implementations

impl Default for Sampling

fn default() -> Self

Returns the “default value” for a type.