Struct GenerateOptions

pub struct GenerateOptions<'a> {

    pub min_length: Option<i64>,
    pub max_length: Option<i64>,
    pub max_new_tokens: Option<i64>,
    pub early_stopping: Option<bool>,
    pub num_return_sequences: Option<i64>,
    pub num_beams: Option<i64>,
    pub num_beam_groups: Option<i64>,
    pub do_sample: Option<bool>,
    pub temperature: Option<f64>,
    pub top_k: Option<i64>,
    pub top_p: Option<f64>,
    pub repetition_penalty: Option<f64>,
    pub length_penalty: Option<f64>,
    pub no_repeat_ngram_size: Option<i64>,
    pub diversity_penalty: Option<f64>,
    pub decoder_start_token_id: Option<i64>,
    pub forced_bos_token_id: Option<i64>,
    pub prefix_allowed_tokens_fn: Option<PrefixAllowedFunction<'a>>,
    pub bad_word_ids: Option<&'a Vec<Vec<i64>>>,
    pub output_scores: bool,
}

Type alias for a function defining allowed tokens based on current tokens generated. This function should take a batch_id and associated tensor of already generated tokens and should return a vector of allowed tokens. This is useful for controlled generation, i.e. deterministic generation of a token continuation if a sequence of token occurs.

Generation options for text generation.

When provided to a generate method, these options will take priority over the GenerateConfig used to create the LanguageGenerator. Some of these options may be left as None, options without a value will individually default to the GenerateConfig.

Fields

min_length: Option<i64>

Minimum sequence length

max_length: Option<i64>

Maximum sequence length

max_new_tokens: Option<i64>

Maximum number of new tokens to generate (useful for causal generation models). Only one of max_length and max_new_tokens should be provided. When both are given, max_new_tokens is ignored and the max_length setting is used.

early_stopping: Option<bool>

Early stopping flag indicating if the beam search should stop as soon as num_beam hypotheses have been generated

num_return_sequences: Option<i64>

Number of sequences to return for each prompt text

num_beams: Option<i64>

Number of beams for beam search

num_beam_groups: Option<i64>

do_sample: Option<bool>

Sampling flag. If true, will perform top-k and/or nucleus sampling on generated tokens, otherwise greedy (deterministic) decoding

temperature: Option<f64>

Temperature setting. Values higher than 1 will improve originality at the risk of reducing relevance

top_k: Option<i64>

Top_k values for sampling tokens. Value higher than 0 will enable the feature

top_p: Option<f64>

Top_p value for Nucleus sampling, Holtzman et al.. Keep top tokens until cumulative probability reaches top_p

repetition_penalty: Option<f64>

Repetition penalty (mostly useful for CTRL decoders). Values higher than 1 will penalize tokens that have been already generated.

length_penalty: Option<f64>

Exponential penalty based on the length of the hypotheses generated

no_repeat_ngram_size: Option<i64>

Number of allowed repetitions of n-grams. Values higher than 0 turn on this feature

diversity_penalty: Option<f64>

Diversity penalty for diverse beam search. High values will enforce more difference between beam groups

decoder_start_token_id: Option<i64>

Decoder start token id

forced_bos_token_id: Option<i64>

Forced first token generated

prefix_allowed_tokens_fn: Option<PrefixAllowedFunction<'a>>

Function to control the generation process. The function should take a batch_id (i64) and a tensor of token_ids already generated and returns a Vec<i64> of allowed tokens.

bad_word_ids: Option<&'a Vec<Vec<i64>>>

List of bad word ids (may be a sequence of word ids) that will be banned during the generation

output_scores: bool

Flag indicating if text generation scores should be returned

Trait Implementations

impl<'a> Clone for GenerateOptions<'a>

fn clone(&self) -> GenerateOptions<'a>

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<'a> Default for GenerateOptions<'a>

fn default() -> GenerateOptions<'a>

Returns the “default value” for a type.

impl<'a> Copy for GenerateOptions<'a>