pub struct InferenceParameters {
pub n_threads: usize,
pub n_batch: usize,
pub top_k: usize,
pub top_p: f32,
pub repeat_penalty: f32,
pub temperature: f32,
pub bias_tokens: TokenBias,
pub repetition_penalty_last_n: usize,
}
Expand description
The parameters for text generation.
This needs to be provided during all inference calls, but can be changed between calls.
Fields§
§n_threads: usize
The number of threads to use.
n_batch: usize
Controls batch/chunk size for prompt ingestion in InferenceSession::feed_prompt.
top_k: usize
The top K words by score are kept during sampling.
top_p: f32
The cumulative probability after which no more words are kept for sampling.
repeat_penalty: f32
The penalty for repeating tokens. Higher values make the generation less likely to get into a loop, but may harm results when repetitive outputs are desired.
temperature: f32
Temperature (randomness) used for sampling. A higher number is more random.
bias_tokens: TokenBias
A list of tokens to bias against in the process of generation.
repetition_penalty_last_n: usize
The number of tokens to consider for the repetition penalty.
Trait Implementations§
Source§impl Clone for InferenceParameters
impl Clone for InferenceParameters
Source§fn clone(&self) -> InferenceParameters
fn clone(&self) -> InferenceParameters
Returns a duplicate of the value. Read more
1.0.0 · Source§const fn clone_from(&mut self, source: &Self)
const fn clone_from(&mut self, source: &Self)
Performs copy-assignment from
source
. Read moreSource§impl Debug for InferenceParameters
impl Debug for InferenceParameters
Source§impl Default for InferenceParameters
impl Default for InferenceParameters
Source§impl PartialEq for InferenceParameters
impl PartialEq for InferenceParameters
impl StructuralPartialEq for InferenceParameters
Auto Trait Implementations§
impl Freeze for InferenceParameters
impl RefUnwindSafe for InferenceParameters
impl Send for InferenceParameters
impl Sync for InferenceParameters
impl Unpin for InferenceParameters
impl UnwindSafe for InferenceParameters
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more