Struct InferenceRequest

pub struct InferenceRequest<'a> {
    pub prompt: &'a str,
    pub parameters: Option<&'a InferenceParameters>,
    pub play_back_previous_tokens: bool,
    pub maximum_token_count: Option<usize>,
}

Settings specific to InferenceSession::infer.

Fields

prompt: &'a str

The prompt to feed to the model.

parameters: Option<&'a InferenceParameters>

The parameters to use during this inference attempt. If not specified, this will default to the parameters specified in the model.

play_back_previous_tokens: bool

Whether or not to call the callback with the previous tokens that were encountered in this session.

You likely want to turn this on if you’re using a session that has been rehydrated from a snapshot.

maximum_token_count: Option<usize>

The maximum number of tokens to generate.

Trait Implementations

impl<'a> Clone for InferenceRequest<'a>

fn clone(&self) -> InferenceRequest<'a>

Returns a duplicate of the value.

const fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<'a> Debug for InferenceRequest<'a>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'a> Default for InferenceRequest<'a>

fn default() -> InferenceRequest<'a>

Returns the “default value” for a type.

impl<'a> PartialEq for InferenceRequest<'a>

fn eq(&self, other: &InferenceRequest<'a>) -> bool

Tests for self and other values to be equal, and is used by ==.

const fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl<'a> Copy for InferenceRequest<'a>

impl<'a> StructuralPartialEq for InferenceRequest<'a>