Struct InferRequest 

pub struct InferRequest<'a> {
    pub prompt: &'a str,
    pub max_tokens: u32,
    pub images: &'a [Vec<u8>],
}

One inference request.

A struct rather than a growing parameter list. images was added after the fact and would have broken every implementation had it been positional; temperature, grammars, and stop sequences are all coming, and each would do the same. Adding a field here with a sensible default does not.

Fields

prompt: &'a str

What to generate from.

max_tokens: u32

Upper bound on generated tokens.

images: &'a [Vec<u8>]

Images accompanying the prompt, already loaded by the host.

Empty for ordinary text inference. A backend whose model has no vision capability should fail loudly rather than ignore these — silently dropping an image produces an answer about nothing, which is worse than an error because it looks like a bad model.

Implementations

impl<'a> InferRequest<'a>

pub fn new(prompt: &'a str, max_tokens: u32) -> Self

A text-only request.

Trait Implementations

impl<'a> Debug for InferRequest<'a>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'a> Default for InferRequest<'a>

fn default() -> InferRequest<'a>

Returns the “default value” for a type.