Struct GenerateContext

pub struct GenerateContext {

    pub prompt_tokens: Vec<u16>,
    pub prompt_cached: CachedPrompt,
    pub prefix: Tokens,
    pub suffix: Tokens,
    pub output: Option<TensorCpu<f32>>,
    pub choices: Vec<Tokens>,
    pub model_text: Vec<u8>,
    pub buffer: Vec<u8>,
    pub model_tokens: Vec<u16>,
    pub formatters: Vec<Arc<RwLock<dyn Formatter + Send + Sync>>>,
    pub instant: Option<Instant>,
    pub request: GenerateRequest,
    pub sender: Sender<Token>,
}

Fields

prompt_tokens: Vec<u16>

Tokens that are provided at first.

prompt_cached: CachedPrompt

Whether the prompt has already been processed and cached.

prefix: Tokens

Tokens that have been computed and cached.

suffix: Tokens

Tokens to be computed.

output: Option<TensorCpu<f32>>

The output of the model from the last run.

choices: Vec<Tokens>

Tokens to be chosen if this is a choose request.

model_text: Vec<u8>

Texts that are output by the model.

buffer: Vec<u8>

Model may output partial utf-8. This makes sure the output is always valid.

model_tokens: Vec<u16>

Tokens that are output by the model.

formatters: Vec<Arc<RwLock<dyn Formatter + Send + Sync>>>

Compiled BNF schema, if any.

instant: Option<Instant>

For measuring time used.

request: GenerateRequest

Generate request provided by the caller.

sender: Sender<Token>

To send back generated tokens.

Implementations

impl GenerateContext

pub async fn new( request: GenerateRequest, sender: Sender<Token>, tokenizer: &Tokenizer, ) -> Result<Self>

Trait Implementations

impl Clone for GenerateContext

fn clone(&self) -> GenerateContext

Returns a duplicate of the value.

const fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for GenerateContext

fn fmt(&self, __f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.