Skip to main content

batch_generate

Function batch_generate 

Source
pub fn batch_generate(
    engine: &mut InferenceEngine<'_>,
    prompts: &[Vec<u32>],
    max_tokens: usize,
) -> Vec<RuntimeResult<BatchResult>> 
Expand description

Process a batch of prompts sequentially (sharing the engine).

Each prompt is processed independently: the engine state is reset between prompts. Returns one result per prompt.