Skip to main content

batch_generate_with_timeout

oxibonsai_runtime::batch_engine

Function batch_generate_with_timeout 

Source 
pub fn batch_generate_with_timeout(
    engine: &mut InferenceEngine<'_>,
    prompts: &[Vec<u32>],
    config: &BatchConfig,
) -> Vec<RuntimeResult<BatchResult>> ⓘ
Expand description

Process a batch with timeout per request.

Uses BatchConfig to control generation limits and timeouts. If a request exceeds its timeout, it is terminated and marked with FinishReason::Timeout.