Skip to main content

ngram_cache_draft

Function ngram_cache_draft 

Source
pub fn ngram_cache_draft(
    tokens: &[LlamaToken],
    n_draft: i32,
    ngram_min: i32,
    ngram_max: i32,
    context: Option<&mut NgramCache>,
    dynamic: Option<&mut NgramCache>,
    statik: Option<&mut NgramCache>,
) -> Result<Vec<LlamaToken>, NgramError>
Expand description

Draft a continuation from up to three caches.

Any cache may be None. tokens must be non-empty — the last token seeds the lookup — and the returned draft excludes it.

§Errors

Returns NgramError::InvalidArg for empty tokens, or NgramError::Failed if llama.cpp throws.