pub struct NgramCache { /* private fields */ }Expand description
A statistical n-gram cache: which tokens tend to follow which n-grams.
Where ngram_simple_draft takes the single most recent repeat,
this accumulates a distribution over many observations and drafts the most
likely continuation. It can be persisted, so a cache built once from a
corpus — or grown across a user’s sessions — keeps paying off.
llama.cpp consults up to three caches at once, in priority order:
- context — built from the current conversation, most specific;
- dynamic — built from this user’s past generations;
- static — built offline from a large corpus, used to validate.
Wraps common_ngram_cache.
Implementations§
Source§impl NgramCache
impl NgramCache
Sourcepub fn load(path: &str) -> Result<Self, NgramError>
pub fn load(path: &str) -> Result<Self, NgramError>
Load a cache written by Self::save.
§Errors
Returns NgramError::Failed if the file is missing or malformed, or
NgramError::Nul for an interior NUL in path.
Sourcepub fn save(&mut self, path: &str) -> Result<(), NgramError>
pub fn save(&mut self, path: &str) -> Result<(), NgramError>
Write this cache to disk.
§Errors
Returns NgramError::Failed if the file cannot be written, or
NgramError::Nul for an interior NUL in path.
Sourcepub fn merge(&mut self, other: &mut NgramCache) -> Result<(), NgramError>
pub fn merge(&mut self, other: &mut NgramCache) -> Result<(), NgramError>
Sourcepub fn update(
&mut self,
ngram_min: i32,
ngram_max: i32,
tokens: &[LlamaToken],
nnew: i32,
print_progress: bool,
) -> Result<(), NgramError>
pub fn update( &mut self, ngram_min: i32, ngram_max: i32, tokens: &[LlamaToken], nnew: i32, print_progress: bool, ) -> Result<(), NgramError>
Learn from a token sequence.
nnew is how many tokens were appended since the last call, so a live
conversation only pays for its new tokens. Upstream requires tokens to
be append-only: editing the middle invalidates the statistics and
needs a rebuild from scratch.
§Errors
Returns NgramError::Failed if llama.cpp throws.