pub struct QueryGrams { /* private fields */ }Expand description
Streaming query n-gram state.
This state accepts one character at a time, can be cloned while traversing an automaton, and can emit grams incrementally. It uses the same compact bigram-priority model as indexing, but emits the minimum number of grams that cover the input stream. Its state space is fully represented by the content buffer and can be shrunk on demand when only part of the stream needs to be retained.
Implementations§
Source§impl QueryGrams
impl QueryGrams
Sourcepub fn state(&self) -> (u32, u64)
pub fn state(&self) -> (u32, u64)
Returns the canonical, position-independent fingerprint of the active window as
(content_len, content).
content_len is the number of active bytes and content is exactly those bytes packed into
a u64 (newest in the low byte), with all higher bytes masked off. The active window starts
two bytes before the front boundary (the oldest candidate still able to start a gram) and
runs to the newest fed byte: the front boundary is a bigram, so both of its source bytes —
which determine its priority and the left half of the next bigram — must be retained for the
state to fully predict future grams. Everything before that has already been covered by an
emitted gram and can no longer influence future grams, so it is excluded; this makes the
fingerprint independent of the absolute stream position and of already-drained history.
PartialEq, Eq and Hash are all defined in terms of it, so two states with
identical active windows compare and hash equal.
When the queue is empty there is no front boundary, so the whole packed buffer is active:
(0, 0) for a fresh state and (1, last_byte) once a single trailing byte has been
retained (e.g. after consume_first collapses). The retained byte is part of the state
because it becomes the left half of the next bigram.
Sourcepub fn min_priority(&self) -> u32
pub fn min_priority(&self) -> u32
Returns the smallest active boundary priority.
Sourcepub fn append_char<F>(&mut self, c: char, consumer: F)
pub fn append_char<F>(&mut self, c: char, consumer: F)
Appends a single character to the n-gram state.
The character is index-folded using the casefold crate and may trigger one or more grams.
Sourcepub fn append_byte<F>(&mut self, right: u8, consumer: F)
pub fn append_byte<F>(&mut self, right: u8, consumer: F)
Appends a single already index-folded byte to the n-gram state.
This is the byte-level counterpart of append_char: the caller has
already index-folded the character (e.g. buffered it as a byte), so no further folding is
applied. May trigger one or more grams.
Trait Implementations§
Source§impl Clone for QueryGrams
impl Clone for QueryGrams
Source§fn clone(&self) -> QueryGrams
fn clone(&self) -> QueryGrams
1.0.0 (const: unstable) · Source§fn clone_from(&mut self, source: &Self)
fn clone_from(&mut self, source: &Self)
source. Read more