pub struct QueryGrams { /* private fields */ }Expand description
Streaming query n-gram state.
This state accepts one character at a time, can be cloned while traversing an automaton, and can emit grams incrementally. It uses the same compact bigram-priority model as indexing, but emits the minimum number of grams that cover the input stream. Its state space is fully represented by the content buffer and can be shrunk on demand when only part of the stream needs to be retained.
§Consumer callback
Every emitting method (append_char, append_byte,
flush, consume_first) takes a consumer of shape
FnMut(NGram, u32, Option<u8>, &[u8]), called once per emitted gram with:
gram— the compactNGramidentifier to look up in an index.end— the position of the character just after the gram in the fed stream.follow— that following (index-folded) byte, when it has already been fed;Noneat the current stream end.bytes— the gram’s index-folded bytes, in reading order. They are borrowed from a stack buffer that is only valid for the duration of the call, so a consumer that needs to keep them must copy them. Nothing is allocated per gram, and a consumer that ignores the argument optimizes back to code identical to not reporting the bytes at all.
Implementations§
Source§impl QueryGrams
impl QueryGrams
Sourcepub fn state(&self) -> (u32, u64)
pub fn state(&self) -> (u32, u64)
Returns the canonical, position-independent fingerprint of the active window as
(content_len, content).
content_len is the number of active bytes and content is exactly those bytes packed into
a u64 (newest in the low byte), with all higher bytes masked off. The active window starts
two bytes before the front boundary (the oldest candidate still able to start a gram) and
runs to the newest fed byte: the front boundary is a bigram, so both of its source bytes —
which determine its priority and the left half of the next bigram — must be retained for the
state to fully predict future grams. Everything before that has already been covered by an
emitted gram and can no longer influence future grams, so it is excluded; this makes the
fingerprint independent of the absolute stream position and of already-drained history.
PartialEq, Eq and Hash are all defined in terms of it, so two states with
identical active windows compare and hash equal.
When the queue is empty there is no front boundary, so the whole packed buffer is active:
(0, 0) for a fresh state and (1, last_byte) once a single trailing byte has been
retained (e.g. after consume_first collapses). The retained byte is part of the state
because it becomes the left half of the next bigram.
Sourcepub fn min_priority(&self) -> u32
pub fn min_priority(&self) -> u32
Returns the smallest active boundary priority, or u32::MAX when there is no active
boundary left to consume.
Callers use this to repeatedly drain the lowest-priority boundary across a set of states
(e.g. the regex state-set reducer). A state with an empty queue has nothing left to consume,
so it must report the maximum priority: otherwise it would be selected ahead of states that
can still make progress, consume_first on it would be a no-op, and the reducer would loop
forever.
Sourcepub fn append_char<F>(&mut self, c: char, consumer: F)
pub fn append_char<F>(&mut self, c: char, consumer: F)
Appends a single character to the n-gram state.
The character is index-folded using the casefold crate and may trigger one or more grams.
See the type-level docs for the consumer arguments.
Sourcepub fn append_byte<F>(&mut self, right: u8, consumer: F)
pub fn append_byte<F>(&mut self, right: u8, consumer: F)
Appends a single already index-folded byte to the n-gram state.
This is the byte-level counterpart of append_char: the caller has
already index-folded the character (e.g. buffered it as a byte), so no further folding is
applied. May trigger one or more grams.