Struct TextSegment 

pub struct TextSegment { /* private fields */ }

One shard’s inverted segment.

postings maps token → (key → tf) so a pruned list is PROBED per accumulated candidate (O(candidates)) instead of walked (O(postings)); docs keeps each row’s original text so an update removes exactly its own tokens (re-tokenize the old text) instead of scanning every posting list.

Implementations

impl TextSegment

pub fn freeze_docs(&mut self, keys: &[Vec<u8>]) -> Option<FrozenBucket>

Freeze keys out of the hot index: read each document’s terms, term frequencies and positions blobs FIRST (withdraw consumes the stored source text they are derived from), then withdraw — reclaiming the doc record, its postings slots and its positions in one motion. Keys not indexed are skipped. None when nothing froze.

impl TextSegment

pub fn matches(&self, query: &[u8], limit: usize) -> Vec<TextMatch>

BM25-ranked matches for query (tokenized with the same rules; OR semantics), best limit hits, score-descending.

MaxScore pruning: query tokens process rarest-first; once the running top-limit threshold exceeds the summed upper bounds of the remaining (commoner) tokens, documents seen ONLY in those lists can no longer enter — their lists are then probed per accumulated doc instead of walked. Selection is a bounded heap over borrowed keys (no per-candidate allocation).

pub fn matches_scored( &self, query: &[u8], limit: usize, stats: Option<&CorpusStats>, ) -> Vec<TextMatch>

TextSegment::matches, scored against externally-supplied corpus statistics instead of this shard’s local ones.

None uses the local stats — the shard-local BM25 that matches has always used, byte-identical. Some is the global-BM25 path: a cross-shard query aggregates each shard’s n_docs, avgdl and per-query-token df into one CorpusStats and scores every shard against it, so hits from different shards are comparable. The MaxScore upper bound uses the same injected numbers, so pruning stays a valid bound.

A query token absent from THIS shard’s postings contributes no score here regardless — its documents live on other shards — so only the idf (via global df) crosses shard boundaries, never a posting.

impl TextSegment

pub fn stats(&self) -> TextStats

Live counters — O(1): every term is a running counter maintained at the mutation sites (the per-tick stat walk of these structures was a consumer’s measured tiering idle/write-load CPU term, F16a). [Self::recompute_stats] is the walking reference the tests hold these to.

pub fn contains(&self, key: &[u8]) -> bool

Verify hook: is key indexed here?

pub fn docs(&self) -> u64

This shard’s live document count — the Σ n_docs half of the same global-BM25 sum Self::total_len feeds.

Its own accessor because the number is a len() and Self::stats is not: stats also computes approx_bytes, which walks every token, every posting and every positional blob. Pass 1 of a cross-shard query read stats().docs and dropped the rest, so a phrase query over a million documents spent 82% of its CPU re-measuring the index’s memory footprint once per shard per query.

pub fn total_len(&self) -> u64

This shard’s total document length in tokens (unweighted) — one of the three numbers a cross-shard query sums for global BM25 (step 4b, pass 1): global avgdl = Σ total_len / Σ n_docs.

pub fn local_df(&self, token: &[u8]) -> u32

This shard’s document frequency for token — how many local documents contain it. Summed across shards for one query token’s global df. 0 when the token is absent here.

impl TextSegment

pub fn phrase_matches( &self, phrase: &[u8], limit: usize, stats: Option<&CorpusStats>, ) -> Vec<TextMatch>

BM25-ranked documents that contain phrase’s tokens adjacent and in order, best limit hits, score-descending.

A single-token phrase is an ordinary term query (adjacency is trivial). A multi-token phrase needs the positional side-channel: on a segment created without positions it returns empty. stats injects global corpus statistics (the two-pass cross-shard path); None scores shard-local.

pub fn matches_query( &self, text: &[u8], limit: usize, stats: Option<&CorpusStats>, ) -> Vec<TextMatch>

BM25-ranked matches for a query text that may mix bare terms and double-quoted phrases (foo "quick brown" bar), best limit hits.

The query is the OR of its clauses — each bare term and each phrase — scored by the summed BM25 an OR query would give, with a phrase clause contributing only to documents where its tokens are adjacent. With no quoted phrase in text this is byte-identical to TextSegment::matches_scored (the pruned hot path); the phrase branch trades that pruning for exactness and is what the positional side-channel exists for. stats injects global corpus statistics (the cross-shard path); None scores shard-local.

pub fn matches_query_typo( &self, text: &[u8], limit: usize, stats: Option<&CorpusStats>, typo: u32, ) -> Vec<TextMatch>

TextSegment::matches_query with a typo budget: each bare term also matches the dictionary terms within typo edits of it (TYPO n). A budget of 0 is the exact query, byte-identical.

Only bare terms are fuzzed — a phrase asks for those exact tokens adjacent, and a prefix is already an inexact match, so widening either would answer a question the user did not ask.

pub fn matches_query_with( &self, text: &[u8], limit: usize, opts: QueryOpts<'_>, ) -> Vec<TextMatch>

TextSegment::matches_query with every option a MATCH carries: injected corpus statistics, a typo budget, the field positions the query is restricted to (IN <field…>, empty = every field), and the non-scoring predicates it must satisfy (FILTER).

A scoped query is a field-scoped BM25, not a filter over whole-document scores: frequency, length and document frequency all come from the wanted fields alone, so a match in a short title is not diluted by a long body that never mentioned the term.

pub fn matches_query_faceted( &self, text: &[u8], limit: usize, opts: QueryOpts<'_>, facets: &[Facet<'_>], ) -> FacetedMatches

TextSegment::matches_query_with, additionally counting the values of stored fields over the whole match set.

Counted before the top-K, because a facet is about what matched and the page is only limit of it. FILTER restricts the count — a filtered-out document did not match — but DISTINCT does not: collapsing decides which documents are shown, not which matched.

pub fn matches_prefix( &self, prefix: &[u8], limit: usize, stats: Option<&CorpusStats>, ) -> Vec<TextMatch>

BM25-ranked documents holding any indexed term that begins with prefix — a search-as-you-type prefix* query, scored as the OR of its expansion terms, best limit hits.

prefix is ASCII-lowercased first so it matches the stored token form (Latin tokens are lowercased on the way in). This scans the term dictionary; an ordered dictionary would binary-search to the prefix range instead — the cost it trades is one linear pass over the distinct terms, weighed against the write-path cost of keeping the dictionary ordered.

pub fn query_df_terms(&self, text: &[u8]) -> Vec<Vec<u8>>

The terms whose document frequency a cross-shard query aggregates for global BM25: the bare tokens, every phrase’s tokens, and every expansion of a word* prefix (expanded against THIS shard’s dictionary, since which terms share the prefix is shard-local). Deduplicated. For a query with no prefix this is exactly the tokenized query, so pass 1 is unchanged.

pub fn query_df_terms_typo(&self, text: &[u8], typo: u32) -> Vec<Vec<u8>>

TextSegment::query_df_terms with a typo budget, so a fuzzed term’s neighbours get their df aggregated globally too.

pub fn query_df_in( &self, text: &[u8], opts: QueryOpts<'_>, ) -> Vec<(Vec<u8>, u32)>

The document frequency this shard contributes for each of a query’s terms, over the query’s field scope.

Unscoped this is the ordinary posting-list length. Scoped it is the number of documents holding the term in the wanted fields — counted by the same walk that would score them, because summing stored per-field counts would count a document twice when it holds the term in two of the fields.

impl TextSegment

pub fn highlight_spans( &self, key: &[u8], query: &[u8], ) -> Vec<(usize, Vec<(usize, usize)>)>

Byte spans in key’s stored fields where query matched: a bare term highlights every occurrence, a phrase only its adjacent runs. Returns (field_index, spans) for each field with a match, each span list sorted and de-duplicated. Empty when key is not indexed.

It re-analyses the winning document’s own text — the fields are stored for re-indexing already — so it needs no positional side-channel: highlighting a handful of hits is cheap.

impl TextSegment

pub fn total_len_in(&self, fields: &[usize]) -> u64

Corpus token total over fields — the numerator of a field-scoped average document length, and what each shard reports in the first pass of a scoped cross-shard query. Empty fields = the whole document.

impl TextSegment

pub fn new() -> Self

Empty segment (ranking only, no positional postings).

pub fn with_positions() -> Self

Empty segment that records token positions for phrase, proximity and highlight queries — the WITH POSITIONS form. Every other operation behaves identically; only the positional side-channel (and its memory cost) is added.

pub fn with_shape(shape: SegmentShape) -> Self

Empty segment shaped by what an index declares.

Each optional channel exists only when the declaration calls for it, so an index pays for what it asked for and nothing else.

pub fn has_positions(&self) -> bool

Whether this segment records token positions.

pub fn field_arity(&self) -> usize

How many fields this segment scores separately; 1 when it keeps no per-field breakdown (a single-field index needs none).

pub fn value_arity(&self) -> usize

How many value fields this segment stores per document; 0 when it stores none.

pub fn stored_value(&self, key: &[u8], field: usize) -> Option<&[u8]>

One row’s stored value for a declared value field, as raw bytes. None when the row is not indexed here, the field was not declared, or this document has no value for it.

The cross-shard merge needs each returned hit’s sort value, and it is cheaper to look it up for the handful of hits a shard returns than to carry it through the ranking.

pub fn apply(&mut self, key: &[u8], text: Option<&[u8]>)

(Re-)index one row’s text (None = row removed / excluded).

Single-field sugar over TextSegment::apply_fields at neutral weight, so the two paths cannot diverge.

pub fn apply_fields(&mut self, key: &[u8], fields: Option<&[(Vec<u8>, f32)]>)

(Re-)index one row from its declared fields, each with its BM25 weight. None removes the row.

A weight scales that field’s term frequencies, so a term in a weight-3 title counts as if seen three times. Document length is summed unweighted: length normalisation measures how much text there is to dilute a match, and weighting it would make a heavily-weighted field penalise itself.

pub fn apply_doc( &mut self, key: &[u8], fields: Option<&[(Vec<u8>, f32)]>, values: &[Option<&[u8]>], )

TextSegment::apply_fields, also storing the row’s declared value fields (VALUES) so FILTER and friends can read them back per document. values is positional against the declaration; a short slice leaves the rest absent.

Trait Implementations

impl Debug for TextSegment

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for TextSegment

fn default() -> TextSegment

Returns the “default value” for a type.