pub struct SegmentIndex { /* private fields */ }Expand description
One segment’s postings and lengths, addressed by dense row id.
§Why lengths are dense arrays and not a map
Rows are dense and ascending by the task 07 invariant, so a
BTreeMap<(row, field), u32> spends a pointer-chasing descent — about
seventeen levels at 171,000 documents — on what an array answers with one
index. The scorer performs that lookup once per scored posting, so the
descent was among the largest per-posting costs in the engine.
§Why the term table is a hash map and not a tree
Accumulation looks a term up once per (document, term, field): tens of
millions of probes on a real corpus, against a table with millions of
entries. A B-tree descent is ~20 pointer-chasing key compares per
probe; an xxh3 probe is one hash and usually one compare. The sorted
view the seal needs is materialized ONCE, in Self::postings,
instead of being maintained on every insert.
Implementations§
Source§impl SegmentIndex
impl SegmentIndex
Sourcepub fn postings(&self) -> impl Iterator<Item = (&TermKey, &PostingList)>
pub fn postings(&self) -> impl Iterator<Item = (&TermKey, &PostingList)>
Returns the posting lists, ordered by term then field.
The sorted view is built here, once per call, rather than being maintained on every insert; seal is its one hot caller.
Sourcepub fn posting_list(&self, term: &[u8], field: FieldId) -> Option<&PostingList>
pub fn posting_list(&self, term: &[u8], field: FieldId) -> Option<&PostingList>
Returns one term’s posting list within one field.
Sourcepub fn field_length(&self, row: u32, field: FieldId) -> u32
pub fn field_length(&self, row: u32, field: FieldId) -> u32
Returns the analyzed token count of one row’s field.
Sourcepub fn fields(&self) -> impl Iterator<Item = FieldId> + '_
pub fn fields(&self) -> impl Iterator<Item = FieldId> + '_
Returns the fields this segment recorded lengths for, ascending.
Sourcepub fn field_lengths(&self, field: FieldId) -> Option<&[u32]>
pub fn field_lengths(&self, field: FieldId) -> Option<&[u32]>
Returns one field’s dense length array, if the field was ever set.
The slice is exactly Self::row_count long, so a caller scoring a
single unweighted field can borrow it rather than rebuild it. A
linear scan over the field table is right here: a schema has a
handful of fields, and a map would trade an index for a descent.
Sourcepub fn document_length(&self, row: u32) -> u32
pub fn document_length(&self, row: u32) -> u32
Returns the analyzed token count of one row across every field.
Sourcepub fn total_tokens(&self) -> u64
pub fn total_tokens(&self) -> u64
Returns the segment’s total analyzed token count.
Sourcepub fn push_document(
&mut self,
analyzer: &Analyzer,
document: &Document,
) -> Result<u32, IndexError>
pub fn push_document( &mut self, analyzer: &Analyzer, document: &Document, ) -> Result<u32, IndexError>
Appends one analyzed document as the next dense row.
§Errors
Returns IndexError::Postings when the analyzer produces a token
stream a posting list refuses, which would be a tokenizer contract
violation rather than bad input.
Sourcepub fn replace_document(
&mut self,
analyzer: &Analyzer,
row: u32,
previous: &Document,
replacement: &Document,
) -> Result<(), IndexError>
pub fn replace_document( &mut self, analyzer: &Analyzer, row: u32, previous: &Document, replacement: &Document, ) -> Result<(), IndexError>
Replaces one existing row’s document in place.
previous must be the document the row was indexed from. The row’s
postings for every term of previous are removed, a term left with
no posting is dropped, the row’s lengths are reset, and replacement
is folded in at the same row. The result answers exactly as a
segment rebuilt with the row’s document swapped, and the cost is
proportional to the two documents, not to the segment.
§Errors
Returns IndexError::ReplacedRowOutOfRange for a row the segment
does not hold, IndexError::ReplacedPostingMissing when previous
is not what the row was indexed from, and IndexError::Postings
as Self::push_document. On an error the segment is partly
updated; the caller discards it.
Sourcepub fn push_documents(
&mut self,
analyzer: &Analyzer,
documents: &[Document],
threads: usize,
) -> Result<(), IndexError>
pub fn push_documents( &mut self, analyzer: &Analyzer, documents: &[Document], threads: usize, ) -> Result<(), IndexError>
Appends many documents, analyzing them in parallel.
The result is identical to pushing the same documents one at a time in slice order: analysis is independent per document and runs on explicit scoped threads, while accumulation happens strictly in row order on the calling thread — so nothing about the outcome depends on scheduling, and the single-writer invariant is untouched.
threads is the caller’s policy; one or fewer runs sequentially.
Analysis is batched so the in-flight token working set stays
proportional to the batch, never to the corpus.
§Errors
As Self::push_document. On an error, documents before the
failing one remain indexed, exactly as a sequential loop would
leave them.