postings

Inverted-index postings lists and codecs.

Supports u32 term frequencies for classical IR and f32 weights for learned sparse retrieval.

Data Model & Invariants

• Doc IDs: u32. Sparse ids are supported; smaller gaps compress better and dense ids keep dense scratch paths cheap.
• Ordering: Postings lists are always sorted by Doc ID.
• Updates: PostingsIndex supports in-memory add/delete. Raw file segments are immutable; a store or application manifest owns deletes and compaction.
• Storage: In-memory by default; optional persistence and raw file-backed segment readers.

Usage

[dependencies]
postings = "0.2"

Example (index + candidates):

use postings::{PostingsIndex, PlannerConfig};

let mut idx = PostingsIndex::new();
idx.add_document(0, &["the".to_string(), "quick".to_string(), "fox".to_string()])
    .unwrap();
idx.add_document(1, &["quick".to_string(), "brown".to_string(), "dog".to_string()])
    .unwrap();

// Conjunctive (AND) candidates.
assert_eq!(
    idx.candidates_all_terms(&["quick".to_string(), "dog".to_string()]),
    vec![1]
);

let cfg = PlannerConfig::default();
let plan = idx.plan_candidates(&["quick".to_string()], cfg);
assert!(matches!(plan, postings::CandidatePlan::Candidates(_)));

Example (learned-sparse top-k):

use postings::PostingsIndex;

let mut idx: PostingsIndex<String, f32> = PostingsIndex::new();
idx.add_weighted_document(
    0,
    &[
        ("neural".to_string(), 1.8),
        ("retrieval".to_string(), 0.4),
    ],
)
.unwrap();
idx.add_weighted_document(
    1,
    &[
        ("retrieval".to_string(), 2.6),
        ("search".to_string(), 2.2),
    ],
)
.unwrap();

let ranking = idx.top_k_weighted(&[("neural", 1.5), ("retrieval", 2.0)], 10);
assert_eq!(ranking[0].0, 1);

Examples

Runnable examples live in examples/:

• durable_roundtrip pairs postings with durability to build a crash-recoverable inverted index: update events go to a record log, snapshots to a checkpoint, and the index rebuilds from both, the persistence pattern a search engine needs to survive restarts.
• raw_segment_file writes immutable raw impact files and queries them with file-backed top-k search. Run with --features raw-segment.
• splade_weighted scores a small learned-sparse collection with f32 weights and verifies top-k sparse inner-product search.

File-backed segments

The raw-segment feature exposes postings::raw, a numeric-term segment format with a byte-backed reader and a file-backed reader. The file reader keeps the fixed directories in memory and range-reads posting payloads for the query terms. New raw files carry directory and posting-block checksums; legacy unchecked raw files remain readable. It is the path intended for large lexical and learned-sparse indexes whose posting payloads should not be rebuilt into a full PostingsIndex on every open. RawSegmentFile::resident_metadata_len and RawSegmentFile::posting_payload_len report that split for callers that enforce per-generation or per-query memory budgets. RawSegment::for_each_term_id and RawSegmentFile::for_each_term_id stream the term directory when callers need generation-level statistics without allocating a full term-id list. RawSegment::for_each_term_meta and RawSegmentFile::for_each_term_meta also stream document frequency, maximum weight, and total weight from the same directory pass.

Raw segments can be encoded from a slice, document iterator, sorted document iterator, or term-major posting lists into a Vec<u8>, caller-provided Write sink, or seekable writer. The writer API avoids requiring the final segment as one contiguous allocation when the caller wants to stream bytes into its own durability layer. When document ids are already strictly increasing, the sorted-iterator writer also avoids the encoder's whole-corpus document map before postings are written. When an external sorter or merge already has term-major lists, the term-major writer avoids doc-to-term transposition too; the seekable term-major writer also avoids a second payload pass for local-file style sinks.

RawSegmentFile::top_k_weighted_u32 scores one raw file by sparse inner product; top_k_weighted_u32_files merges exact top-k results across raw files when document ids are globally unique. The file-backed scorer uses block metadata for bounded reads and safe top-k pruning where the query weights make that possible. top_k_weighted_u32_files_and_index fuses sealed files with one live PostingsIndex<u64, u32> shard and uses the live shard's current top-k threshold to skip low-bound sealed files. Use top_k_weighted_u32_files_with_stats when you need searched/pruned segment counts for profiling. Segment pruning is layout-sensitive, so measure it against representative segment construction before treating it as a storage win. Use lexir::raw for BM25 over one or more raw files.

This is not a full index lifecycle by itself: callers still own term-id mapping, commit publication, deletes, compaction, and crash-safety policy. Pair raw files with durability, segstore sidecars, or an application manifest when those guarantees are needed.

Features

• serde: enable serde for the in-memory structures.
• persistence: enable save/load helpers via durability + postcard.
• sbits: enable succinct monotone sequences (Elias-Fano) under postings::codec::ef.
• positional: enable positional postings (postings::positional::PositionalIndex).
• cnk-compression: enable optional compressed-candidate helpers under postings::positional::cnk_candidates.
• raw-segment: enable the experimental checked byte- and file-backed raw segment reader.

Optional: positional postings

Enable positional postings behind a feature flag:

[dependencies]
postings = { version = "0.2", features = ["positional"] }

Then use postings::positional::PositionalIndex for phrase/proximity evaluation. phrase_match_strs and near_match_terms_strs accept borrowed query terms when a parser already holds &strs. PosingsIndex remains as the historical name from the older posings crate.

cnk-compression is a helper for sorted candidate doc-id sets produced by positional workflows. It is not a storage backend, postings codec, or lifecycle layer.

Development

cargo test

License

MIT OR Apache-2.0