docs-search 0.3.0

Client-side search index for docs sites: FST term dictionary, tf postings with full BM25, optional quantized-vector section (vectors feature), wasm32-clean bytes-in/bytes-out format (docs-core extension program, SPEC.md REQ-Q)
Documentation

docs-search

Client-side search index for documentation sites: an FST term dictionary with tf postings and full BM25 ranking, plus an optional quantized-vector section (the vectors feature) for cosine nearest-neighbor queries, all serialized as a single deterministic binary blob. Bytes-in/bytes-out: a native docs compiler writes it, a wasm32 browser runtime queries it. Part of the docs-core extension program (docs-core/SPEC.md REQ-Q).

Module map

Module Contents
IndexBuilder document ingestion (add_doc), deterministic serialization (build)
Index total decoding (from_bytes), AND queries with BM25 (query), cosine vectors (query_vector, vectors feature)
Hit one scored result: doc id + score
IndexError typed, exhaustive build/decode errors; decoding never panics

Format (v2)

magic   "DSIDX2"
u32     format version (2)
u64     document count
u64     FST byte length
u64     vectors-section byte length (0 when absent)
u32     vector dimension (0 when absent)
u64 * n document byte-lengths
bytes   FST dictionary: term -> postings offset
bytes   postings: concatenated [u32 byte_len][ (doc u32, tf u16) * ]
bytes   vectors (vectors feature): [u32 count][ (doc u32, scale f32, norm f32, dim * i8) * ]

Format 1 (DSIDX1, presence bitmaps, unit tf) is retired; a v1 blob fails decoding with InvalidMagic. Quantization is fixed max-abs scaling into i8 with per-vector scale + norm - deterministic arithmetic, no model.

Serialization is byte-for-byte reproducible for the same input; the query half treats the whole blob as untrusted input (bounds-checked, typed errors, total: see the corruption-sweep and property tests).

Scope (0.2, honest)

  • Ranking is full BM25 over true term-frequency postings: the 0.1 unit-tf placeholder is retired, and repetition now matters (saturated by k1 = 1.2) alongside the length norm.
  • Queries are boolean AND; the vectors feature adds approximate-cosine nearest-neighbor over the quantized section (documents without vectors never match; dimension mismatches report emptiness, not errors).
  • Phrase queries, field weights, positional postings, and learned-model codecs do not exist. Text extraction, snippets, and highlight spans are the compiler's job; this crate stores terms, frequencies, lengths, and (optionally) quantized vectors.

Where the truth lives

Concern Truth Path
Format layout (v2 magic, header fields, section order) source of record src/format.rs
Build, decode, BM25 query, quantized vectors source of record src/index.rs
Typed build/decode failures source of record src/error.rs
REQ-Q anchors (client runtime contract) SPEC docs-core/SPEC.md
Layer declaration + dependency floor manifest metadata Cargo.toml

Layer

L0 - leaf. Zero estate-internal dependencies. External floor: fst, thiserror (roaring left with the bitmap postings in 0.2). Builds unmodified for wasm32-unknown-unknown, default features and vectors both (CI enforces the default build; the feature build is checked locally per release).

Tests

cargo test (golden-path, corruption sweep, truncation/flip property tests) under clippy -D warnings with the estate decoder-pattern denies (unwrap/expect/panic/indexing_slicing lifted in test targets only).

Licensed under MIT OR Apache-2.0.