Module sparse 

Local wire-compatible BM25 sparse vectors (no network). Client-side BM25 sparse embeddings, wire-compatible with Qdrant’s qdrant/bm25 model defaults.

Token IDs are murmur3-32 (seed 0, |i32| made positive) — identical to the Qdrant server, Qdrant Edge, and FastEmbed’s Qdrant/bm25. The text pipeline mirrors the server defaults: word tokenizer (split on non-alphanumeric), Unicode lowercasing, English stopword removal, and English snowball stemming. Queries embed with unit term weights; documents with BM25 term-frequency saturation (k1=1.2, b=0.75, avg_len=256, or explicit Bm25Params). IDF is applied server-side via the sparse vector modifier: idf.

Because token IDs and formulas match the server, vectors produced here can be mixed with server-side qdrant/bm25 inference on the same collection.

Tuning BM25 (k1, b, avg_len)

Bm25Params is a client-side, write-path-only setting: it shapes how documents are encoded (tf saturation via k1, length normalization via b, and the expected average document length via avg_len). It is not a collection or wire setting, does not affect query-side weights (always unit weights), and does not change server-side qdrant/bm25 inference. A wrong avg_len silently misjudges every document, so tune it to the corpus being written; documents embedded before the change keep their vectors — re-ingest to apply.

FastEmbed Query Weighting Parity Note

FastEmbed’s Python Qdrant/bm25 emits a uniform scaling factor (~1.665) on query term weights, whereas Qdrant’s server-side inference and QQL use unit weights (1.0). Because this factor is uniform across all terms in a query, ranking order is mathematically identical, but raw score magnitudes will scale by ~1.665x.

Structs

Bm25Params

Validated BM25 hyperparameters for document-side local encoding.

SparseVector

Sparse embedding (indices + values). Transport-neutral — not a protobuf type.

Constants

DEFAULT_AVGDL

BM25 expected average document length in tokens, matching Qdrant’s qdrant/bm25 default.

DEFAULT_B

BM25 document-length normalization, matching Qdrant’s qdrant/bm25 default.

DEFAULT_K1

BM25 term-frequency saturation, matching Qdrant’s qdrant/bm25 default.

Functions

embed_document

Embed document text with BM25 term-frequency saturation using Qdrant’s default parameters (k1=1.2, b=0.75, avg_len=256).

embed_document_with

Embed document text with explicit BM25 parameters.

embed_document_with_params

Embed document text with validated Bm25Params.

embed_query

Embed query text: unique token IDs (sorted) with unit weights — identical to Qdrant’s qdrant/bm25 query embedding.

for_each_token

Tokenize and iterate over processed tokens (default English pipeline: word tokenizer, lowercase, English stopwords, English stemming).

for_each_token_id

Tokenize and iterate directly over u32 token IDs without intermediate allocations.

token_id

Token → u32 ID. Wire-compatible with Qdrant’s BM25 sparse vectors: murmur3 32-bit (seed 0), then |i32| to make it positive.

tokenize

Server-default text pipeline: word tokenizer (split on non-alphanumeric), Unicode lowercase, English stopword removal, English snowball stemming.