Module sparse 

Client-side BM25 sparse embeddings, wire-compatible with Qdrant’s qdrant/bm25 model defaults.

Token IDs are murmur3-32 (seed 0, |i32| made positive) — identical to the Qdrant server, Qdrant Edge, and FastEmbed’s Qdrant/bm25. The text pipeline mirrors the server defaults: word tokenizer (split on non-alphanumeric), Unicode lowercasing, English stopword removal, and English snowball stemming. Queries embed with unit term weights; documents with BM25 term-frequency saturation (k1=1.2, b=0.75, avg_len=256). IDF is applied server-side via the sparse vector modifier: idf.

Because token IDs and formulas match the server, vectors produced here can be mixed with server-side qdrant/bm25 inference on the same collection.

Structs

SparseVector

Sparse embedding (indices + values). Transport-neutral — not a protobuf type.

Constants

DEFAULT_AVGDL

BM25 expected average document length in tokens, matching Qdrant’s qdrant/bm25 default.

DEFAULT_B

BM25 document-length normalization, matching Qdrant’s qdrant/bm25 default.

DEFAULT_K1

BM25 term-frequency saturation, matching Qdrant’s qdrant/bm25 default.

Functions

embed_document

Embed document text with BM25 term-frequency saturation using Qdrant’s default parameters (k1=1.2, b=0.75, avg_len=256).

embed_document_with

Embed document text with explicit BM25 parameters.

embed_query

Embed query text: unique token IDs (sorted) with unit weights — identical to Qdrant’s qdrant/bm25 query embedding.

for_each_token

Tokenize and iterate over stemmed tokens without intermediate heap allocations.

for_each_token_id

Tokenize and iterate directly over u32 token IDs without intermediate allocations.

token_id

Token → u32 ID. Wire-compatible with Qdrant’s BM25 sparse vectors: murmur3 32-bit (seed 0), then |i32| to make it positive.

tokenize

Server-default text pipeline: word tokenizer (split on non-alphanumeric), Unicode lowercase, English stopword removal, English snowball stemming.