qql-embed

Shared embedding resolution: host-agnostic [Embedder] trait, local wire-compatible BM25 [SparseEmbedder], [resolve_embeddings], and schema [resolve_query_vector_kinds].

Proposition

One embed owner for every backend. No Qdrant I/O, no HTTP client. Runtime (HttpEmbedder), edge (FastEmbedder), and WASM adapters implement the trait. Schema fills USING kinds before embed; unknown kinds fail closed (QQL-VECTOR-KIND) — never silent dense defaults for named vectors.

Embedder trait

pub trait Embedder: Send + Sync {
    async fn embed_dense(&self, text: &str, model: &str) -> Result<Vec<f32>>;
    /// Sparse embedding for query text (default: local wire-compatible BM25, unit weights).
    async fn embed_sparse_query(&self, text: &str, model: &str) -> Result<SparseVector>;
    /// Sparse embedding for document text (default: local wire-compatible BM25, tf saturation).
    async fn embed_sparse_document(&self, text: &str, model: &str) -> Result<SparseVector>;
    /// Batch document-side sparse embedding. Default loops `embed_sparse_document`.
    async fn embed_sparse_document_batch(
        &self,
        texts: &[String],
        model: &str,
    ) -> Result<Vec<SparseVector>>;
    /// Batch query-side sparse embedding. Default loops `embed_sparse_query`.
    async fn embed_sparse_query_batch(
        &self,
        texts: &[String],
        model: &str,
    ) -> Result<Vec<SparseVector>>;
    /// Dense dimension when known (model checking); `None` skips the check.
    fn dimension(&self) -> Option<usize>;
    /// Multivector row dimension when known; `None` skips the check.
    fn multi_dimension(&self) -> Option<usize>;
    /// Whether a dense `MODEL` name is served; single-model hosts reject the rest.
    fn accepts_model(&self, model: &str) -> bool;
    /// Dense embedding — batch API, grouped by model.
    async fn embed_dense_batch(&self, texts: &[String], model: &str) -> Result<Vec<Vec<f32>>>;
    /// Multivector (ColBERT-style). Default rejects with QQL-EMBEDDING-MULTI.
    async fn embed_multi(&self, text: &str, model: &str) -> Result<Vec<Vec<f32>>>;
    async fn embed_multi_batch(&self, texts: &[String], model: &str) -> Result<Vec<Vec<Vec<f32>>>>;
    /// Image / CLIP vision embedding. Default rejects with QQL-EMBEDDING-IMAGE.
    async fn embed_image(&self, source: &str, model: &str) -> Result<Vec<f32>>;
    /// Batch image embedding. Default loops `embed_image`.
    async fn embed_image_batch(&self, sources: &[String], model: &str) -> Result<Vec<Vec<f32>>>;
    /// Cross-encoder pair scoring: (query, documents[i]) → scores. Default rejects with QQL-RERANK-CROSS.
    async fn rerank_pairs(&self, query: &str, documents: &[String], model: &str) -> Result<Vec<f32>>;
    /// Single-pass joint embeddings (dense + sparse + multi in one pass for BGE-M3).
    async fn embed_joint(&self, text: &str, model: &str) -> Result<JointEmbeddingOutput>;
    async fn embed_joint_batch(&self, texts: &[String], model: &str) -> Result<Vec<JointEmbeddingOutput>>;
}

Dense embedding is batched by model when the target is single-vector dense. Sparse is role-split: queries embed with unit term weights (embed_sparse_query), documents with BM25 term-frequency saturation (embed_sparse_document) — both matching Qdrant's qdrant/bm25 defaults (tunable via Embedder::bm25_params, see SparseEmbedder). Multivector defaults reject until the host opts in (embed_multi), as does image embedding (embed_image).

FastEmbed-style host mapping

CLIP is dual-encoder dense, never multivector. Multivector is late-interaction bags only.

Language:

• QUERY IMAGE 'path-or-url' [MODEL '…'] → embed_image → Dense
• UPSERT … USING IMAGE MODEL '…' ON FIELD image INTO image

Schema topology before embed

Parse leaves USING name as kind: null. Execution prep must fill kinds:

use qql_embed::{resolve_query_vector_kinds, resolve_embeddings, TopologyNames};

// From collection schema (runtime / WASM):
let topology = TopologyNames {
    dense: vec!["dense".into(), "colbert".into()],
    sparse: vec!["sparse".into()],
    multivector: vec!["colbert".into()], // dense names with multivector_config
};
resolve_query_vector_kinds("docs", &mut query, &topology)?;
resolve_embeddings(&mut stmt, &embedder).await?;

resolve_embeddings — AST rewriter

use qql_embed::{resolve_embeddings, DENSE_VECTOR_NAME, SPARSE_VECTOR_NAME};

let mut stmt = Parser::parse("UPSERT INTO docs VALUES {id: 1, text: 'hello'}").unwrap();
resolve_embeddings(&mut stmt, &embedder).await?;
// stmt now has text → dense vector for point[0]

Resolution happens in these cases:

Vector roles and default names

Query targets carry an optional role (DENSE or SPARSE) plus a multi flag. Arbitrary names such as semantic_v2 and lexical_v2 are supported; embedding behavior never depends on a target literally being named dense or sparse.

• DENSE_VECTOR_NAME: "dense" (constant)
• SPARSE_VECTOR_NAME: "sparse" (constant)

These constants are used only when materializing a new default topology.

SparseEmbedder — local wire-compatible BM25

Client-side BM25 that is wire-compatible with Qdrant's qdrant/bm25 model: murmur3-32 token IDs (same hash the server uses), word tokenizer (split on non-alphanumeric), Unicode lowercasing, English stopword removal, and English snowball stemming — the server's documented defaults. Queries embed with unit term weights; documents with BM25 tf saturation (k1=1.2, b=0.75, avg_len=256). IDF is applied server-side via the sparse vector modifier: idf. No network, no model downloads. A synchronous helper backing the default Embedder::embed_sparse_query / embed_sparse_document implementations.

use qql_embed::SparseEmbedder;

let q = SparseEmbedder::embed_query("quantum computing");   // unit weights
let d = SparseEmbedder::embed_document("quantum computing"); // tf saturation
// q/d.indices: [u32; N], q/d.values: [f32; N]

Tuning k1, b, avg_len

qql_embed::Bm25Params makes the BM25 hyperparameters configurable:

use qql_embed::{Bm25Params};

// Defaults: 1.2 / 0.75 / 256 (Qdrant qdrant/bm25).
let params = Bm25Params::new(1.2, 0.75, 8.0)?;
let d = qql_embed::sparse::embed_document_with_params("short doc", &params);
// or, via the helper:
let d = qql_embed::SparseEmbedder::embed_document_with("short doc", &params);

This is a client-side, write-path-only setting:

• It shapes documents only — k1 controls tf saturation, b controls length normalization, and avg_len is the expected average document length in tokens. A wrong avg_len silently misjudges every document (rare terms and long docs get the wrong normalization), so estimate it from the corpus being written.
• It does not change query-side weights (always unit) and does not change server-side qdrant/bm25 inference — the server keeps its own defaults unless configured separately.
• It is not a collection/wire setting: existing vectors keep the weights they were written with. Re-ingest to apply a change.
• Invalid values fail closed with QQL-VALIDATION-CONFIG: k1 >= 0 (0 = binary weighting, like Qdrant's validator), b in [0, 1], avg_len > 0, all finite (NaN/±Inf rejected).

Unset configuration is byte-identical to the previous hardcoded behavior (Bm25Params::default() == 1.2 / 0.75 / 256).

Host surfaces (numeric knobs everywhere; text knobs wherever the host exposes them — see each row):

They only apply when the built-in local BM25 encoder is used. When an ONNX sparse model (SPLADE / BGE-M3) or a remote sparse endpoint is configured, sparse vectors come from that model and these parameters are inert.

Vectors produced here can be mixed with server-side qdrant/bm25 inference on the same collection (a golden test pins the exact server output from the Qdrant docs).

Text processing: languages, tokenizers, folding

qql_embed::Bm25TextConfig mirrors Qdrant's Bm25Config option surface with the same defaults (word tokenizer, English, lowercase on, folding off, language stopwords/stemmer, no length limits):

use qql_embed::Bm25TextConfig;

// Spanish pipeline: Spanish stopwords + Snowball stemmer, like Qdrant's
// `options: {"language": "spanish"}`.
let config = Bm25TextConfig::resolve(
    None, None, None,
    Some("spanish"),  // language (name or alias: "es", "zh", …)
    None,             // tokenizer: word (default) | whitespace | prefix
    None,             // lowercase (default true)
    None,             // ascii_folding (default false)
    None,             // stopwords: None = language default, Some(list) replaces
    None,             // stemmer: None = default, Some("none") disables
    None, None,       // min/max token length (chars)
    None,             // extra stopwords languages merged with stopwords
)?;
let pipeline = config.pipeline();
let d = pipeline.embed_document("La Máquina del Tiempo")?;

Thirty languages carry Qdrant's stopword lists; seventeen add a Snowball stemmer (Armenian and Tamil are explicit-stemmer-only, like Qdrant — no language variant, no stopword lists); the rest pass tokens through, exactly like Qdrant, whose stemmer defaults resolve per language the same way). ASCII folding uses Qdrant's Lucene-derived mapping. multilingual parses but fails closed at embed time — script-aware segmentation (charabia/vaporetto) is intentionally not a dependency of this lean core. The qql-edge engine path forwards the same knobs to Qdrant's real pipeline instead, and a cross-implementation test asserts both produce identical vectors.

Estimating avg_len from real data

The 256 default assumes document-length text; titles and tags need far smaller values. estimate_avg_len measures the mean post-pipeline token count over real field texts — the same doc_len the formula consumes — and Executor::estimate_bm25_avg_len(collection, field, sample) samples it straight from a collection via SCROLL:

let estimate = executor
    .estimate_bm25_avg_len("books", "title", 1000)
    .await?
    .expect("collection has title texts");
// feed estimate.mean back into bm25_avg_len, then re-ingest to apply

None means the sample held no usable texts (keep the default rather than storing a meaningless zero).

Like the server defaults, the default pipeline is English-only (snowball English stemmer + English stopwords). Other languages select Bm25TextConfig above; non-English corpora that need server-side inference should still use qdrant/bm25 with explicit language / stemmer / stopwords options instead.

Known WASM limitation

WASM Client prepares statements like the native executor: fetch collection topology, resolve kinds, then embed (when an embedder is configured). Hosts that need ColBERT must implement embed_multi on their embedder adapter.

Features

• std (default): std::error::Error impl
• All types are Send + Sync on non-wasm targets; ?Send on wasm32

Verification

cargo test -p qql-embed -- --test-threads=4

Tests cover:

• Dense / sparse / multi query resolution
• Fail-closed USING name without kind
• Schema multivector → MultiDense
• RERANK + AS MULTI
• Hybrid, UPSERT, EMBED directives
• Sparse BM25 tokenization