Weavatrix Search Vector

weavatrix-search-vector is a first-party vector-candidate engine for Weavatrix and other Rust applications. It provides deterministic hybrid HNSW plus multi-probe SimHash, exact and scalar-int8 search, versioned persistence, read-only memory mapping, mutable overlays, metadata filters, embedding adapters, KNN graph construction, and bounded shard coordination.

The public API is safe Rust, requires Rust 1.88, and has no runtime dependencies, native libraries, helper processes, or external vector engines. Two private audited modules isolate runtime-detected std::arch kernels and read-only OS mapping on Windows, Linux, and macOS. All other code remains safe Rust.

Installation

cargo add weavatrix-search-vector

Or add the crate directly to Cargo.toml:

[dependencies]
weavatrix-search-vector = "0.2"

Boundary

Vector Search owns:

• vector and query validation;
• deterministic HNSW construction;
• uncertainty-driven SimHash candidate recovery;
• approximate and exact top-K candidate search;
• stable equal-distance ordering by caller-provided u64 key;
• bounded batch workers;
• versioned snapshots with payload checksums;
• owned and zero-copy read-only snapshot loading;
• incremental upsert/delete overlays with deterministic compaction;
• composable metadata predicates with exact filtered fallback;
• scalar-int8 candidates with optional exact f32 reranking;
• model-neutral embedding provider integration;
• directed KNN candidate graph construction;
• transport-neutral bounded shard fan-out and stable merging;
• recall and retained-allocation evidence.

It does not know about the Weavatrix domain graph, semantic thresholds, embedding models, provenance, mutual/union policies, text search, or repository discovery. Semantic consumers remain responsible for exact rescoring and relationship policy.

Example

use weavatrix_search_vector::{IndexConfig, VectorIndex};

let first = [1.0, 0.0, 0.0];
let second = [0.9, 0.1, 0.0];
let third = [0.0, 1.0, 0.0];
let vectors = [(10, first.as_slice()), (20, second.as_slice()), (30, third.as_slice())];

let index = VectorIndex::build(IndexConfig::new(3), &vectors)?;
let hits = index.search(&[1.0, 0.0, 0.0], 2)?;

assert_eq!(hits[0].key, 10);
assert_eq!(hits[1].key, 20);
# Ok::<(), weavatrix_search_vector::SearchError>(())

VectorIndex::search_batch preserves query order and reuses one visited set and heap set per bounded standard-library worker. VectorIndex::search_exact and ExactIndex provide deterministic ground truth for recall tests.

Persistence and live updates

use weavatrix_search_vector::{
    IndexConfig, MappedVectorIndex, Metadata, MutableVectorIndex, VectorIndex,
    VectorRecord,
};

let vector = [1.0, 0.0, 0.0];
let index = VectorIndex::build(IndexConfig::new(3), &[(10, vector.as_slice())])?;
index.save("vectors.wvx")?;

let mapped = MappedVectorIndex::open("vectors.wvx")?;
assert_eq!(mapped.search(&vector, 1)?[0].key, 10);

let mutable = MutableVectorIndex::build(
    IndexConfig::new(3),
    &[VectorRecord::new(10, vector.to_vec())],
)?;
mutable.upsert(20, &[0.9, 0.1, 0.0], Metadata::new())?;
mutable.delete(10);
mutable.compact()?;
# Ok::<(), weavatrix_search_vector::SearchError>(())

VectorIndex::save writes normalized vectors, routing buckets, and all HNSW replicas through a flushed temporary sibling. MappedVectorIndex validates the version, ranges, graph references, finite values, and checksum before serving queries directly from mapped vectors and graph arrays. Structure-only validation is available for trusted immutable artifacts.

MutableVectorIndex uses an exact delta and tombstones over an immutable base. Readers never observe a partial compaction; three conflicting optimistic rebuilds return a typed MutationConflict.

Extended search primitives

• MetadataIndex and MetadataFilter compose equality, existence, numeric ranges, text prefixes, Boolean operations, and exact fallback.
• ScalarQuantizedIndex stores one signed byte per component and can rerank a wider candidate set against VectorIndex.
• EmbeddingProvider and EmbeddingIndex batch external local or hosted encoders without selecting a model or adding a runtime dependency.
• KnnGraph builds deterministic directed candidate edges in compressed-row form; semantic thresholds and graph meaning stay with the consumer.
• SearchShard and DistributedIndex merge heterogeneous local, mapped, quantized, mutable, or remote-backed shards under one worker budget.

Algorithm

Each HNSW replica uses:

• key-sorted normalized vector storage;
• seed/key-derived levels and insertion permutation;
• deterministic parallel bulk-construction waves;
• greedy upper-layer descent;
• bounded best-first layer search;
• diversified outgoing links and retained reverse links;
• doubled layer-zero outgoing degree;
• a compact 14-bit SimHash table with five uncertainty-driven probes;
• runtime-dispatched first-party cosine kernels;
• exact cosine distances for every returned hit.

Independent replicas and the construction work inside each replica share the configured build-worker budget. The built index is immutable, Send, and Sync. Memory is proportional to vector storage and retained graph links, never to the square of the vector count.

Reference benchmark

The reference gate is 10,000 vectors x 384 dimensions, cosine distance, top-8, one warm-up and three release runs:

• build plus all 10,000 approximate queries at most 3 seconds on the reference Windows host;
• recall@8 at least 99.9% against the exact oracle;
• retained allocation estimate below 256 MiB;
• identical API behavior on Windows, Linux, and macOS;
• Rust 1.88, rustfmt, Clippy and rustdoc with warnings denied.

The 2026-07-27 Windows run passes the local performance, recall, and allocation gates:

Machine: Intel Core Ultra 7 255U, 12 cores / 14 logical processors, Windows 11 Enterprise 10.0.26200, rustc 1.97.1 GNU. The corpus is deterministic, synthetic, and clustered. The allocation figure is calculated from retained vector/link capacities; it is not process RSS. See docs/benchmark-2026-07-27.md for commands, raw results, and limitations.

cargo bench --bench vector_search -- run

Environment variables WV_VECTOR_COUNT, WV_VECTOR_DIMENSIONS, WV_VECTOR_TOP_K, WV_VECTOR_RUNS, WV_VECTOR_EXACT_QUERIES, WV_VECTOR_CONNECTIVITY, WV_VECTOR_EXPANSION_BUILD, WV_VECTOR_EXPANSION_QUERY, and WV_VECTOR_REPLICAS control the reproducible corpus and policy. Set WV_VECTOR_EXACT_QUERIES=10000 for the complete reference recall calculation.

These figures establish this crate's disclosed workload, not a general speed claim against another vector engine. Cross-engine benchmarks require identical vectors, query sets, thread budgets, recall, and process-memory measurement.

Competitors

The five-run quality-gated comparison against hnsw_rs 0.3.4 and usearch 2.26.0 found:

Weavatrix is fastest in build, query-only, and build-plus-query on the reference corpus while staying above the 99.9% recall gate. At 50,000 x 384, Weavatrix retained 99.95% sampled recall with a 0.956-second all-query pass; the tested usearch policies were slower and did not retain a 99.9% three-run minimum. See docs/competitive-benchmark-2026-07-27.md for policies, three-run minimum recall, full-oracle evidence, memory caveats, functional gaps, and reproduction commands.

0.2 feature evidence

On the same 10,000 x 384 corpus, the median of three separately launched release runs opened the fully checksummed mapped snapshot in 50.096 ms, executed 1,000 mapped HNSW queries in 42.577 ms, applied 100 in-memory upserts in 0.165 ms, and compacted them in 241.023 ms. The 16.850 MiB snapshot and 3.777 MiB scalar-int8 retained estimate are different representations and should not be compared as equal-recall algorithms.

See docs/benchmark-0.2.0-2026-07-27.md for raw runs, operation definitions, the full-recall core gate, and limitations.

Deliberate boundaries

The crate does not bundle embedding models, tokenizers, GPU runtimes, network transports, distributed consensus, semantic thresholds, provenance policy, or the Weavatrix domain graph. It supplies the vector, storage, mutation, filtering, candidate-graph, and shard-coordination primitives those layers use.