iqdb 0.3.0

Embedded vector database for Rust. Lock-free, allocation-free hot path; cross-platform similarity search.
Documentation

Design Goals

iQDB is engineered against the Rust Efficiency & Performance Standards (REPS). Every architectural decision is graded against the same hard constraints:

  • Embedded by default — no daemon, no socket, no separate process. The library opens a path and gets out of the way.
  • Sub-millisecond queries — exact and approximate search paths are budgeted in microseconds, not milliseconds, with the hot path measured under Criterion every release.
  • Lock-free hot paths — atomic reads on the query path; writers do not contend with concurrent readers.
  • Allocation-free steady state — buffers are pooled and reused. Per-query allocation is a design defect.
  • Cache-aware layout — vectors land contiguous in memory and on disk; index nodes are padded to the cache line.
  • Pluggable indices — flat, IVF, and HNSW share a common trait surface so callers can swap strategies without touching their query code.
  • Pluggable storage — durable file-backed storage, transient in-memory storage, and the org-default storage-io substrate are all addressable behind the same handle.
  • Crash-safe writes — the durable path uses write-ahead logging and atomic file replacement. A pulled power cord must never corrupt the index.
  • Tier-1 cross-platform — Linux (x86_64, aarch64), macOS (x86_64, Apple Silicon), Windows (x86_64) all compile and pass the full test suite on every commit.

Status & Roadmap

iQDB ships milestone-by-milestone. Each tag below corresponds to a published release; everything above the current line is shipped, everything below is planned.

Milestone Status Surface delivered
v0.1.0 — scaffolding shipped Crate scaffolding, lifecycle handle (open / open_in_memory / flush / close), Error type, integration test, criterion harness, CI matrix on all three Tier-1 platforms.
v0.2.0 — vector primitives shipped Vector (validated f32 embeddings), DistanceMetric (L2 / Cosine / Dot), Payload & PayloadValue (typed metadata), RecordId, Record. In-memory store with thread-safe upsert / get / delete / len / is_empty. Optional serde feature.
v0.3.0 — search current Iqdb::search / search_with / search_batch / search_batch_with over the flat index. SearchResult { id, score, payload }. Monomorphic predicate filters. NaN-aware ranking with id tie-break. Property-based ranking tests via proptest. docs/API.md published.
v0.4.0 — durable storage planned File-backed storage substrate. Write-ahead log, atomic-replace snapshots, crash recovery. Iqdb::open(path) becomes load-bearing.
v0.5.0 — approximate indices planned IVF and HNSW indices behind the same trait the flat index implements. Build-time index selection via the builder.
v0.6.0 — async surface planned async-feature-gated mirror of the public API. Driven by Tokio. Cancellation-safe.
v0.7.0 — collections planned Named collections / namespaces with per-collection metric and dimensionality. Collection-scoped iteration.
v1.0.0 — API freeze planned Frozen public API. SemVer guarantees. Full benchmark suite. Migration guide from 0.x.

The per-release detail — what was added, what changed, and what was verified — lives in the CHANGELOG and the per-version notes under docs/release/.

Installation

Add to your Cargo.toml:

[dependencies]
iqdb = "0.3"

Enable the optional serde feature to derive Serialize / Deserialize on every public data type:

[dependencies]
iqdb = { version = "0.3", features = ["serde"] }

iQDB compiles on stable Rust 1.75 and newer. The default build pulls zero runtime dependencies; the serde feature pulls only serde itself.

Quick Start

The 0.3.0 surface exposes typed vector primitives, the in-memory store, and exact top-k similarity search. The durable file-backed substrate lands in v0.4.0; approximate indices (IVF, HNSW) in v0.5.0 will sit alongside the flat kernel rather than replacing it — exact search remains the correctness baseline.

use iqdb::{DistanceMetric, Iqdb, Payload, Record, RecordId, Result, Vector};

fn main() -> Result<()> {
    let db = Iqdb::open_in_memory();

    let mut meta = Payload::new();
    meta.insert("kind", "doc");

    db.upsert(Record::with_payload(
        RecordId::new(1),
        Vector::new(vec![1.0, 0.0, 0.0])?,
        meta,
    ))?;
    db.upsert(Record::new(
        RecordId::new(2),
        Vector::new(vec![0.99, 0.10, 0.0])?,
    ))?;

    // Top-k similarity search. Results are sorted ascending under the
    // smaller-is-closer rule; ties break on id for determinism.
    let probe = Vector::new(vec![1.0, 0.0, 0.0])?;
    let hits = db.search(&probe, 5, DistanceMetric::Cosine)?;
    assert_eq!(hits.first().map(|h| h.id), Some(RecordId::new(1)));

    db.close()
}

Filtered and batch search

Filters are generic — the predicate monomorphises into the search loop, so there is no per-record dynamic dispatch. The smaller-is-closer convention holds across all three metrics; Dot returns −(a · b) so a single ordering rule covers L2, Cosine, and Dot.

use iqdb::{DistanceMetric, Iqdb, Payload, PayloadValue, Record, RecordId, Result, Vector};

fn main() -> Result<()> {
    let db = Iqdb::open_in_memory();

    let mut doc = Payload::new();
    doc.insert("kind", "doc");
    db.upsert(Record::with_payload(
        RecordId::new(1),
        Vector::new(vec![1.0, 0.0])?,
        doc,
    ))?;

    let mut image = Payload::new();
    image.insert("kind", "image");
    db.upsert(Record::with_payload(
        RecordId::new(2),
        Vector::new(vec![0.99, 0.10])?,
        image,
    ))?;

    let probe = Vector::new(vec![1.0, 0.0])?;

    // Filter the candidate set before heap admission.
    let docs_only = db.search_with(&probe, 5, DistanceMetric::Cosine, |rec| {
        rec.payload()
            .and_then(|p| p.get("kind"))
            .and_then(PayloadValue::as_text)
            == Some("doc")
    })?;
    assert_eq!(docs_only.len(), 1);
    assert_eq!(docs_only[0].id, RecordId::new(1));

    // Batch — one top-k result list per query, preserves input order.
    let probes = vec![
        Vector::new(vec![1.0, 0.0])?,
        Vector::new(vec![0.0, 1.0])?,
    ];
    let batches = db.search_batch(&probes, 1, DistanceMetric::Cosine)?;
    assert_eq!(batches.len(), 2);

    Ok(())
}

Handling the staged surface

Because Iqdb::open(path) and Iqdb::flush still belong to the durable-storage milestone, downstream callers can wire them in advance and gate behaviour on the error variant — the Err arm disappears when v0.4.0 ships:

use iqdb::{Error, Iqdb};

fn flush_if_supported(db: &Iqdb) -> Result<(), Error> {
    match db.flush() {
        Ok(()) => Ok(()),
        Err(Error::NotImplemented) => {
            // The active milestone does not yet support flushing.
            // In v0.4.0 (durable storage) this branch disappears.
            Ok(())
        }
        Err(err) => Err(err),
    }
}

let db = Iqdb::open_in_memory();
flush_if_supported(&db).unwrap();

API Overview

The full API reference lives at docs/API.md; the rustdoc-generated docs at docs.rs/iqdb carry the same information in browseable form. The currently-stable items are:

  • Iqdb — the top-level database handle.
    • Iqdb::open(path) — open or create a file-backed database (planned for v0.4.0 — currently returns Error::NotImplemented).
    • Iqdb::open_in_memory() — open an ephemeral instance backed entirely by RAM.
    • Iqdb::upsert(record) — insert or replace a record. Idempotent.
    • Iqdb::get(id) — look up by id. Returns Ok(None) when absent.
    • Iqdb::delete(id) — remove by id. Returns whether the id was present.
    • Iqdb::len() / Iqdb::is_empty() — store cardinality.
    • Iqdb::search(query, k, metric) — exact top-k similarity search, no filter.
    • Iqdb::search_with(query, k, metric, filter) — top-k with a per-record predicate. The filter monomorphises into the scan loop; no per-record dynamic dispatch.
    • Iqdb::search_batch(queries, k, metric) / search_batch_with(...) — sequential batch variants. Preserves input order.
    • Iqdb::flush() — flush pending writes to durable storage (planned for v0.4.0).
    • Iqdb::close(self) — close the handle and release any held resources.
  • Vector — owned, contiguous, validated f32 embedding. Vector::new(Vec<f32>) / Vector::from_slice(&[f32]) validate at the system boundary (no empty vectors, no NaN, no infinity); as_slice / dim / norm / norm_squared are non-allocating.
  • DistanceMetric — L2, Cosine, Dot. metric.distance(a, b) returns a Result<f32> under the smaller-is-closer convention; dimensional homogeneity is enforced.
  • Payload — typed BTreeMap<String, PayloadValue> for metadata. Deterministic iteration order makes payloads stable across serde round-trips and test assertions.
  • PayloadValue — Null / Bool / Int / Float / Text / Bytes / Array / nested Object. From<T> conversions cover the primitives.
  • RecordId — transparent newtype around u64. Cheap to copy, hash, and compare.
  • Record — (id, vector, optional payload) aggregate. Record::new / Record::with_payload are the two constructors; into_parts decomposes without a clone.
  • SearchResult — { id, score, payload } returned by the search methods. Sorted ascending by score; ties broken on id; NaN scores sort to the tail.
  • Error — the unified error type. #[non_exhaustive].
  • Result<T> — alias for core::result::Result<T, Error>.

Error variants

Variant Meaning Recovery
Error::Io(std::io::Error) A lower-level I/O failure occurred — disk full, permission denied, missing path, etc. Inspect the wrapped ErrorKind. Retry transient errors; surface permanent ones.
Error::InvalidConfig(&'static str) Configuration supplied at open time was invalid (e.g., zero-length path, unsupported metric). Programmer error — fix the construction site.
Error::InvalidVector { reason } A vector failed boundary validation — empty, or contains NaN / ±∞. Sanitise the input at the producer side; Vector::new rejects bad data before it enters the store.
Error::DimensionMismatch { left, right } A distance-metric or store operation combined two vectors of different dimensionality. Enforce a homogeneous schema at the producer side or surface a typed error to the caller.
Error::NotImplemented The requested operation belongs to a later milestone and has no engine behind it yet. Either upgrade to a release where the verb is implemented, or branch on the variant and fall back.

The enum is #[non_exhaustive]. New variants will appear in minor releases as new failure modes emerge. Exhaustive match arms are a forward-compatibility hazard — always include _.

Examples

Self-contained examples live in examples/ and are declared in Cargo.toml. Run them with cargo run --example <name>.

  • Lifecycle (basic) — open an in-memory instance, upsert a vector, read it back, close.

  • In-memory store walk-through (in_memory_store) — populate the store with three records (vectors + typed payloads), compare distances between them under L2 and cosine, then delete one.

  • Top-k search (search) — unfiltered cosine top-k, payload-filtered search, and batch search across three probes in one file.

Approximate-index examples land alongside their milestone (v0.5.0).

Benchmarks

A criterion harness is wired in benches/vector_ops.rs. v0.3.0 ships four groups:

  • vector_new — construction-time validation cost across small / medium / large dimensionalities (32 / 128 / 1024).
  • distance — single-shot distance computation under each of the three DistanceMetric variants at dim 128.
  • store — upsert and get throughput against a populated in-memory store at 1 000 records, dim 128.
  • search — flat top-k search at 1 000 and 10 000 records, dim 128. Three variants: unfiltered, payload-filtered (~50% pruning), and batch-of-4.

Run with:

cargo bench --bench vector_ops

Criterion writes reports to target/criterion/. Approximate-index benches land with v0.5.0 and the durable-write benches land with v0.4.0; CI will gate merges on a regression threshold once the benches are stable.

Testing

Every public path has happy / error / edge-case coverage:

  • Unit tests live in #[cfg(test)] mod tests blocks inside each source file.
  • Integration tests live in tests/:
    • tests/in_memory.rs — CRUD surface plus serde round-trips behind a feature gate.
    • tests/search.rs — the four search entry points: top-k ordering, filter pruning, batch order, dimension-mismatch handling, payload preservation, concurrent readers.
    • tests/properties.rs — proptest-driven property tests for distance-metric algebra (symmetry, identity, range bounds) and search-ranking invariants (length bound, ascending order, perfect-match presence, no-filter parity).
    • tests/smoke.rs — minimal lifecycle smoke check.
  • Doc tests run as part of cargo test and validate every # Examples block in the rustdoc.
# Full test sweep (unit + integration + doc tests)
cargo test
cargo test --all-features

# Documentation build with no warnings (matches CI gating)
RUSTDOCFLAGS="-D warnings" cargo doc --no-deps
RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features

# Lint at the strict profile CI enforces
cargo clippy --all-targets -- -D warnings
cargo clippy --all-targets --all-features -- -D warnings

# Formatting check (no diffs)
cargo fmt --all -- --check

Cross-Platform Support

Tier 1 targets — every commit is built and tested on:

  • Linux (x86_64-unknown-linux-gnu, aarch64-unknown-linux-gnu)
  • macOS (x86_64-apple-darwin, aarch64-apple-darwin)
  • Windows (x86_64-pc-windows-msvc)

Platform-specific paths — when the durable storage substrate lands, the platform-conditional code paths become observable:

  • Linux: io_uring submission for batch writes; fsync for journal durability.
  • macOS: F_FULLFSYNC for journal durability; mmap for read-mostly indices.
  • Windows: FlushFileBuffers for journal durability; MoveFileExW with MOVEFILE_REPLACE_EXISTING for atomic snapshot replacement.

No platform is silently degraded — fallbacks are explicit and documented inline at each #[cfg] boundary, per the REPS Platform-Specific Code rule.

Configuration

Feature flags

Feature flags are strictly additive (per REPS) — enabling any combination never removes or weakens existing functionality.

Feature Default Available Description
serde off shipping Derives Serialize / Deserialize on every public data type.
async off planned v0.6.0 Tokio-driven async mirror of the public API.
mmap off planned v0.4.0 Memory-mapped read path for hot indices.
io-uring off planned v0.4.0 Linux-only io_uring submission for batch writes.
full off planned post-1.0 All stable features in one switch. 
iqdb = { version = "0.2", features = ["serde"] }

Runtime configuration

Iqdb::open_in_memory() takes no parameters by design. A builder (IqdbBuilder) is introduced in v0.4.0 once the file-backed path lands and has more than two open-time knobs. Until then, the constructor surface is the entire configuration surface.

Architecture

The crate is split along strict module boundaries — each module owns one concern and exposes a single trait or type as its contract:

  • src/lib.rs — crate root, lint profile, public re-exports.
  • src/db.rs — the Iqdb handle and the public CRUD verbs.
  • src/vector.rs — the Vector primitive and the DistanceMetric dispatch.
  • src/payload.rs — the Payload / PayloadValue typed metadata layer.
  • src/record.rs — the RecordId / Record aggregate.
  • src/store.rs — the crate-internal MemoryStore (the read/write engine behind open_in_memory).
  • src/error.rs — the Error enum and Result alias.

As milestones land, the tree grows along the same bounded-responsibility pattern (index/, journal/, query/, async_impl.rs). The boundary between layers is always a trait or a concrete type with a typed surface — concrete backend implementations are crate-internal and gated behind pub(crate).

Compile-time guarantees

The crate root enables the strict REPS lint profile in src/lib.rs:

#![deny(warnings)]
#![deny(missing_docs)]
#![deny(unsafe_op_in_unsafe_fn)]
#![deny(unused_must_use)]
#![deny(unused_results)]
#![deny(clippy::unwrap_used)]
#![deny(clippy::expect_used)]
#![deny(clippy::todo)]
#![deny(clippy::unimplemented)]
#![deny(clippy::print_stdout)]
#![deny(clippy::print_stderr)]
#![deny(clippy::dbg_macro)]
#![deny(clippy::unreachable)]
#![deny(clippy::undocumented_unsafe_blocks)]

Test modules locally relax the unwrap_used / expect_used lints — the strict profile is for production library code, not assertion scaffolding inside #[cfg(test)] blocks.

Contributing

Pull requests are welcome. Before opening one, please make sure the full CI gate passes locally:

cargo fmt --all -- --check
cargo clippy --all-targets -- -D warnings
cargo clippy --all-targets --all-features -- -D warnings
cargo test
cargo test --all-features
RUSTDOCFLAGS="-D warnings" cargo doc --no-deps
RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features
cargo deny check
cargo audit

Every contribution is expected to honour the standards in REPS.md — performance, security, error handling, testing, documentation, and dependency hygiene are all enforced as merge gates, not afterthoughts. Commit messages are imperative, lowercase, and scoped to a single logical change.

Links