iqdb 0.7.0

Design Goals

iQDB is engineered against the Rust Efficiency & Performance Standards (REPS). Every architectural decision is graded against the same hard constraints:

• Embedded by default — no daemon, no socket, no separate process. The library opens a path and gets out of the way.
• Sub-millisecond queries — exact and approximate search paths are budgeted in microseconds, not milliseconds, with the hot path measured under Criterion every release.
• Enum-dispatched hot paths — the index seam dispatches through a closed match, never dyn, so the query loop sees a concrete index with no virtual indirection.
• Allocation-aware steady state — vector payloads are shared as Arc<[f32]> between the authoritative store and the derived index, so a vector that lives in both costs one allocation, not two.
• Pluggable indices — flat, IVF, and HNSW share the iqdb-index trait surface so callers swap strategies through IqdbConfig without touching their query code. Flat is the exact recall ground truth the approximate indices are measured against.
• Crash-safe writes — the durable path uses write-ahead logging and atomic snapshot replacement. A pulled power cord must never corrupt the database.
• Tier-1 cross-platform — Linux (x86_64, aarch64), macOS (x86_64, Apple Silicon), Windows (x86_64) all compile and pass the full test suite on every commit.

Status & Roadmap

iQDB ships milestone-by-milestone. Each tag below corresponds to a published release; everything above the current line is shipped, everything below is planned.

The per-release detail — what was added, what changed, and what was verified — lives in the CHANGELOG and the per-version notes under docs/release/.

Installation

Add to your Cargo.toml:

[dependencies]
iqdb = "0.5"

Optional features (all additive):

[dependencies]
iqdb = { version = "0.5", features = ["serde", "parallel", "zstd"] }

iQDB compiles on stable Rust 1.87 and newer. As of 0.5.0 it composes the published iqdb family crates (iqdb-types, iqdb-index, iqdb-flat, iqdb-hnsw, iqdb-ivf, iqdb-build, iqdb-persist, iqdb-cache, and their transitive dependencies); it is no longer a zero-dependency build.

Quick Start

A database fixes its dimensionality and distance metric at open time, then exposes a small surface: upsert / get / delete for records, search / search_with for queries.

use iqdb::{DistanceMetric, Iqdb, Result, Vector, VectorId};

fn main() -> Result<()> {
    // A 3-dimensional, in-memory database compared under cosine distance.
    let db = Iqdb::open_in_memory(3, DistanceMetric::Cosine)?;

    db.upsert(VectorId::from(1u64), Vector::new(vec![1.0, 0.0, 0.0])?, None)?;
    db.upsert(VectorId::from(2u64), Vector::new(vec![0.99, 0.10, 0.0])?, None)?;

    // Top-k similarity search. Results are sorted nearest-first under the
    // smaller-is-closer rule; ties break on insertion order for determinism.
    let hits = db.search(&Vector::new(vec![1.0, 0.0, 0.0])?, 5)?;
    assert_eq!(hits[0].id, VectorId::from(1u64));

    db.close()
}

Filtered and batch search

Filters are declarative Filter expressions evaluated against each record's Metadata. On the exact flat index the filter is applied before scoring, so the result is exact.

use iqdb::{DistanceMetric, Filter, Iqdb, Metadata, Result, Value, Vector, VectorId};

fn main() -> Result<()> {
    let db = Iqdb::open_in_memory(2, DistanceMetric::Cosine)?;

    let doc: Metadata = [("kind".to_string(), Value::String("doc".into()))]
        .into_iter().collect();
    let img: Metadata = [("kind".to_string(), Value::String("image".into()))]
        .into_iter().collect();
    db.upsert(VectorId::from(1u64), Vector::new(vec![1.0, 0.0])?, Some(doc))?;
    db.upsert(VectorId::from(2u64), Vector::new(vec![0.99, 0.10])?, Some(img))?;

    // Only documents, ranked by cosine distance.
    let filter = Filter::eq("kind", Value::String("doc".into()));
    let hits = db.search_with(&Vector::new(vec![1.0, 0.0])?, 5, filter)?;
    assert_eq!(hits.len(), 1);
    assert_eq!(hits[0].id, VectorId::from(1u64));

    // Batch — one top-k result list per query, preserving input order.
    let queries = vec![Vector::new(vec![1.0, 0.0])?, Vector::new(vec![0.0, 1.0])?];
    let batches = db.search_batch(&queries, 1)?;
    assert_eq!(batches.len(), 2);

    Ok(())
}

Choosing an index

Tier 1 (open_in_memory / open) defaults to the exact flat index. Tier 2 (open_in_memory_with / open_with) takes an IqdbConfig that selects an approximate index and tunes it, and can attach a result cache.

use iqdb::{CacheConfig, DistanceMetric, HnswConfig, IndexKind, Iqdb, IqdbConfig, Result};

fn main() -> Result<()> {
    let cfg = IqdbConfig::new(128, DistanceMetric::Cosine)
        .index(IndexKind::Hnsw(HnswConfig::default().with_ef_search(96)))
        .cache(CacheConfig::new().capacity(10_000));
    let db = Iqdb::open_in_memory_with(cfg)?;
    assert!(db.is_empty());
    Ok(())
}

On the approximate indices the metadata filter is applied after graph / cluster traversal, so a highly selective filter can return fewer than k hits — widen the search (HNSW filter_widen, IVF n_probes) when that matters. IVF is trained lazily from the stored vectors on the first search; after many writes, Iqdb::optimize retrains its centroids.

Durable, file-backed storage

Iqdb::open(path, dim, metric) opens or creates a durable database. The path is the snapshot file; a write-ahead log lives beside it. Acknowledged writes survive a crash; reopening replays the log onto the snapshot.

use iqdb::{DistanceMetric, Iqdb, Result, Vector, VectorId};

fn main() -> Result<()> {
    let db = Iqdb::open("./data/vectors.iqdb", 3, DistanceMetric::Cosine)?;
    db.upsert(VectorId::from(1u64), Vector::new(vec![0.1, 0.2, 0.3])?, None)?;
    db.flush()?; // compact: fold the WAL into a fresh snapshot
    db.close()
}

A reopen whose requested dim / metric disagrees with the stored database fails with Error::Config. The stored index kind is part of the database identity and is restored from the snapshot regardless of the kind requested on reopen.

By default every acknowledged write is fsynced and the snapshot is uncompressed. Trade durability for throughput, or shrink the snapshot, through IqdbConfig:

use iqdb::{Compression, DistanceMetric, FsyncPolicy, Iqdb, IqdbConfig};
use std::time::Duration;

# fn run() -> iqdb::Result<()> {
let cfg = IqdbConfig::new(128, DistanceMetric::Cosine)
    .fsync(FsyncPolicy::Periodic(Duration::from_millis(50))) // bound the un-synced window
    .compression(Compression::Zstd { level: 3 });            // requires the `zstd` feature
let db = Iqdb::open_with("./data/vectors.iqdb", cfg)?;
# let _ = db;
# Ok(())
# }

Async (the async feature)

The family is synchronous by design, so the async surface is a thin Tokio adapter: AsyncIqdb holds an Arc<Iqdb> and runs each blocking call on Tokio's blocking pool via spawn_blocking, so awaiting a search or a write never stalls the executor. It is Clone + Send + Sync. Enable the async feature and bring your own runtime.

use iqdb::{AsyncIqdb, DistanceMetric, Result, Vector, VectorId};

#[tokio::main]
async fn main() -> Result<()> {
    let db = AsyncIqdb::open_in_memory(3, DistanceMetric::Cosine).await?;
    db.upsert(VectorId::from(1u64), Vector::new(vec![1.0, 0.0, 0.0])?, None).await?;

    let hits = db.search(Vector::new(vec![1.0, 0.0, 0.0])?, 1).await?;
    assert_eq!(hits[0].id, VectorId::from(1u64));
    db.close().await
}

API Overview

The full API reference lives at docs/API.md; the rustdoc at docs.rs/iqdb carries the same information in browseable form. The public surface:

• Iqdb — the database handle.
  • Iqdb::open_in_memory(dim, metric) — an ephemeral, exact-flat database.
  • Iqdb::open_in_memory_with(config) — an in-memory database from a full IqdbConfig.
  • Iqdb::open(path, dim, metric) / Iqdb::open_with(path, config) — a durable, file-backed database.
  • Iqdb::upsert(id, vector, metadata) — insert or replace. Rejects a wrong-dimension vector at the boundary.
  • Iqdb::get(id) — look up the stored vector and metadata. Ok(None) when absent.
  • Iqdb::delete(id) — remove by id; returns whether it was present.
  • Iqdb::len() / Iqdb::is_empty() — cardinality.
  • Iqdb::search(query, k) — top-k similarity search under the database metric.
  • Iqdb::search_with(query, k, filter) — top-k restricted by a metadata Filter.
  • Iqdb::search_batch(...) / search_batch_with(...) — order-preserving batch variants.
  • Iqdb::optimize() — rebuild / retrain the approximate index over the current vectors.
  • Iqdb::cache_stats() — cache hit/miss statistics, when a cache is configured.
  • Iqdb::flush() — compact a file-backed store; no-op in memory.
  • Iqdb::close(self) — final compaction, then release.
• AsyncIqdb — (async feature) a Tokio adapter mirroring the Iqdb surface; offloads each blocking call via spawn_blocking. Clone + Send + Sync.
• IqdbConfig — fluent construction config: dim, metric, an IndexKind, and an optional CacheConfig.
• IndexKind — Flat (exact), Hnsw(HnswConfig), Ivf(IvfConfig).
• HnswConfig / IvfConfig / CacheConfig — re-exported tuning structs for the approximate indices and the cache.
• Vector / VectorId / Metadata / Value / Hit / Filter / DistanceMetric — the shared vocabulary, re-exported from iqdb-types.
• Error / Result<T> — the unified error type (#[non_exhaustive]) and its Result alias.

Error variants

The enum is #[non_exhaustive]; always include a _ arm in a match.

Examples

Self-contained examples live in examples/. Run them with cargo run --example <name>.

• basic — open, upsert, get, search, delete. examples/basic.rs
• in_memory_store — metadata, replace-on-upsert, and a metadata-filtered search. examples/in_memory_store.rs
• search — top-k, batch, and the effect of the distance metric. examples/search.rs
• persistence — three sessions against one durable file, showing data survives reopen. examples/persistence.rs
• index_selection — flat vs HNSW vs IVF through IqdbConfig, plus a cache and optimize. examples/index_selection.rs
• async_search (async feature) — concurrent searches fanned out across Tokio tasks. examples/async_search.rs

cargo run --example index_selection
cargo run --example async_search --features async

Benchmarks

A Criterion harness lives in benches/search.rs:

• flat/search_dim64_n1000_k10 / hnsw/search_dim64_n1000_k10 — top-k query throughput on the exact and approximate paths over 1 000 vectors at dim 64.
• flat/upsert_dim64 — write throughput building a fresh database.

cargo bench --bench search

Criterion writes reports to target/criterion/. A regression beyond the REPS threshold (5% on a tracked metric) blocks a release.

Testing

Every public path has happy / error / edge-case coverage:

• Unit tests live in #[cfg(test)] mod tests blocks inside each source file.
• Integration tests live in tests/:
  • tests/persistence.rs — durable lifecycle: open / upsert / close / reopen, delete and metadata persistence, WAL replay without close, dim/metric-mismatch rejection, IVF round-trip, multi-session accumulation.
  • tests/properties.rs — proptest-driven invariants: flat ranking (sorted, bounded, unique) and the durable round-trip preserving arbitrary record sets.
  • tests/recall.rs — recall@k of HNSW and IVF measured against the exact flat oracle on deterministic synthetic data.
• Doc tests run as part of cargo test and validate every # Examples block.

cargo test
cargo test --all-features
RUSTDOCFLAGS="-D warnings" cargo doc --no-deps
RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features
cargo clippy --all-targets -- -D warnings
cargo clippy --all-targets --all-features -- -D warnings
cargo fmt --all -- --check

Cross-Platform Support

Tier 1 targets — every commit is built and tested on:

• Linux (x86_64-unknown-linux-gnu, aarch64-unknown-linux-gnu)
• macOS (x86_64-apple-darwin, aarch64-apple-darwin)
• Windows (x86_64-pc-windows-msvc)

Durable storage is provided by iqdb-persist, which takes the strongest power-loss sync each platform offers and replaces snapshots atomically. The on-disk format is little-endian on every platform, so a database written on one architecture reads back identically on another.

Configuration

Feature flags

Feature flags are strictly additive (per REPS) — enabling any combination never removes or weakens existing functionality.

iqdb = { version = "0.5", features = ["serde"] }

Runtime configuration

Iqdb::open_in_memory(dim, metric) and Iqdb::open(path, dim, metric) cover the common case. Index selection, tuning, and caching are configured through the fluent IqdbConfig passed to the _with constructors.

Architecture

The crate is the integration layer over the iqdb family; each module owns one concern:

• src/lib.rs — crate root, lint profile, vocabulary re-exports.
• src/handle.rs — the Iqdb handle and its RwLock-guarded in-memory / file-backed storage seam.
• src/config.rs — the fluent IqdbConfig and the IndexKind union.
• src/error.rs — the unified Error wrapping the family error vocabularies.
• src/engine/mod.rs — IqdbCore, the owned engine that implements the iqdb-index and iqdb-persist traits over an authoritative row store plus a derived index.
• src/engine/store.rs — the authoritative, insertion-ordered row store (the single source of truth for len and rebuilds).
• src/engine/index.rs — AnyIndex, the closed enum over FlatIndex / HnswIndex / IvfIndex with the IVF training hooks.
• src/engine/codec.rs — the little-endian on-disk payload codec inside the iqdb-persist frame.

Compile-time guarantees

The crate root enables the strict REPS lint profile in src/lib.rs:

#![deny(warnings)]
#![deny(missing_docs)]
#![deny(unsafe_op_in_unsafe_fn)]
#![deny(unused_must_use)]
#![deny(unused_results)]
#![deny(clippy::unwrap_used)]
#![deny(clippy::expect_used)]
#![deny(clippy::todo)]
#![deny(clippy::unimplemented)]
#![deny(clippy::print_stdout)]
#![deny(clippy::print_stderr)]
#![deny(clippy::dbg_macro)]
#![deny(clippy::unreachable)]
#![deny(clippy::undocumented_unsafe_blocks)]

Test modules locally relax the unwrap_used / expect_used lints — the strict profile is for production library code, not assertion scaffolding inside #[cfg(test)] blocks. The crate contains no unsafe code.

Contributing

Pull requests are welcome. Before opening one, please make sure the full CI gate passes locally:

cargo fmt --all -- --check
cargo clippy --all-targets -- -D warnings
cargo clippy --all-targets --all-features -- -D warnings
cargo test
cargo test --all-features
RUSTDOCFLAGS="-D warnings" cargo doc --no-deps
RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features
cargo deny check
cargo audit

Every contribution is expected to honour the standards in REPS.md — performance, security, error handling, testing, documentation, and dependency hygiene are all enforced as merge gates, not afterthoughts. Commit messages are imperative, lowercase, and scoped to a single logical change.

Links

• Documentation (docs.rs)
• Crates.io
• Repository
• Issues
• Changelog
• Standards (REPS)