iqdb 0.9.0

Design Goals

iQDB is engineered against the Rust Efficiency & Performance Standards (REPS). Every architectural decision is graded against the same hard constraints:

• Embedded by default — no daemon, no socket, no separate process. The library opens a path and gets out of the way.
• Sub-millisecond queries — exact and approximate search paths are budgeted in microseconds, not milliseconds, with the hot path measured under Criterion every release.
• Enum-dispatched hot paths — the index seam dispatches through a closed match, never dyn, so the query loop sees a concrete index with no virtual indirection.
• Allocation-aware steady state — vector payloads are shared as Arc<[f32]> between the authoritative store and the derived index, so a vector that lives in both costs one allocation, not two.
• Pluggable indices — flat, IVF, and HNSW share the iqdb-index trait surface so callers swap strategies through IqdbConfig without touching their query code. Flat is the exact recall ground truth the approximate indices are measured against.
• Crash-safe writes — the durable path uses write-ahead logging and atomic snapshot replacement. A pulled power cord must never corrupt the database.
• Tier-1 cross-platform — Linux (x86_64, aarch64), macOS (x86_64, Apple Silicon), Windows (x86_64) all compile and pass the full test suite on every commit.

Status & Roadmap

iQDB ships milestone-by-milestone. Each tag below corresponds to a published release; everything above the current line is shipped, everything below is planned.

Milestone	Status	Surface delivered
v0.1.0 — scaffolding	shipped	Crate scaffolding, lifecycle handle, Error type, CI matrix on all three Tier-1 platforms.
v0.2.0 — vector primitives	shipped	Validated vectors, distance metrics, typed payloads, in-memory store with thread-safe CRUD.
v0.3.0 — search	shipped	Flat top-k search, filters, batch variants, NaN-aware ranking, property-based tests.
v0.4.0 — durable storage	shipped	Directory-backed store, snapshot + WAL, cross-platform sync, atomic compaction, corrupt-tail recovery.
v0.5.0 — family composition + approximate indices	shipped	Re-platformed onto the iqdb crate family. Re-exported vocabulary (Vector, VectorId, Metadata, Value, Hit, Filter, DistanceMetric). Selectable index — exact Flat, plus Hnsw and Ivf through IqdbConfig. Durable storage via iqdb-persist; optional result cache via iqdb-cache. Recall validated against the flat oracle.
v0.6.0 — async surface	shipped	async-feature-gated AsyncIqdb: a Tokio adapter that offloads each blocking call via spawn_blocking. Additive; the synchronous API and default build are unchanged.
v0.7.0 — durability tuning (alpha)	shipped	IqdbConfig::fsync (WAL fsync cadence) and IqdbConfig::compression (snapshot zstd / lz4), wiring the compression features through. Additive; defaults unchanged.
v0.8.0 — decoder hardening (beta)	shipped	Bounded every on-disk-decoder allocation against hostile length fields; fuzz-style robustness tests for the frame decoder; verified cargo deny / cargo audit pass. No API change.
v0.9.0 — release candidate	current	Crash-recovery integration tests (corrupt WAL tail / corrupt snapshot); captured criterion benchmark baselines. No API change.
v1.0.0 — API freeze	planned	Frozen public API and on-disk format. SemVer guarantees. Full benchmark suite.

The per-release detail — what was added, what changed, and what was verified — lives in the CHANGELOG and the per-version notes under docs/release/.

Installation

Add to your Cargo.toml:

[dependencies]
iqdb = "0.5"

Optional features (all additive):

[dependencies]
iqdb = { version = "0.5", features = ["serde", "parallel", "zstd"] }

iQDB compiles on stable Rust 1.87 and newer. As of 0.5.0 it composes the published iqdb family crates (iqdb-types, iqdb-index, iqdb-flat, iqdb-hnsw, iqdb-ivf, iqdb-build, iqdb-persist, iqdb-cache, and their transitive dependencies); it is no longer a zero-dependency build.

Quick Start

A database fixes its dimensionality and distance metric at open time, then exposes a small surface: upsert / get / delete for records, search / search_with for queries.

use iqdb::{DistanceMetric, Iqdb, Result, Vector, VectorId};

fn main() -> Result<()> {
    // A 3-dimensional, in-memory database compared under cosine distance.
    let db = Iqdb::open_in_memory(3, DistanceMetric::Cosine)?;

    db.upsert(VectorId::from(1u64), Vector::new(vec![1.0, 0.0, 0.0])?, None)?;
    db.upsert(VectorId::from(2u64), Vector::new(vec![0.99, 0.10, 0.0])?, None)?;

    // Top-k similarity search. Results are sorted nearest-first under the
    // smaller-is-closer rule; ties break on insertion order for determinism.
    let hits = db.search(&Vector::new(vec![1.0, 0.0, 0.0])?, 5)?;
    assert_eq!(hits[0].id, VectorId::from(1u64));

    db.close()
}

Filtered and batch search

Filters are declarative Filter expressions evaluated against each record's Metadata. On the exact flat index the filter is applied before scoring, so the result is exact.

use iqdb::{DistanceMetric, Filter, Iqdb, Metadata, Result, Value, Vector, VectorId};

fn main() -> Result<()> {
    let db = Iqdb::open_in_memory(2, DistanceMetric::Cosine)?;

    let doc: Metadata = [("kind".to_string(), Value::String("doc".into()))]
        .into_iter().collect();
    let img: Metadata = [("kind".to_string(), Value::String("image".into()))]
        .into_iter().collect();
    db.upsert(VectorId::from(1u64), Vector::new(vec![1.0, 0.0])?, Some(doc))?;
    db.upsert(VectorId::from(2u64), Vector::new(vec![0.99, 0.10])?, Some(img))?;

    // Only documents, ranked by cosine distance.
    let filter = Filter::eq("kind", Value::String("doc".into()));
    let hits = db.search_with(&Vector::new(vec![1.0, 0.0])?, 5, filter)?;
    assert_eq!(hits.len(), 1);
    assert_eq!(hits[0].id, VectorId::from(1u64));

    // Batch — one top-k result list per query, preserving input order.
    let queries = vec![Vector::new(vec![1.0, 0.0])?, Vector::new(vec![0.0, 1.0])?];
    let batches = db.search_batch(&queries, 1)?;
    assert_eq!(batches.len(), 2);

    Ok(())
}

Choosing an index

Tier 1 (open_in_memory / open) defaults to the exact flat index. Tier 2 (open_in_memory_with / open_with) takes an IqdbConfig that selects an approximate index and tunes it, and can attach a result cache.

use iqdb::{CacheConfig, DistanceMetric, HnswConfig, IndexKind, Iqdb, IqdbConfig, Result};

fn main() -> Result<()> {
    let cfg = IqdbConfig::new(128, DistanceMetric::Cosine)
        .index(IndexKind::Hnsw(HnswConfig::default().with_ef_search(96)))
        .cache(CacheConfig::new().capacity(10_000));
    let db = Iqdb::open_in_memory_with(cfg)?;
    assert!(db.is_empty());
    Ok(())
}

On the approximate indices the metadata filter is applied after graph / cluster traversal, so a highly selective filter can return fewer than k hits — widen the search (HNSW filter_widen, IVF n_probes) when that matters. IVF is trained lazily from the stored vectors on the first search; after many writes, Iqdb::optimize retrains its centroids.

Durable, file-backed storage

Iqdb::open(path, dim, metric) opens or creates a durable database. The path is the snapshot file; a write-ahead log lives beside it. Acknowledged writes survive a crash; reopening replays the log onto the snapshot.

use iqdb::{DistanceMetric, Iqdb, Result, Vector, VectorId};

fn main() -> Result<()> {
    let db = Iqdb::open("./data/vectors.iqdb", 3, DistanceMetric::Cosine)?;
    db.upsert(VectorId::from(1u64), Vector::new(vec![0.1, 0.2, 0.3])?, None)?;
    db.flush()?; // compact: fold the WAL into a fresh snapshot
    db.close()
}

A reopen whose requested dim / metric disagrees with the stored database fails with Error::Config. The stored index kind is part of the database identity and is restored from the snapshot regardless of the kind requested on reopen.

By default every acknowledged write is fsynced and the snapshot is uncompressed. Trade durability for throughput, or shrink the snapshot, through IqdbConfig:

use iqdb::{Compression, DistanceMetric, FsyncPolicy, Iqdb, IqdbConfig};
use std::time::Duration;

# fn run() -> iqdb::Result<()> {
let cfg = IqdbConfig::new(128, DistanceMetric::Cosine)
    .fsync(FsyncPolicy::Periodic(Duration::from_millis(50))) // bound the un-synced window
    .compression(Compression::Zstd { level: 3 });            // requires the `zstd` feature
let db = Iqdb::open_with("./data/vectors.iqdb", cfg)?;
# let _ = db;
# Ok(())
# }

Async (the async feature)

The family is synchronous by design, so the async surface is a thin Tokio adapter: AsyncIqdb holds an Arc<Iqdb> and runs each blocking call on Tokio's blocking pool via spawn_blocking, so awaiting a search or a write never stalls the executor. It is Clone + Send + Sync. Enable the async feature and bring your own runtime.

use iqdb::{AsyncIqdb, DistanceMetric, Result, Vector, VectorId};

#[tokio::main]
async fn main() -> Result<()> {
    let db = AsyncIqdb::open_in_memory(3, DistanceMetric::Cosine).await?;
    db.upsert(VectorId::from(1u64), Vector::new(vec![1.0, 0.0, 0.0])?, None).await?;

    let hits = db.search(Vector::new(vec![1.0, 0.0, 0.0])?, 1).await?;
    assert_eq!(hits[0].id, VectorId::from(1u64));
    db.close().await
}

API Overview

The full API reference lives at docs/API.md; the rustdoc at docs.rs/iqdb carries the same information in browseable form. The public surface:

• Iqdb — the database handle.
  • Iqdb::open_in_memory(dim, metric) — an ephemeral, exact-flat database.
  • Iqdb::open_in_memory_with(config) — an in-memory database from a full IqdbConfig.
  • Iqdb::open(path, dim, metric) / Iqdb::open_with(path, config) — a durable, file-backed database.
  • Iqdb::upsert(id, vector, metadata) — insert or replace. Rejects a wrong-dimension vector at the boundary.
  • Iqdb::get(id) — look up the stored vector and metadata. Ok(None) when absent.
  • Iqdb::delete(id) — remove by id; returns whether it was present.
  • Iqdb::len() / Iqdb::is_empty() — cardinality.
  • Iqdb::search(query, k) — top-k similarity search under the database metric.
  • Iqdb::search_with(query, k, filter) — top-k restricted by a metadata Filter.
  • Iqdb::search_batch(...) / search_batch_with(...) — order-preserving batch variants.
  • Iqdb::optimize() — rebuild / retrain the approximate index over the current vectors.
  • Iqdb::cache_stats() — cache hit/miss statistics, when a cache is configured.
  • Iqdb::flush() — compact a file-backed store; no-op in memory.
  • Iqdb::close(self) — final compaction, then release.
• AsyncIqdb — (async feature) a Tokio adapter mirroring the Iqdb surface; offloads each blocking call via spawn_blocking. Clone + Send + Sync.
• IqdbConfig — fluent construction config: dim, metric, an IndexKind, and an optional CacheConfig.
• IndexKind — Flat (exact), Hnsw(HnswConfig), Ivf(IvfConfig).
• HnswConfig / IvfConfig / CacheConfig — re-exported tuning structs for the approximate indices and the cache.
• Vector / VectorId / Metadata / Value / Hit / Filter / DistanceMetric — the shared vocabulary, re-exported from iqdb-types.
• Error / Result<T> — the unified error type (#[non_exhaustive]) and its Result alias.

Error variants

Variant	Meaning	Recovery
Error::Index(IqdbError)	A failure from the index / vocabulary layer — dimension mismatch, absent id, invalid metric for the chosen index, malformed filter.	Inspect the wrapped [iqdb_types::IqdbError] kind and fix the construction or query site.
Error::Persist(PersistError)	A failure from the durable-storage layer — snapshot / WAL I/O, a corrupt or truncated file, a checksum mismatch, or an unsupported compression feature.	Inspect the wrapped [iqdb_persist::PersistError]. A corrupt WAL tail is truncated automatically; a corrupt snapshot fails the open.
Error::Config(&'static str)	A handle-level consistency check failed — most often a reopen whose dim / metric does not match the stored database.	Open with the values the database was created with.

The enum is #[non_exhaustive]; always include a _ arm in a match.

Examples

Self-contained examples live in examples/. Run them with cargo run --example <name>.

• basic — open, upsert, get, search, delete. examples/basic.rs
• in_memory_store — metadata, replace-on-upsert, and a metadata-filtered search. examples/in_memory_store.rs
• search — top-k, batch, and the effect of the distance metric. examples/search.rs
• persistence — three sessions against one durable file, showing data survives reopen. examples/persistence.rs
• index_selection — flat vs HNSW vs IVF through IqdbConfig, plus a cache and optimize. examples/index_selection.rs
• async_search (async feature) — concurrent searches fanned out across Tokio tasks. examples/async_search.rs

cargo run --example index_selection
cargo run --example async_search --features async

Benchmarks

A Criterion harness lives in benches/search.rs:

• flat/search_dim64_n1000_k10 / hnsw/search_dim64_n1000_k10 — top-k query throughput on the exact and approximate paths over 1 000 vectors at dim 64.
• flat/upsert_dim64 — write throughput building a fresh database.

cargo bench --bench search

Indicative baselines on a developer machine (dim 64, 1 000 vectors): flat search ≈ 7.9 µs, HNSW search ≈ 35.8 µs, flat upsert of 1 000 vectors ≈ 185 µs. At this corpus size the exact flat scan beats HNSW's graph traversal — the approximate index earns its overhead at much larger scale.

Criterion writes reports to target/criterion/. A regression beyond the REPS threshold (5% on a tracked metric) blocks a release.

Testing

Every public path has happy / error / edge-case coverage:

• Unit tests live in #[cfg(test)] mod tests blocks inside each source file.
• Integration tests live in tests/:
  • tests/persistence.rs — durable lifecycle: open / upsert / close / reopen, delete and metadata persistence, WAL replay without close, dim/metric-mismatch rejection, IVF round-trip, multi-session accumulation.
  • tests/properties.rs — proptest-driven invariants: flat ranking (sorted, bounded, unique) and the durable round-trip preserving arbitrary record sets.
  • tests/recall.rs — recall@k of HNSW and IVF measured against the exact flat oracle on deterministic synthetic data.
  • tests/recovery.rs — crash recovery: a torn WAL tail is truncated (prior records survive), a corrupt snapshot fails the open, a non-database file is rejected.
• Doc tests run as part of cargo test and validate every # Examples block.

cargo test
cargo test --all-features
RUSTDOCFLAGS="-D warnings" cargo doc --no-deps
RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features
cargo clippy --all-targets -- -D warnings
cargo clippy --all-targets --all-features -- -D warnings
cargo fmt --all -- --check

Cross-Platform Support

Tier 1 targets — every commit is built and tested on:

• Linux (x86_64-unknown-linux-gnu, aarch64-unknown-linux-gnu)
• macOS (x86_64-apple-darwin, aarch64-apple-darwin)
• Windows (x86_64-pc-windows-msvc)

Durable storage is provided by iqdb-persist, which takes the strongest power-loss sync each platform offers and replaces snapshots atomically. The on-disk format is little-endian on every platform, so a database written on one architecture reads back identically on another.

Configuration

Feature flags

Feature flags are strictly additive (per REPS) — enabling any combination never removes or weakens existing functionality.

Feature	Default	Description
serde	off	Derive Serialize / Deserialize on the public data types (forwards to the family serde features).
parallel	off	Rayon-backed parallel distance scan on the flat index (forwards to iqdb-flat).
zstd	off	Zstandard snapshot compression (forwards to iqdb-persist).
lz4	off	LZ4 snapshot compression (forwards to iqdb-persist).
async	off	Tokio-driven AsyncIqdb mirror of the public API. Pulls tokio (only the rt feature).

iqdb = { version = "0.5", features = ["serde"] }

Runtime configuration

Iqdb::open_in_memory(dim, metric) and Iqdb::open(path, dim, metric) cover the common case. Index selection, tuning, and caching are configured through the fluent IqdbConfig passed to the _with constructors.

Architecture

The crate is the integration layer over the iqdb family; each module owns one concern:

• src/lib.rs — crate root, lint profile, vocabulary re-exports.
• src/handle.rs — the Iqdb handle and its RwLock-guarded in-memory / file-backed storage seam.
• src/config.rs — the fluent IqdbConfig and the IndexKind union.
• src/error.rs — the unified Error wrapping the family error vocabularies.
• src/engine/mod.rs — IqdbCore, the owned engine that implements the iqdb-index and iqdb-persist traits over an authoritative row store plus a derived index.
• src/engine/store.rs — the authoritative, insertion-ordered row store (the single source of truth for len and rebuilds).
• src/engine/index.rs — AnyIndex, the closed enum over FlatIndex / HnswIndex / IvfIndex with the IVF training hooks.
• src/engine/codec.rs — the little-endian on-disk payload codec inside the iqdb-persist frame.

Compile-time guarantees

The crate root enables the strict REPS lint profile in src/lib.rs:

#![deny(warnings)]
#![deny(missing_docs)]
#![deny(unsafe_op_in_unsafe_fn)]
#![deny(unused_must_use)]
#![deny(unused_results)]
#![deny(clippy::unwrap_used)]
#![deny(clippy::expect_used)]
#![deny(clippy::todo)]
#![deny(clippy::unimplemented)]
#![deny(clippy::print_stdout)]
#![deny(clippy::print_stderr)]
#![deny(clippy::dbg_macro)]
#![deny(clippy::unreachable)]
#![deny(clippy::undocumented_unsafe_blocks)]

Test modules locally relax the unwrap_used / expect_used lints — the strict profile is for production library code, not assertion scaffolding inside #[cfg(test)] blocks. The crate contains no unsafe code.

Contributing

Pull requests are welcome. Before opening one, please make sure the full CI gate passes locally:

cargo fmt --all -- --check
cargo clippy --all-targets -- -D warnings
cargo clippy --all-targets --all-features -- -D warnings
cargo test
cargo test --all-features
RUSTDOCFLAGS="-D warnings" cargo doc --no-deps
RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features
cargo deny check
cargo audit

Every contribution is expected to honour the standards in REPS.md — performance, security, error handling, testing, documentation, and dependency hygiene are all enforced as merge gates, not afterthoughts. Commit messages are imperative, lowercase, and scoped to a single logical change.

Links

• Documentation (docs.rs)
• Crates.io
• Repository
• Issues
• Changelog
• Standards (REPS)