iqdb 0.1.0

Embedded vector database for Rust. Lock-free, allocation-free hot path; cross-platform similarity search.
Documentation

Design Goals

iQDB is engineered against the Rust Efficiency & Performance Standards (REPS). Every architectural decision is graded against the same hard constraints:

  • Embedded by default — no daemon, no socket, no separate process. The library opens a path and gets out of the way.
  • Sub-millisecond queries — exact and approximate search paths are budgeted in microseconds, not milliseconds, with the hot path measured under Criterion every release.
  • Lock-free hot paths — atomic reads on the query path; writers do not contend with concurrent readers.
  • Allocation-free steady state — buffers are pooled and reused. Per-query allocation is a design defect.
  • Cache-aware layout — vectors land contiguous in memory and on disk; index nodes are padded to the cache line.
  • Pluggable indices — flat, IVF, and HNSW share a common trait surface so callers can swap strategies without touching their query code.
  • Pluggable storage — durable file-backed storage, transient in-memory storage, and the org-default storage-io substrate are all addressable behind the same handle.
  • Crash-safe writes — the durable path uses write-ahead logging and atomic file replacement. A pulled power cord must never corrupt the index.
  • Tier-1 cross-platform — Linux (x86_64, aarch64), macOS (x86_64, Apple Silicon), Windows (x86_64) all compile and pass the full test suite on every commit.

Status & Roadmap

iQDB ships milestone-by-milestone. Each tag below corresponds to a published release; everything above the current line is shipped, everything below is planned.

Milestone Status Surface delivered
v0.1.0 — scaffolding current Crate scaffolding, lifecycle handle (open / open_in_memory / flush / close), Error type, integration test, criterion harness, CI matrix on all three Tier-1 platforms.
v0.2.0 — vector primitives planned Vector, DistanceMetric (Cosine, L2, Dot), Payload, RecordId. Exact (flat) index. upsert(id, vec) / get(id) / delete(id) against an in-memory store.
v0.3.0 — search planned search(query, k) over the flat index. Batch search. Filter-by-payload predicate. Result ranking with score + id.
v0.4.0 — durable storage planned File-backed storage substrate. Write-ahead log, atomic-replace snapshots, crash recovery. Iqdb::open(path) becomes load-bearing.
v0.5.0 — approximate indices planned IVF and HNSW indices behind the same trait the flat index implements. Build-time index selection via the builder.
v0.6.0 — async surface planned async-feature-gated mirror of the public API. Driven by Tokio. Cancellation-safe.
v0.7.0 — collections planned Named collections / namespaces with per-collection metric and dimensionality. Collection-scoped iteration.
v1.0.0 — API freeze planned Frozen public API. SemVer guarantees. Full benchmark suite. Migration guide from 0.x.

The per-release detail — what was added, what changed, and what was verified — lives in the CHANGELOG and the per-version notes under docs/release/.

Installation

Once 0.1.0 is published, add to your Cargo.toml:

[dependencies]
iqdb = "0.1"

Pre-publish, depend on the repository directly:

[dependencies]
iqdb = { git = "https://github.com/jamesgober/iqdb", branch = "main" }

iQDB compiles on stable Rust 1.75 and newer. There are no required system libraries — only the Rust toolchain is needed for an out-of-the-box build.

Quick Start

The current 0.1.0 surface exposes the database lifecycle. Query verbs (upsert, search, delete) land in subsequent releases — see the roadmap above. The scaffolding compiles, lints clean, and passes the full test suite, but stubs return Error::NotImplemented until the engine lands behind them.

use iqdb::{Iqdb, Result};

fn main() -> Result<()> {
    // Open an ephemeral, in-memory instance. Never touches disk;
    // the entire dataset is dropped when the handle is dropped.
    let db = Iqdb::open_in_memory();

    // File-backed open lands in v0.4.0 — currently returns
    // Error::NotImplemented so call sites can be wired now and
    // light up automatically when the engine ships.
    // let db = Iqdb::open("./data/vectors.iqdb")?;

    // Release the handle. Cleanup happens here (sync, lock release,
    // file close — whatever the active backend needs).
    db.close()?;
    Ok(())
}

Handling the staged surface

Because methods like Iqdb::open and Iqdb::flush are stubs until their respective milestones, downstream callers can safely wire them in advance and gate behaviour on the error variant:

use iqdb::{Error, Iqdb};

fn flush_if_supported(db: &Iqdb) -> Result<(), Error> {
    match db.flush() {
        Ok(()) => Ok(()),
        Err(Error::NotImplemented) => {
            // The active milestone does not yet support flushing.
            // In v0.4.0 (durable storage) this branch disappears.
            Ok(())
        }
        Err(err) => Err(err),
    }
}

let db = Iqdb::open_in_memory();
flush_if_supported(&db).unwrap();

API Overview

A dedicated docs/API.md reference lands alongside the query surface in a subsequent release. Until then, the rustdoc-generated API documentation at docs.rs/iqdb is the canonical reference. The currently-stable items are:

  • Iqdb — the top-level database handle. Owns the open backend and exposes the lifecycle verbs.
    • Iqdb::open(path) — open or create a file-backed database (planned for v0.4.0 — currently returns Error::NotImplemented).
    • Iqdb::open_in_memory() — open an ephemeral instance backed entirely by RAM. Never touches the filesystem.
    • Iqdb::flush() — flush pending writes to durable storage (planned for v0.4.0).
    • Iqdb::close(self) — close the handle and release all held resources.
  • Error — the unified error type. #[non_exhaustive]; new variants are added as new failure modes appear. Always include a _ arm when matching.
  • Result<T> — alias for core::result::Result<T, Error>.

Error variants

Variant Meaning Recovery
Error::Io(std::io::Error) A lower-level I/O failure occurred — disk full, permission denied, missing path, etc. Inspect the wrapped ErrorKind. Retry transient errors; surface permanent ones.
Error::InvalidConfig(&'static str) Configuration supplied at open time was invalid (e.g., zero-length path, unsupported metric). Programmer error — fix the construction site.
Error::NotImplemented The requested operation belongs to a later milestone and has no engine behind it yet. Either upgrade to a release where the verb is implemented, or fall back to an in-memory backend.

The enum is #[non_exhaustive]. New variants will appear in minor releases as new failure modes emerge. Exhaustive match arms are a forward-compatibility hazard — always include _.

Examples

Self-contained examples live in examples/ and are declared in Cargo.toml. Run them with cargo run --example <name>.

  • Lifecycle (basic) — open an in-memory instance and close it cleanly.

Additional examples land alongside their milestones — vector upsert + search in v0.2.0 / v0.3.0, durable open in v0.4.0, approximate search in v0.5.0, async usage in v0.6.0.

Benchmarks

A criterion harness is wired in benches/scaffold.rs. The current bench is a no-op placeholder — it exists so the harness is proven to compile and run on every CI build. Real workloads land alongside the milestones they exercise:

  • v0.2.0 — upsert throughput, get latency.
  • v0.3.0 — exact-search latency by k and dimensionality.
  • v0.4.0 — durable write throughput, recovery time.
  • v0.5.0 — IVF and HNSW build time, recall vs. latency.

Run the current scaffold to confirm the harness works on your machine:

cargo bench --bench scaffold

Criterion writes reports to target/criterion/. Compare runs over time to detect regressions; CI will gate merges on a regression threshold once the real benches land.

Testing

Coverage today is intentionally scoped to the stub surface:

  • Unit tests live in #[cfg(test)] mod tests blocks inside each source file.
  • Integration tests live in tests/ — currently tests/smoke.rs covers the lifecycle on the in-memory backend.
  • Doc tests run as part of cargo test and validate every code block in the rustdoc.
# Full test sweep (unit + integration + doc tests)
cargo test

# Documentation build with no warnings (matches CI gating)
RUSTDOCFLAGS="-D warnings" cargo doc --no-deps

# Lint at the strict profile CI enforces
cargo clippy --all-targets -- -D warnings

# Formatting check (no diffs)
cargo fmt --all -- --check

As real engine code lands, every public path will be backed by happy / error / edge-case tests per the REPS testing mandate.

Cross-Platform Support

Tier 1 targets — every commit is built and tested on:

  • Linux (x86_64-unknown-linux-gnu, aarch64-unknown-linux-gnu)
  • macOS (x86_64-apple-darwin, aarch64-apple-darwin)
  • Windows (x86_64-pc-windows-msvc)

Platform-specific paths — when the durable storage substrate lands, the platform-conditional code paths become observable:

  • Linux: io_uring submission for batch writes; fsync for journal durability.
  • macOS: F_FULLFSYNC for journal durability; mmap for read-mostly indices.
  • Windows: FlushFileBuffers for journal durability; MoveFileExW with MOVEFILE_REPLACE_EXISTING for atomic snapshot replacement.

No platform is silently degraded — fallbacks are explicit and documented inline at each #[cfg] boundary, per the REPS Platform-Specific Code rule.

Configuration

Feature flags

The default feature set at 0.1.0 is empty — the scaffolding has no optional code paths yet. Feature flags will be additive only (per REPS) as subsystems land:

Feature Default Planned for Description
(none yet) All current functionality is unconditional.
async off v0.6.0 Tokio-driven async mirror of the public API.
serde off v0.2.0 Serialize / Deserialize for Vector and payloads.
mmap off v0.4.0 Memory-mapped read path for hot indices.
io-uring off v0.4.0 Linux-only io_uring submission for batch writes.
full off post-1.0 All stable features in one switch. 
# Future: pick a curated set
iqdb = { version = "0.1", features = ["serde"] }

Runtime configuration

Iqdb::open_in_memory() takes no parameters by design. A builder (IqdbBuilder) is introduced in v0.4.0 once the file-backed path lands and has more than two open-time knobs. Until then, the constructor surface is the entire configuration surface.

Architecture

The crate is split along strict module boundaries — each module owns one concern and exposes a single trait or type as its contract:

  • src/lib.rs — crate root, lint profile, public re-exports.
  • src/db.rs — the Iqdb handle and lifecycle verbs.
  • src/error.rs — the Error enum and Result alias.

As milestones land, the tree grows along bounded responsibilities (vector/, index/, storage/, journal/, query/, async_impl.rs). The boundary between layers is always a trait — concrete implementations are crate-internal and gated behind pub(crate).

Compile-time guarantees

The crate root enables the strict REPS lint profile in src/lib.rs:

#![deny(warnings)]
#![deny(missing_docs)]
#![deny(unsafe_op_in_unsafe_fn)]
#![deny(unused_must_use)]
#![deny(unused_results)]
#![deny(clippy::unwrap_used)]
#![deny(clippy::expect_used)]
#![deny(clippy::todo)]
#![deny(clippy::unimplemented)]
#![deny(clippy::print_stdout)]
#![deny(clippy::print_stderr)]
#![deny(clippy::dbg_macro)]
#![deny(clippy::unreachable)]
#![deny(clippy::undocumented_unsafe_blocks)]

Test modules locally relax the unwrap_used / expect_used lints — the strict profile is for production library code, not assertion scaffolding inside #[cfg(test)] blocks.

Contributing

Pull requests are welcome. Before opening one, please make sure the full CI gate passes locally:

cargo fmt --all -- --check
cargo clippy --all-targets -- -D warnings
cargo test
RUSTDOCFLAGS="-D warnings" cargo doc --no-deps
cargo deny check
cargo audit

Every contribution is expected to honour the standards in REPS.md — performance, security, error handling, testing, documentation, and dependency hygiene are all enforced as merge gates, not afterthoughts. Commit messages are imperative, lowercase, and scoped to a single logical change.

Links