iqdb 0.4.0

Design Goals

iQDB is engineered against the Rust Efficiency & Performance Standards (REPS). Every architectural decision is graded against the same hard constraints:

Embedded by default — no daemon, no socket, no separate process. The library opens a path and gets out of the way.
Sub-millisecond queries — exact and approximate search paths are budgeted in microseconds, not milliseconds, with the hot path measured under Criterion every release.
Lock-free hot paths — atomic reads on the query path; writers do not contend with concurrent readers.
Allocation-free steady state — buffers are pooled and reused. Per-query allocation is a design defect.
Cache-aware layout — vectors land contiguous in memory and on disk; index nodes are padded to the cache line.
Pluggable indices — flat, IVF, and HNSW share a common trait surface so callers can swap strategies without touching their query code.
Pluggable storage — durable file-backed storage, transient in-memory storage, and the org-default storage-io substrate are all addressable behind the same handle.
Crash-safe writes — the durable path uses write-ahead logging and atomic file replacement. A pulled power cord must never corrupt the index.
Tier-1 cross-platform — Linux (x86_64, aarch64), macOS (x86_64, Apple Silicon), Windows (x86_64) all compile and pass the full test suite on every commit.

Status & Roadmap

iQDB ships milestone-by-milestone. Each tag below corresponds to a published release; everything above the current line is shipped, everything below is planned.

Milestone	Status	Surface delivered
`v0.1.0` — scaffolding	shipped	Crate scaffolding, lifecycle handle (`open` / `open_in_memory` / `flush` / `close`), `Error` type, integration test, criterion harness, CI matrix on all three Tier-1 platforms.
`v0.2.0` — vector primitives	shipped	`Vector` (validated f32 embeddings), `DistanceMetric` (L2 / Cosine / Dot), `Payload` & `PayloadValue` (typed metadata), `RecordId`, `Record`. In-memory store with thread-safe `upsert` / `get` / `delete` / `len` / `is_empty`. Optional `serde` feature.
`v0.3.0` — search	shipped	`Iqdb::search` / `search_with` / `search_batch` / `search_batch_with` over the flat index. `SearchResult { id, score, payload }`. Monomorphic predicate filters. NaN-aware ranking with id tie-break. Property-based ranking tests via `proptest`. `docs/API.md` published.
`v0.4.0` — durable storage	current	Directory-backed `Iqdb::open(path)`. Write-ahead log + snapshot. Cross-platform `full_sync` (`F_FULLFSYNC` on macOS, `fsync(2)` on other Unix, `FlushFileBuffers` on Windows). Atomic-replace snapshot compaction on `close`. Recovery handles corrupt WAL tails by truncating to last known-good offset. New `Error::Corrupt` variant.
`v0.5.0` — approximate indices	planned	IVF and HNSW indices behind the same trait the flat index implements. Build-time index selection via the builder.
`v0.6.0` — async surface	planned	`async`-feature-gated mirror of the public API. Driven by Tokio. Cancellation-safe.
`v0.7.0` — collections	planned	Named collections / namespaces with per-collection metric and dimensionality. Collection-scoped iteration.
`v1.0.0` — API freeze	planned	Frozen public API. SemVer guarantees. Full benchmark suite. Migration guide from 0.x.

The per-release detail — what was added, what changed, and what was verified — lives in the CHANGELOG and the per-version notes under docs/release/.

Installation

Add to your Cargo.toml:

[dependencies]
iqdb = "0.4"

Enable the optional serde feature to derive Serialize / Deserialize on every public data type:

[dependencies]
iqdb = { version = "0.4", features = ["serde"] }

iQDB compiles on stable Rust 1.87 and newer. The default build pulls one runtime dependency on Unix (libc, used exclusively for the macOS F_FULLFSYNC call) and zero on Windows; the serde feature additionally pulls serde itself.

Quick Start

The 0.4.0 surface exposes the full embedded-DB lifecycle: typed vector primitives, the in-memory store, exact top-k similarity search, and directory-backed durable storage. Approximate indices (IVF, HNSW) in v0.5.0 will sit alongside the flat kernel rather than replacing it — exact search remains the correctness baseline.

use iqdb::{DistanceMetric, Iqdb, Payload, Record, RecordId, Result, Vector};

fn main() -> Result<()> {
    let db = Iqdb::open_in_memory();

    let mut meta = Payload::new();
    meta.insert("kind", "doc");

    db.upsert(Record::with_payload(
        RecordId::new(1),
        Vector::new(vec![1.0, 0.0, 0.0])?,
        meta,
    ))?;
    db.upsert(Record::new(
        RecordId::new(2),
        Vector::new(vec![0.99, 0.10, 0.0])?,
    ))?;

    // Top-k similarity search. Results are sorted ascending under the
    // smaller-is-closer rule; ties break on id for determinism.
    let probe = Vector::new(vec![1.0, 0.0, 0.0])?;
    let hits = db.search(&probe, 5, DistanceMetric::Cosine)?;
    assert_eq!(hits.first().map(|h| h.id), Some(RecordId::new(1)));

    db.close()
}

Filtered and batch search

Filters are generic — the predicate monomorphises into the search loop, so there is no per-record dynamic dispatch. The smaller-is-closer convention holds across all three metrics; Dot returns −(a · b) so a single ordering rule covers L2, Cosine, and Dot.

use iqdb::{DistanceMetric, Iqdb, Payload, PayloadValue, Record, RecordId, Result, Vector};

fn main() -> Result<()> {
    let db = Iqdb::open_in_memory();

    let mut doc = Payload::new();
    doc.insert("kind", "doc");
    db.upsert(Record::with_payload(
        RecordId::new(1),
        Vector::new(vec![1.0, 0.0])?,
        doc,
    ))?;

    let mut image = Payload::new();
    image.insert("kind", "image");
    db.upsert(Record::with_payload(
        RecordId::new(2),
        Vector::new(vec![0.99, 0.10])?,
        image,
    ))?;

    let probe = Vector::new(vec![1.0, 0.0])?;

    // Filter the candidate set before heap admission.
    let docs_only = db.search_with(&probe, 5, DistanceMetric::Cosine, |rec| {
        rec.payload()
            .and_then(|p| p.get("kind"))
            .and_then(PayloadValue::as_text)
            == Some("doc")
    })?;
    assert_eq!(docs_only.len(), 1);
    assert_eq!(docs_only[0].id, RecordId::new(1));

    // Batch — one top-k result list per query, preserves input order.
    let probes = vec![
        Vector::new(vec![1.0, 0.0])?,
        Vector::new(vec![0.0, 1.0])?,
    ];
    let batches = db.search_batch(&probes, 1, DistanceMetric::Cosine)?;
    assert_eq!(batches.len(), 2);

    Ok(())
}

Directory-backed durable store

Iqdb::open(path) opens or creates a directory-backed database. The directory contains a snapshot file and a write-ahead log; records survive process restart. flush syncs the WAL to durable storage; close runs a compaction (snapshot rewrite + WAL truncate) so the next open is a single-file load with no replay.

use iqdb::{Iqdb, Record, RecordId, Result, Vector};

fn main() -> Result<()> {
    // Open or create at the given path. If the path does not exist,
    // it is created as a directory. Subsequent opens replay the WAL
    // on top of the snapshot to reconstruct the in-memory state.
    let db = Iqdb::open("./data/my-db")?;

    db.upsert(Record::new(
        RecordId::new(1),
        Vector::new(vec![0.1, 0.2, 0.3])?,
    ))?;

    // Drive the WAL to durable storage. Uses F_FULLFSYNC on macOS,
    // fsync(2) on other Unix, FlushFileBuffers on Windows.
    db.flush()?;

    // Compact: write a fresh snapshot, atomically replace the old one,
    // truncate the WAL. The next Iqdb::open of the same path will load
    // the snapshot in a single shot.
    db.close()
}

A corrupt WAL tail (from a prior crash) is truncated to the last known-good offset on open. A corrupt snapshot surfaces as Error::Corrupt { reason } — recovery from a checkpoint chain is a future-milestone concern.

API Overview

The full API reference lives at docs/API.md; the rustdoc-generated docs at docs.rs/iqdb carry the same information in browseable form. The currently-stable items are:

Iqdb — the top-level database handle.
- Iqdb::open(path) — open or create a directory-backed durable database (snapshot + WAL). Replays the WAL on top of the snapshot during recovery and truncates corrupt tails.
- Iqdb::open_in_memory() — open an ephemeral instance backed entirely by RAM.
- Iqdb::upsert(record) — insert or replace a record. Idempotent.
- Iqdb::get(id) — look up by id. Returns Ok(None) when absent.
- Iqdb::delete(id) — remove by id. Returns whether the id was present.
- Iqdb::len() / Iqdb::is_empty() — store cardinality.
- Iqdb::search(query, k, metric) — exact top-k similarity search, no filter.
- Iqdb::search_with(query, k, metric, filter) — top-k with a per-record predicate. The filter monomorphises into the scan loop; no per-record dynamic dispatch.
- Iqdb::search_batch(queries, k, metric) / search_batch_with(...) — sequential batch variants. Preserves input order.
- Iqdb::flush() — sync the WAL to durable storage. F_FULLFSYNC on macOS, fsync(2) on other Unix, FlushFileBuffers on Windows. No-op for the in-memory backend.
- Iqdb::close(self) — close the handle. For the file backend, runs a compaction (snapshot rewrite + WAL truncate).
Vector — owned, contiguous, validated f32 embedding. Vector::new(Vec<f32>) / Vector::from_slice(&[f32]) validate at the system boundary (no empty vectors, no NaN, no infinity); as_slice / dim / norm / norm_squared are non-allocating.
DistanceMetric — L2, Cosine, Dot. metric.distance(a, b) returns a Result<f32> under the smaller-is-closer convention; dimensional homogeneity is enforced.
Payload — typed BTreeMap<String, PayloadValue> for metadata. Deterministic iteration order makes payloads stable across serde round-trips and test assertions.
PayloadValue — Null / Bool / Int / Float / Text / Bytes / Array / nested Object. From<T> conversions cover the primitives.
RecordId — transparent newtype around u64. Cheap to copy, hash, and compare.
Record — (id, vector, optional payload) aggregate. Record::new / Record::with_payload are the two constructors; into_parts decomposes without a clone.
SearchResult — { id, score, payload } returned by the search methods. Sorted ascending by score; ties broken on id; NaN scores sort to the tail.
Error — the unified error type. #[non_exhaustive].
Result<T> — alias for core::result::Result<T, Error>.

Error variants

Variant	Meaning	Recovery
`Error::Io(std::io::Error)`	A lower-level I/O failure occurred — disk full, permission denied, missing path, etc.	Inspect the wrapped `ErrorKind`. Retry transient errors; surface permanent ones.
`Error::InvalidConfig(&'static str)`	Configuration supplied at open time was invalid (e.g., the path exists but is not a directory).	Programmer error — fix the construction site.
`Error::InvalidVector { reason }`	A vector failed boundary validation — empty, or contains `NaN` / ±∞.	Sanitise the input at the producer side; `Vector::new` rejects bad data before it enters the store.
`Error::DimensionMismatch { left, right }`	A distance-metric or store operation combined two vectors of different dimensionality.	Enforce a homogeneous schema at the producer side or surface a typed error to the caller.
`Error::Corrupt { reason }`	On-disk data failed an integrity check during recovery (bad magic, unknown version, CRC mismatch, truncated frame).	Surfaced by `Iqdb::open(path)`. WAL tails are auto-truncated; a corrupt snapshot fails the open.
`Error::NotImplemented`	Reserved for methods that defer their implementation to a later milestone. No public v0.4.0 method returns this variant.	n/a

The enum is #[non_exhaustive]. New variants will appear in minor releases as new failure modes emerge. Exhaustive match arms are a forward-compatibility hazard — always include _.

Examples

Self-contained examples live in examples/ and are declared in Cargo.toml. Run them with cargo run --example <name>.

Lifecycle (basic) — open an in-memory instance, upsert a vector, read it back, close.
- File: examples/basic.rs
- Run:
```
cargo run --example basic --release
```
In-memory store walk-through (in_memory_store) — populate the store with three records (vectors + typed payloads), compare distances between them under L2 and cosine, then delete one.
- File: examples/in_memory_store.rs
- Run:
```
cargo run --example in_memory_store --release
```
Top-k search (search) — unfiltered cosine top-k, payload-filtered search, and batch search across three probes in one file.
- File: examples/search.rs
- Run:
```
cargo run --example search --release
```
Directory-backed persistence (persistence) — open at a path, upsert records, close. Reopen, verify the records survived, run a search, delete one, close. Reopen a third time and confirm the delete persisted. Three sessions, all consistent against the same on-disk database.
- File: examples/persistence.rs
- Run:
```
cargo run --example persistence --release
```

Approximate-index examples land alongside their milestone (v0.5.0).

Benchmarks

A criterion harness is wired in benches/vector_ops.rs. v0.4.0 ships five groups:

vector_new — construction-time validation cost across small / medium / large dimensionalities (32 / 128 / 1024).
distance — single-shot distance computation under each of the three DistanceMetric variants at dim 128.
store — upsert and get throughput against a populated in-memory store at 1 000 records, dim 128.
search — flat top-k search at 1 000 and 10 000 records, dim 128. Three variants: unfiltered, payload-filtered (~50% pruning), and batch-of-4.
file_store — durable upsert + flush to a fresh on-disk database, and snapshot-only open with 1 000 records, dim 128.

Run with:

cargo bench --bench vector_ops

Criterion writes reports to target/criterion/. Approximate-index benches land with v0.5.0; CI will gate merges on a regression threshold once the benches are stable.

Testing

Every public path has happy / error / edge-case coverage:

Unit tests live in #[cfg(test)] mod tests blocks inside each source file.
Integration tests live in tests/:
- tests/in_memory.rs — CRUD surface plus serde round-trips behind a feature gate.
- tests/search.rs — the four search entry points: top-k ordering, filter pruning, batch order, dimension-mismatch handling, payload preservation, concurrent readers.
- tests/persistence.rs — full durable lifecycle: open / upsert / close / reopen, delete persistence, payload round-trip through compaction, WAL replay without close, file-path rejection, corrupt snapshot detection, corrupt WAL tail truncation, multi-cycle state preservation.
- tests/properties.rs — proptest-driven property tests for distance-metric algebra (symmetry, identity, range bounds), search-ranking invariants (length bound, ascending order, perfect-match presence, no-filter parity), and durable round-trip (open → upsert → close → reopen preserves record sets).
- tests/smoke.rs — minimal lifecycle smoke check.
Doc tests run as part of cargo test and validate every # Examples block in the rustdoc.

# Full test sweep (unit + integration + doc tests)
cargo test
cargo test --all-features

# Documentation build with no warnings (matches CI gating)
RUSTDOCFLAGS="-D warnings" cargo doc --no-deps
RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features

# Lint at the strict profile CI enforces
cargo clippy --all-targets -- -D warnings
cargo clippy --all-targets --all-features -- -D warnings

# Formatting check (no diffs)
cargo fmt --all -- --check

Cross-Platform Support

Tier 1 targets — every commit is built and tested on:

Linux (x86_64-unknown-linux-gnu, aarch64-unknown-linux-gnu)
macOS (x86_64-apple-darwin, aarch64-apple-darwin)
Windows (x86_64-pc-windows-msvc)

Platform-specific paths — the durable storage layer takes the strongest sync primitive each platform exposes:

Linux (and other Unix except macOS): fsync(2) via std::fs::File::sync_all. Atomic snapshot replacement via rename(2).
macOS: fcntl(fd, F_FULLFSYNC, 0) — direct libc call, the only platform that needs an escape hatch beyond fsync for true power-loss durability. Atomic snapshot replacement via rename(2).
Windows: FlushFileBuffers via std::fs::File::sync_all. Atomic snapshot replacement via MoveFileExW with MOVEFILE_REPLACE_EXISTING (also via std::fs::rename).

io_uring (Linux batched writes) and mmap (read-mostly indices) are on the v0.4.x / v0.5.0 roadmap behind feature flags; they are not on the default build. No platform is silently degraded — every fallback is explicit and documented inline at each #[cfg] boundary, per the REPS Platform-Specific Code rule.

Configuration

Feature flags

Feature flags are strictly additive (per REPS) — enabling any combination never removes or weakens existing functionality.

Feature	Default	Available	Description
`serde`	off	shipping	Derives `Serialize` / `Deserialize` on every public data type.
`mmap`	off	planned v0.4.x	Memory-mapped read path for hot indices.
`io-uring`	off	planned v0.4.x	Linux-only `io_uring` submission for batch writes.
`async`	off	planned v0.6.0	Tokio-driven async mirror of the public API.
`full`	off	planned post-1.0	All stable features in one switch.

iqdb = { version = "0.4", features = ["serde"] }

Runtime configuration

Iqdb::open_in_memory() takes no parameters; Iqdb::open(path) takes only the path. A builder (IqdbBuilder) for tunable open-time knobs (compaction threshold, WAL fsync cadence, page size) lands in a future milestone once enough knobs accumulate to justify the surface.

Architecture

The crate is split along strict module boundaries — each module owns one concern and exposes a single trait or type as its contract:

src/lib.rs — crate root, lint profile, public re-exports.
src/db.rs — the Iqdb handle, dispatching through the internal Backend enum to the active store.
src/backend.rs — the pub(crate) Backend { Memory, File } enum; pattern-matched dispatch lets the search kernel work over either store with zero dynamic-dispatch cost.
src/vector.rs — the Vector primitive and the DistanceMetric dispatch.
src/payload.rs — the Payload / PayloadValue typed metadata layer.
src/record.rs — the RecordId / Record aggregate.
src/search.rs — the flat top-k search kernel and the SearchResult type.
src/store.rs — the crate-internal MemoryStore (the read/write engine behind open_in_memory).
src/file_store.rs — the crate-internal FileStore (snapshot + WAL behind open(path)).
src/codec.rs — the binary frame encoder / decoder used by the file store. Length-prefixed, CRC32-checked, little-endian throughout.
src/platform.rs — the cross-platform full_sync primitive (F_FULLFSYNC / fsync / FlushFileBuffers).
src/error.rs — the Error enum and Result alias.

As milestones land, the tree grows along the same bounded-responsibility pattern (index/ for IVF + HNSW, async_impl.rs for the Tokio surface). The boundary between layers is always a trait or a concrete type with a typed surface — concrete backend implementations are crate-internal and gated behind pub(crate).

Compile-time guarantees

The crate root enables the strict REPS lint profile in src/lib.rs:

#![deny(warnings)]
#![deny(missing_docs)]
#![deny(unsafe_op_in_unsafe_fn)]
#![deny(unused_must_use)]
#![deny(unused_results)]
#![deny(clippy::unwrap_used)]
#![deny(clippy::expect_used)]
#![deny(clippy::todo)]
#![deny(clippy::unimplemented)]
#![deny(clippy::print_stdout)]
#![deny(clippy::print_stderr)]
#![deny(clippy::dbg_macro)]
#![deny(clippy::unreachable)]
#![deny(clippy::undocumented_unsafe_blocks)]

Test modules locally relax the unwrap_used / expect_used lints — the strict profile is for production library code, not assertion scaffolding inside #[cfg(test)] blocks.

Contributing

Pull requests are welcome. Before opening one, please make sure the full CI gate passes locally:

cargo fmt --all -- --check
cargo clippy --all-targets -- -D warnings
cargo clippy --all-targets --all-features -- -D warnings
cargo test
cargo test --all-features
RUSTDOCFLAGS="-D warnings" cargo doc --no-deps
RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features
cargo deny check
cargo audit

Every contribution is expected to honour the standards in REPS.md — performance, security, error handling, testing, documentation, and dependency hygiene are all enforced as merge gates, not afterthoughts. Commit messages are imperative, lowercase, and scoped to a single logical change.