iqdb-persist 1.0.0

Atomic snapshot persistence with versioned headers and CRC32 integrity for iQDB indexes - part of the iQDB family.
Documentation
  • Atomic snapshot save/load — write-to-temp + fsync + atomic rename + directory fsync; an interrupted write never corrupts an existing good file.
  • Versioned header — magic bytes, format version, index-type tag, dim, metric, vector count. All sizes are fixed-width little-endian u64, so a file is portable across 32- and 64-bit hosts.
  • CRC32 integrity — computed over the payload; a single-bit flip surfaces as ChecksumMismatch on load, never a panic or a silently-wrong result.
  • Write-ahead log & crash recovery — with wal_enabled, every insert / delete is logged and fsynced before it touches memory, then replayed onto the snapshot on load. A crash mid-append leaves a torn tail that recovery detects and discards.
  • Optional compression — Zstd or LZ4 on the snapshot payload, behind the zstd / lz4 cargo features. The CRC32 covers the compressed bytes, so corruption is caught before decompression.
  • Generic over the index — PersistedIndex<I: Index + Persistable> wraps any concrete index; the framing lives here, the payload bytes live in the index's Persistable impl.

Installation

[dependencies]
iqdb-persist = "1.0"
iqdb-index   = "1.0"
iqdb-types   = "1.0"

Snapshot compression is opt-in via cargo features (off by default):

iqdb-persist = { version = "1.0", features = ["zstd", "lz4"] }

Quick start

use iqdb_persist::{PersistConfig, PersistedIndex};

// `MyIndex: iqdb_index::Index + iqdb_persist::Persistable`
let cfg = PersistConfig::new("/var/lib/app/index.iqdb");

// Wrap an in-memory index and save it atomically.
let wrapped = PersistedIndex::open_with(my_index, cfg.clone())?;
wrapped.save()?;

// Later — reconstruct it from disk (verifies magic, version, type, CRC32).
let restored: PersistedIndex<MyIndex> = PersistedIndex::load(cfg)?;
let index = restored.index();

For durable, crash-recoverable mutation, set cfg.wal_enabled = true and mutate through the wrapper — each op is logged before it is applied, and checkpoint() folds the log back into a fresh snapshot:

let mut db = PersistedIndex::open_with(my_index, cfg.clone())?; // writes base snapshot + opens WAL
db.insert(id, vector, metadata)?;   // logged + fsynced, then applied
db.delete(&other_id)?;
db.checkpoint()?;                    // snapshot the state, truncate the WAL
// after a crash: PersistedIndex::load(cfg) replays the WAL onto the snapshot

An index opts in by implementing the two-method Persistable trait. A complete, runnable version lives in examples/save_and_load.rs — run it with cargo run --example save_and_load. Full reference: docs/API.md.

Status

This is v1.0.0 — stable. Atomic snapshots + CRC32 (v0.2), the write-ahead log with replay and crash recovery (v0.3), and optional Zstd/LZ4 snapshot compression (v0.4) are complete; the API and on-disk format were frozen at v0.5 and the parse/recovery paths adversarially hardened; v0.6 added property tests for the core invariants. The public API and on-disk format are now committed under the SemVer 1.x guarantee — no breaking changes before 2.0. The external storage-io substrate is out of scope for 1.0, deferred behind the internal storage seam (an internal swap when it lands, not an API change). See the ROADMAP.

Where It Fits

iqdb-persist is the embedded persistence crate of the iQDB family. It builds on:

  • iqdb-types — core vocabulary (DistanceMetric, IqdbError)
  • iqdb-index — the Index / IndexCore traits it wraps as persistable

Snapshot file I/O goes through a tiny internal Storage seam so the future storage-io substrate (the fsys-rs rename) can drop in unchanged; today it ships one impl over std::fs, and the WAL appends through its own std::fs handle. Adopting storage-io is deferred until that rename lands — an internal swap behind the seam, not an API change.

Contributing

See dev/DIRECTIVES.md for engineering standards and the definition of done. Before a PR: cargo fmt --all, cargo clippy --all-targets --all-features -- -D warnings, and cargo test --all-features must be clean.