iqdb-persist 0.3.0

Atomic snapshot persistence with versioned headers and CRC32 integrity for iQDB indexes - part of the iQDB family.
Documentation
  • Atomic snapshot save/load — write-to-temp + fsync + atomic rename + directory fsync; an interrupted write never corrupts an existing good file.
  • Versioned header — magic bytes, format version, index-type tag, dim, metric, vector count. All sizes are fixed-width little-endian u64, so a file is portable across 32- and 64-bit hosts.
  • CRC32 integrity — computed over the payload; a single-bit flip surfaces as ChecksumMismatch on load, never a panic or a silently-wrong result.
  • Write-ahead log & crash recovery — with wal_enabled, every insert / delete is logged and fsynced before it touches memory, then replayed onto the snapshot on load. A crash mid-append leaves a torn tail that recovery detects and discards.
  • Generic over the index — PersistedIndex<I: Index + Persistable> wraps any concrete index; the framing lives here, the payload bytes live in the index's Persistable impl.

Installation

[dependencies]
iqdb-persist = "0.3"
iqdb-index   = "1.0"
iqdb-types   = "1.0"

Quick start

use iqdb_persist::{PersistConfig, PersistedIndex};

// `MyIndex: iqdb_index::Index + iqdb_persist::Persistable`
let cfg = PersistConfig::new("/var/lib/app/index.iqdb");

// Wrap an in-memory index and save it atomically.
let wrapped = PersistedIndex::open_with(my_index, cfg.clone())?;
wrapped.save()?;

// Later — reconstruct it from disk (verifies magic, version, type, CRC32).
let restored: PersistedIndex<MyIndex> = PersistedIndex::load(cfg)?;
let index = restored.index();

For durable, crash-recoverable mutation, set cfg.wal_enabled = true and mutate through the wrapper — each op is logged before it is applied, and checkpoint() folds the log back into a fresh snapshot:

let mut db = PersistedIndex::open_with(my_index, cfg.clone())?; // writes base snapshot + opens WAL
db.insert(id, vector, metadata)?;   // logged + fsynced, then applied
db.delete(&other_id)?;
db.checkpoint()?;                    // snapshot the state, truncate the WAL
// after a crash: PersistedIndex::load(cfg) replays the WAL onto the snapshot

An index opts in by implementing the two-method Persistable trait. A complete, runnable version lives in examples/save_and_load.rs — run it with cargo run --example save_and_load. Full reference: docs/API.md.

Status

This is v0.3.0: atomic snapshot save/load + versioned header + CRC32 (v0.2) and the write-ahead log with replay and crash recovery (v0.3) are implemented and tested. Zstd/LZ4 compression (v0.4) and the external storage-io substrate (v0.5+) land in later phases per the ROADMAP. The compression knob already exists and rejects cleanly (PersistError::Unsupported) until it ships.

Where It Fits

iqdb-persist is the embedded persistence crate of the iQDB family. It builds on:

  • iqdb-types — core vocabulary (DistanceMetric, IqdbError)
  • iqdb-index — the Index / IndexCore traits it wraps as persistable

Snapshot file I/O goes through a tiny internal Storage seam so the future storage-io substrate (the fsys-rs rename) can drop in unchanged; v0.3 ships one impl over std::fs, and the WAL appends through its own std::fs handle until that substrate lands in v0.5.

Contributing

See dev/DIRECTIVES.md for engineering standards and the definition of done. Before a PR: cargo fmt --all, cargo clippy --all-targets --all-features -- -D warnings, and cargo test --all-features must be clean.