iqdb-persist 0.2.0

Atomic snapshot persistence with versioned headers and CRC32 integrity for iQDB indexes - part of the iQDB family.
Documentation
  • Atomic snapshot save/load — write-to-temp + fsync + atomic rename + directory fsync; an interrupted write never corrupts an existing good file.
  • Versioned header — magic bytes, format version, index-type tag, dim, metric, vector count. All sizes are fixed-width little-endian u64, so a file is portable across 32- and 64-bit hosts.
  • CRC32 integrity — computed over the payload; a single-bit flip surfaces as ChecksumMismatch on load, never a panic or a silently-wrong result.
  • Generic over the index — PersistedIndex<I: Index + Persistable> wraps any concrete index; the framing lives here, the payload bytes live in the index's Persistable impl.

Installation

[dependencies]
iqdb-persist = "0.2"
iqdb-index   = "1.0"
iqdb-types   = "1.0"

Quick start

use iqdb_persist::{PersistConfig, PersistedIndex};

// `MyIndex: iqdb_index::Index + iqdb_persist::Persistable`
let cfg = PersistConfig::new("/var/lib/app/index.iqdb");

// Wrap an in-memory index and save it atomically.
let wrapped = PersistedIndex::open_with(my_index, cfg.clone())?;
wrapped.save()?;

// Later — reconstruct it from disk (verifies magic, version, type, CRC32).
let restored: PersistedIndex<MyIndex> = PersistedIndex::load(cfg)?;
let index = restored.index();

An index opts in by implementing the two-method Persistable trait. A complete, runnable version lives in examples/save_and_load.rs — run it with cargo run --example save_and_load. Full reference: docs/API.md.

Status

This is v0.2.0: atomic snapshot save/load, the versioned header, and CRC32 integrity are implemented and tested. WAL append/replay (v0.3), Zstd/LZ4 compression (v0.4), and the external storage-io substrate (v0.5+) land in later phases per the ROADMAP. The PersistConfig knobs for those features already exist and reject cleanly (PersistError::Unsupported) until they ship.

Where It Fits

iqdb-persist is the embedded persistence crate of the iQDB family. It builds on:

  • iqdb-types — core vocabulary (DistanceMetric, IqdbError)
  • iqdb-index — the Index / IndexCore traits it wraps as persistable

All file I/O goes through a tiny internal Storage seam so the future storage-io substrate (the fsys-rs rename) can drop in unchanged; v0.2 ships one impl over std::fs.

Contributing

See dev/DIRECTIVES.md for engineering standards and the definition of done. Before a PR: cargo fmt --all, cargo clippy --all-targets --all-features -- -D warnings, and cargo test --all-features must be clean.