Crate iqdb_persist 

iqdb-persist

On-disk persistence for the iQDB vector database. Provides atomic snapshot save / load with a versioned file header and a CRC32 integrity check, generic over any type implementing iqdb_index::Index.

Tiered API

• Tier 1 — the lazy path. PersistConfig::new plus PersistedIndex::open_with / PersistedIndex::save / PersistedIndex::load cover the whole common case — wrap an index, save it, load it back — with no builder and no generics to name beyond the index type itself. With the WAL on, PersistedIndex::insert / PersistedIndex::delete / PersistedIndex::checkpoint add durable, crash-recoverable mutation.
• Tier 2 — the configured path. The PersistConfig fields (fsync_policy, compression, wal_enabled) tune durability and on-disk size.
• Tier 3 — the trait seam. An index opts into persistence by implementing Persistable; everything in Tier 1 and Tier 2 then works against it unchanged.

Surface

• Persistable — the trait an index implements. Two methods, save_to(&mut dyn Write) and load_from(&mut dyn Read) -> Result<Self>, plus a stable INDEX_TYPE tag. The impl serializes only the index’s self-contained payload; the file header, CRC32, and atomic write are added by PersistedIndex around it.
• PersistedIndex — wraps an I: Index + Persistable. Two honest constructors: open_with wraps an in-memory index for later save; load reconstructs an index from disk and errors if the file does not exist.
• FileHeader + MAGIC + CURRENT_VERSION — the wire format.
• PersistConfig / FsyncPolicy / Compression — configuration. wal_enabled = true turns on the write-ahead log (v0.3); Compression::Zstd|Lz4 compress the snapshot payload (v0.4, behind the zstd / lz4 cargo features — selecting a scheme whose feature is off yields PersistError::Unsupported).
• PersistError — #[non_exhaustive] and error_forge::ForgeError- integrated.

Three guards

1. The trait impl writes / reads only the index’s self-contained payload. Framing (header + CRC32) lives in PersistedIndex.
2. This crate stays generic over I — it never names a concrete index. The index_type → concrete-type registry that Database::open needs lives in the umbrella iqdb crate.
3. Tests use a tiny in-crate mock Persistable; iqdb-persist never dev-deps a concrete index crate.

Scope

Stable as of v1.0.0. The full surface — atomic snapshots + CRC32, the write-ahead log with replay and crash recovery, and optional Zstd / LZ4 snapshot compression — is complete, the parse/recovery paths are adversarially hardened, and the public API and on-disk format are frozen under the SemVer 1.x guarantee (no breaking changes before 2.0). The external storage-io substrate is deferred behind the internal storage seam and is out of scope for 1.0 (see dev/ROADMAP.md). See CHANGELOG.md.

Example

use iqdb_persist::{FileHeader, CURRENT_VERSION, MAGIC};
use iqdb_types::DistanceMetric;

// A header is just data; tools can inspect a snapshot file without
// loading the index it carries.
let header = FileHeader {
    magic: MAGIC,
    version: CURRENT_VERSION,
    index_type: "flat".to_string(),
    dim: 128,
    metric: DistanceMetric::Cosine,
    n_vectors: 1_000,
    crc32: 0,
};
assert_eq!(header.version, 2);
assert_eq!(&header.magic, b"IQDBPRST");

Re-exports

pub use crate::format::CURRENT_VERSION;

pub use crate::format::FileHeader;

pub use crate::format::MAGIC;

Modules

checksum

CRC32 helpers for snapshot integrity.

format

The on-disk file header and wire format.

Structs

PersistConfig

Configuration for crate::PersistedIndex.

PersistedIndex

A snapshot-persistent wrapper around an in-memory index.

Enums

Compression

Compression applied to the snapshot payload on save (v0.4+).

FsyncPolicy

How aggressively the persistence layer fsyncs to durable storage.

PersistError

An error from an iqdb-persist save, load, or format operation.

Constants

VERSION

The version of this crate, taken from Cargo.toml at compile time.

Traits

Persistable

An index that can be written to and read from a byte stream.

Type Aliases

Result

A specialized Result whose error is PersistError.