Skip to main content

Crate kevy_vlog

Crate kevy_vlog 

Source
Expand description

kevy-vlog — the disposable value log under transparent tiering.

An append-only spill area for cold VALUES: keys and metadata stay in RAM; a demoted value’s bytes live here and are read back with one read_at. Three properties are load-bearing:

  • Disposable. The vlog is NOT durability — the AOF remains the sole durable truth. Vlog::open deletes any leftover vlog files; every boot starts empty and refills during replay. There is deliberately no fsync, no recovery scan, no torn-write healing: a record that fails its CRC on read-back is a bug in this process, not corruption to repair, and surfaces as an error.
  • Pinned readers. A reader that must survive compaction (a snapshot view, an AOF rewrite, a replication ship) holds an Arc<VlogFile> from Vlog::pin. A compacted file is unlinked only when the last pin drops (VlogFile’s Drop), so a pinned view’s offsets stay valid for its whole life. All file IO is positional (read_at/write_at, &File), so pinned reads are thread-safe.
  • Owner-driven compaction. Records carry their key, so a file is self-describing; but only the owner (the store’s cold refs) knows which records are live. Vlog::compact_below scans, asks the owner (CompactOwner::is_live), re-appends survivors, and hands the new ref back (CompactOwner::moved). Every compaction bumps Vlog::epoch — the O(1) “did my ColdRef move?” check.
use kevy_vlog::Vlog;

let dir = kevy_tmpdir::TmpDir::new("vlog-crate-doc");
let mut log = Vlog::open(dir.path(), 1 << 20)?;
let at = log.append(b"user:1", b"a cold value")?;
assert_eq!(log.read(at)?, (b"user:1".to_vec(), b"a cold value".to_vec()));
// the owner overwrote or deleted the key: its bytes are now dead weight
log.note_dead(at);
assert_eq!(log.stats().live_bytes, 0);

Record layout: [body_len u32-LE][crc32c u32-LE][key_len u32-LE][key] [payload], CRC over the body (everything after the 8-byte header). A VlogRef names (file_id, header offset, body_len).

The payload is a kevy-compress frame. Values from one keyspace land together, so the file is a corpus: each file carries a dictionary trained on a sample of its predecessor’s raw payloads (same population, already real bytes — no cold-start window), and records encode against it on append, reaching the cross-value redundancy a per-datum compressor cannot see. The frame lives INSIDE the body, so on-disk framing does not move and the CRC covers exactly the stored bytes. The dictionary lives in the VlogFile it serves and dies with it — the vlog is disposable, so its dictionaries are too, and no format-compatibility burden ever exists. Incompressible payloads store raw inside the frame (never expanded past the 6-byte frame header). Every public item here is documented, and the lint keeps it that way: kevy-vlog is the value log, and [workspace.lints.rust] warnings = "deny" turns a new gap into a compile error rather than a number that drifts. Closed from 65 sites (store) and 7 (vlog) in v6.

Structs§

CompressionStats
The compression accounting over every file the log still holds, dead records included (the same population as crate::VlogStats::bytes).
Vlog
One shard’s value log. Single owner (&mut appends, the shard thread); concurrent readers go through Vlog::pin.
VlogFile
One log file. Shared via Arc: the Vlog holds one, and pinned readers hold more. When compaction retires the file it sets delete_on_drop; the underlying file is unlinked by whichever holder drops last — that is the entire pin protocol.
VlogRef
The address of one spilled record — what a cold stub holds.
VlogStats
Aggregate gauges for INFO (vlog_size / vlog_dead_bytes feeders).

Constants§

DEFAULT_ROTATE_BYTES
Default rotation threshold for the active file (RFC §7: 256 MiB).

Traits§

CompactOwner
Owner callbacks for Vlog::compact_step and Vlog::compact_below — one object, one borrow, so the store can capture its map mutably across both phases.

Functions§

verify_image
Verify a raw record image (the disk_len() bytes at r.offset) and split it into (key, payload). This is the completion half of a batched read: the io_uring path fetches images concurrently and runs each through here; VlogFile::read is exactly one fetch + this. A length or CRC mismatch is InvalidData — this process wrote the record this boot, so a bad image is a bug, never corruption to heal.