Expand description
kevy-vlog — the disposable value log under transparent tiering.
An append-only spill area for cold VALUES: keys and metadata stay
in RAM; a demoted value’s bytes live here and are read back with
one read_at.
Three properties are load-bearing:
- Disposable. The vlog is NOT durability — the AOF remains the sole
durable truth.
Vlog::opendeletes any leftover vlog files; every boot starts empty and refills during replay. There is deliberately no fsync, no recovery scan, no torn-write healing: a record that fails its CRC on read-back is a bug in this process, not corruption to repair, and surfaces as an error. - Pinned readers. A reader that must survive compaction (a snapshot
view, an AOF rewrite, a replication ship) holds an
Arc<VlogFile>fromVlog::pin. A compacted file is unlinked only when the last pin drops (VlogFile’sDrop), so a pinned view’s offsets stay valid for its whole life. All file IO is positional (read_at/write_at,&File), so pinned reads are thread-safe. - Owner-driven compaction. Records carry their key, so a file is
self-describing; but only the owner (the store’s cold refs) knows
which records are live.
Vlog::compact_belowscans, asks the owner (CompactOwner::is_live), re-appends survivors, and hands the new ref back (CompactOwner::moved). Every compaction bumpsVlog::epoch— the O(1) “did my ColdRef move?” check.
use kevy_vlog::Vlog;
let dir = kevy_tmpdir::TmpDir::new("vlog-crate-doc");
let mut log = Vlog::open(dir.path(), 1 << 20)?;
let at = log.append(b"user:1", b"a cold value")?;
assert_eq!(log.read(at)?, (b"user:1".to_vec(), b"a cold value".to_vec()));
// the owner overwrote or deleted the key: its bytes are now dead weight
log.note_dead(at);
assert_eq!(log.stats().live_bytes, 0);Record layout: [body_len u32-LE][crc32c u32-LE][key_len u32-LE][key] [payload], CRC over the body (everything after the 8-byte header).
A VlogRef names (file_id, header offset, body_len).
The payload is a kevy-compress frame. Values from one keyspace
land together, so the file is a corpus: each file carries a
dictionary trained on a sample of its predecessor’s raw payloads
(same population, already real bytes — no cold-start window), and
records encode against it on append, reaching the cross-value
redundancy a per-datum compressor cannot see. The frame lives
INSIDE the body, so on-disk framing does not move and the CRC
covers exactly the stored bytes. The dictionary lives in the
VlogFile it serves and dies with it — the vlog is disposable,
so its dictionaries are too, and no format-compatibility burden
ever exists. Incompressible payloads store raw inside the frame
(never expanded past the 6-byte frame header).
Every public item here is documented, and the lint keeps it that
way: kevy-vlog is the value log, and [workspace.lints.rust] warnings = "deny"
turns a new gap into a compile error rather than a number that
drifts. Closed from 65 sites (store) and 7 (vlog) in v6.
Structs§
- Compression
Stats - The compression accounting over every file the log still holds, dead
records included (the same population as
crate::VlogStats::bytes). - Vlog
- One shard’s value log. Single owner (
&mutappends, the shard thread); concurrent readers go throughVlog::pin. - Vlog
File - One log file. Shared via
Arc: theVlogholds one, and pinned readers hold more. When compaction retires the file it setsdelete_on_drop; the underlying file is unlinked by whichever holder drops last — that is the entire pin protocol. - VlogRef
- The address of one spilled record — what a cold stub holds.
- Vlog
Stats - Aggregate gauges for INFO (
vlog_size/vlog_dead_bytesfeeders).
Constants§
- DEFAULT_
ROTATE_ BYTES - Default rotation threshold for the active file (RFC §7: 256 MiB).
Traits§
- Compact
Owner - Owner callbacks for
Vlog::compact_stepandVlog::compact_below— one object, one borrow, so the store can capture its map mutably across both phases.
Functions§
- verify_
image - Verify a raw record image (the
disk_len()bytes atr.offset) and split it into(key, payload). This is the completion half of a batched read: the io_uring path fetches images concurrently and runs each through here;VlogFile::readis exactly one fetch + this. A length or CRC mismatch isInvalidData— this process wrote the record this boot, so a bad image is a bug, never corruption to heal.