Skip to main content

Module lru

wombatkv_node

Module lru 

Source
Expand description

Per-namespace LRU eviction worker (RFC 0009).

Production-safety story: WombatKV’s block storage in S3 grows unboundedly without a budget cap. This module periodically scans the in-memory MetadataIndex, sums payload_bytes per namespace, and when the sum exceeds the configured byte budget, evicts the oldest entries (by last_access_ns) until the namespace fits inside the budget with a 10% headroom.

§Env vars

  • WMBT_KV_NAMESPACE_MAX_BYTES=<N>, per-namespace byte budget. 0 (default) disables eviction entirely (safe default; existing deployments see no behavior change). 100 GB is the suggested production setting; a single ds4 model footprint fits in well under that.
  • WMBT_KV_DAEMON_EVICTION_INTERVAL_SECS=<N>, cycle interval, default 30 s.

§Race safety

The worker uses a compare-and-delete pattern against InMemoryMetadataIndex::remove_if_unchanged: each eviction candidate carries the last_access_ns snapshot taken at scoring time. If a concurrent get_and_touch raced the worker between snapshot and delete, the CAS fails and the worker silently skips that entry this cycle (revisits next pass). This avoids holding a cross-namespace tokio::sync::Mutex on the hot path: the eviction cost is paid only by the worker thread, the request path pays one extra BlockMeta comparison.

The CAS-failure path is observable in the per-cycle event as skipped_changed. A persistently high count indicates either (a) the namespace is genuinely hotter than the budget allows (raise the budget) or (b) the eviction interval is too long and many blocks are getting touched between scan and delete (shrink the interval).

§What gets deleted, by tier

Per evicted entry the worker:

  1. Removes from InMemoryMetadataIndex (CAS as above).
  2. Removes from SlateDbMetadataIndex (if opened by the caller). Best-effort: a SlateDB delete failure logs+continues; the L0 state is already consistent.
  3. Calls EvictionDeleter::delete_block(namespace, key) which routes through WombatKVKvStore::delete_kv: the object store delete + flat-tier unlink. Foyer is intentionally left to age out naturally (foyer::HybridCache does not expose a public single-key remove in our pinned version; the metadata index is the authoritative budget so this is correctness-safe).

Structs§

EvictionCycleOutcome
Per-cycle outcome counts. Surfaced via the [MyelonInstr] event emitted by default_emit.
LruConfig
Tuning knobs for the LRU eviction worker.
LruEvictionWorker
Owns a background thread that runs the eviction cycle.

Traits§

EvictionDeleter
Sync deleter surface implemented by WombatKVKvStore<S> so the algorithm crate can call delete without depending on the embed crate’s generic S: ObjectStore shape. Mirrors the PrefetchFetcher pattern in block_prefetch.rs.

Functions§

default_emit
Default emit: a [MyelonInstr] JSON line on stderr per cycle.
run_cycle
Run one eviction cycle synchronously. Exposed for unit tests so the cycle can run inline (no thread orchestration).
spawn_worker
Spawn the eviction worker. The returned worker holds the join handle and signals stop on drop. Returns immediately; the loop runs on the spawned thread.

Type Aliases§

EvictionEmit
Closure-based callback for the per-cycle outcome event.