Struct WombatKVKvStore 

pub struct WombatKVKvStore<S: ObjectStore> { /* private fields */ }

Embeddable KV cache store.

Wraps a FoyerHybridCache for hot-path lookups and an ObjectStore (typically S3 / MinIO) for durability. Designed to be embedded in an inference engine binary (e.g. vllm.rs) without dragging in any additional async runtime.

Implementations

impl<S: ObjectStore> WombatKVKvStore<S>

pub fn new(config: EmbedConfig, object_store: S) -> Result<Self, EmbedError>

Build a new store. The foyer cache is created up front; the object store handle is passed in so callers can configure S3 credentials or supply an in-memory backend for tests.

pub fn with_foyer( foyer: Arc<FoyerHybridCache>, object_store: S, s3_prefix: impl Into<String>, write_through_s3: bool, ) -> Result<Self, EmbedError>

Build a store reusing an already-opened foyer instance. Useful when the engine wants to share one foyer across multiple stores.

pub fn compression(&self) -> BlockCompressionConfig

Expose the block-compression policy currently in force. Test helper, production code reads from the put/get paths instead.

pub fn metadata_index(&self) -> Arc<InMemoryMetadataIndex>

Expose the metadata index. Callers (e.g. the FFI Handle on startup) can call bootstrap_world_knowledge to populate it from S3, or query it directly for chain-aware lookups.

pub fn bootstrap_world_knowledge( &self, namespace: &str, ) -> Result<usize, EmbedError>

World-knowledge bootstrap (RFC 0008 §5): walk the S3 prefix for namespace, read each manifest, decode the chunk-hash chain, populate the in-memory metadata index. After this returns, the puffer “knows what’s in the bucket” without per-request S3 round-trips.

Cost is O(M) S3 GETs where M = manifest count. Each manifest is tiny (~800 bytes for 23 chunks). For 100 cached prompts: ~100 small GETs ≈ 1-2 s on local MinIO. Run at startup behind an env gate; not on the request hot path.

Pagination: the underlying ObjectStore::list_prefix is responsible for exhausting S3’s 1000-keys-per-page continuation loop and returning every match, see the S3ObjectStore::list_prefix impl, which iterates the rust-s3 Vec<ListBucketResult> already returned in fully-paged form. We add a defensive log + warn here so an unexpectedly large bucket surfaces in operator logs instead of silently bottling up: real production buckets above BOOTSTRAP_KEY_LIMIT_WARN keys deserve an operator look (likely a stale-data sweep is overdue), and above BOOTSTRAP_KEY_LIMIT_HARD we still process them but emit an explicit “exceeded” event so the caller can decide to bound the work.

pub fn bootstrap_from_slatedb( &self, slatedb_index: &SlateDbMetadataIndex, ) -> Result<usize, EmbedError>

L1 bootstrap from SlateDbMetadataIndex (RFC 0008 §5 fast path).

Snapshots all (hash, meta) pairs from the persistent SlateDB-backed index and bulk-loads them into the in-memory metadata_index. On a fresh process this lets us rehydrate “what’s in the world” in milliseconds, one local SlateDB scan vs the O(M) S3 GETs the S3-based bootstrap_world_knowledge would issue.

Idempotent: InMemoryMetadataIndex::bulk_load skips already-present hashes, so a second call with the same SlateDB returns the same loaded count and does not clobber any in-memory access stamps.

Returns the number of entries pulled out of SlateDB (this is the SlateDB row count, not the net-new RAM-index inserts, by design, mirroring bootstrap_world_knowledge’s “blocks loaded” semantic).

pub fn object_key(&self, namespace: &str, key: &str) -> String

Compose the S3 object key for a given namespace + cache key. Layout: {s3_prefix}/{namespace}/{key}. Namespace and key MUST be callers’ responsibility to keep filesystem-safe.

pub fn put_kv( &self, namespace: &str, key: &str, payload: Bytes, ) -> Result<(), EmbedError>

Write a payload through both tiers.

Always inserts into foyer synchronously. When write_through_s3 is true the call also blocks on the S3 PUT and surfaces any error; when false the S3 write is best-effort and only logs on failure.

pub fn put_kv_async_s3( this: Arc<Self>, namespace: &str, key: &str, payload: Bytes, )

Foyer-sync write, S3 write spawned on a detached thread.

Returns as soon as foyer has the bytes, typically within a few hundred microseconds, so callers (e.g. ds4 right after Metal prefill) can move on to decode while the slow ObjectStore PUT happens off-thread. Foyer is updated atomically inside this call, so any subsequent get_kv against the same key (from the same process) will hit foyer-RAM, not race the in-flight S3 write.

Trade-offs vs the synchronous Self::put_kv:

• Cross-process GET against the same key racing the background S3 PUT will miss in S3 until the write completes (foyer is process-local). Not safe for cross-engine sharing under that pattern; safe for the single-client-per-host shape ds4 uses.
• Spawn failure (rare) loses the S3 write entirely; foyer still has it. We log to stderr and otherwise swallow because the caller (ds4) has already moved on.
• One detached thread per call. For sustained high-rate puts a bounded executor would be safer; the current ds4 pattern is one put per chat completion (a few per minute at most), so the thread cost is negligible.

pub fn get_kv( &self, namespace: &str, key: &str, ) -> Result<GetOutcome, EmbedError>

Look up a payload, foyer-first, S3-fallback. On S3 hit the value is promoted into foyer so subsequent calls hit the warm path.

Each load emits a [MyelonInstr] JSON line attributing latency to the actual tier that served the hit (foyer RAM, foyer SSD, S3, or miss). Critical for diagnosing the “blob too big for RAM” pattern, e.g. qwen3’s pre-allocated 4.7 GiB KV cache against a 2 GiB RAM budget always streams from SSD, which a single LoadFoyer bucket can’t tell apart from a fast in-memory hit.

pub fn exists_kv(&self, namespace: &str, key: &str) -> Result<bool, EmbedError>

Check for a key without materializing the payload.

This is intentionally separate from get_kv: vLLM’s scheduler calls exists while deciding whether a prefix can be loaded. If exists falls through to a full GET, large KV payloads traverse the daemon once during lookup and then again during the real load.

pub fn list_namespace(&self, namespace: &str) -> Result<Vec<String>, EmbedError>

List keys for a namespace as their S3 object keys.

pub fn list_kv_keys(&self, namespace: &str) -> Result<Vec<String>, EmbedError>

List keys for a namespace relative to that namespace.

pub fn restore_from_s3(&self, namespace: &str) -> Result<usize, EmbedError>

Rehydrate foyer from S3. Useful at engine startup so the warm tier is primed with whatever survived the previous process. Returns the number of keys restored.

pub fn delete_kv(&self, namespace: &str, key: &str) -> Result<bool, EmbedError>

Delete one block from the object store (and best-effort from the flat tier). Used by the LRU eviction worker (RFC 0009 §4); not called on the hot path.

Returns true iff the object store reported a delete. Foyer is left untouched: foyer::HybridCache does not expose a single- key remove on its public API in the version we pin; the bytes will age out naturally as new inserts evict them. The metadata index (the authority for the per-namespace byte budget) is updated separately by the worker so the budget accounting stays correct even while foyer still holds the bytes briefly.

pub fn clear_foyer(&self)

Drop foyer state. Object-store data is unaffected.

pub fn clear_flat_cache(&self)

Drop flat-file blob-cache state. Object-store data is unaffected.

The flat tier sits in front of foyer on the get path (see commit 2ca65cb). Tests that want to exercise the object-store fallback must clear both tiers, otherwise the flat hit short-circuits before foyer is consulted. Use Self::clear_foyer for the foyer tier.

pub fn foyer(&self) -> &Arc<FoyerHybridCache>

Borrow the underlying foyer cache (e.g. for stats or sharing).

pub fn object_store(&self) -> &S

Borrow the underlying object store (e.g. for direct list/delete).

pub fn start_eviction_worker( self: &Arc<Self>, config: LruConfig, slatedb: Option<Arc<SlateDbMetadataIndex>>, ) -> LruEvictionWorker

Spawn the background block-prefetch worker (RFC 0008 §6).

The worker periodically scores the metadata index per the recency / chain-head / model-affinity heuristic and (v2) issues get_kv GETs for the top-K candidates, materializing the payloads into the local flat tier so subsequent requests hit warm. See crate::block_prefetch for the heuristic.

Behavior is selected at construction time by the WMBT_KV_PREFETCH_DRY_RUN=1 env: when set, the worker scores and logs only (the v1 escape hatch). Default is v2.

Holds an Arc to self via the PrefetchFetcher impl, so the worker can issue GETs. Dropping the returned worker signals stop and joins the thread. Spawn the background LRU eviction worker (RFC 0009 §4).

The worker periodically scans the in-memory metadata index for the configured namespace, sums payload_bytes, and when the sum exceeds LruConfig::namespace_max_bytes, evicts the oldest entries (by last_access_ns) until the budget has a 10% headroom.

Caller is responsible for handing in the optional SlateDB index so the L1 persistence is kept in sync. If None, only the L0 in-memory index and the object store are touched.

Dropping the returned worker signals stop and joins the thread.

pub fn start_prefetcher( self: &Arc<Self>, config: PrefetchConfig, ) -> PrefetchWorker