Module result_cache 

Persistent result-cache publication (TODO §2).

Goal: result-cache inserts must do no serialization, encryption, write, sync, or rename on the query thread. The query thread enqueues a cheap PersistableEntry (Arc-shared rows + scalars) onto a bounded coalescing pending map; a background worker drains the map and performs the actual disk write through a PersistentCacheIo abstraction.

Layout: frame format + encryption helpers first, then the writer queue, then the PersistentCacheIo trait + production RealPersistentCacheIo, then the test module, then the worker spawn function. Clippy’s items_after_test_module lint flags this ordering; we keep it intentionally because tests sit alongside the writer queue they cover and the worker lives at the bottom because it depends on the writer.

Frame format (stable across versions, written in little-endian order):

[magic: 4B = b"MLCP"]
[format_version: u16 LE]
[reserved: u16 LE = 0]
[table_id: u64 LE]
[schema_id: u64 LE]
[run_generation: u64 LE]
[cache_key: u64 LE]
[entry_generation: u64 LE]
[payload_len: u32 LE]
[payload: u8 × payload_len]            # bincode(SerializedEntry) or
                                       #   encrypted bincode(...)
[crc32c: u32 LE]                       # over the prefix + payload_len + payload

A loader rejects frames whose magic, version, table_id, schema_id, or run_generation does not match the open table, or whose CRC does not match the stored bytes. Stale cache_keys (already invalidated) are also rejected on reopen by the worker, which compares its queued entry’s per-key generation against the latest clear generation captured on enqueue.

Structs

CachePersistError

Error returned by encode_persisted_entry when the payload cannot be framed (e.g. encryption failure). The caller treats this as a transient per-op failure and increments the persist error counter.

DrainedOp

A queued op + the key/generation snapshot the worker must verify before publishing. The worker re-reads the queue’s per-key generation under the lock and rejects the op if it has been superseded.

IoError

Errors raised by PersistentCacheIo. Stored as opaque strings so the worker can record them on its metrics without converting every I/O error into a MongrelError.

PendingCacheState

State of the pending-operation map. The worker publishes stores only when the queued op’s per-key generation still matches the latest generation seen for that key — a newer invalidate cannot be silently overwritten.

PersistContext

Context passed to encode_persisted_entry and used by the worker to validate the on-disk frame matches what the queue said should be written.

PersistableEntry

Cheap-to-share body for a queued store. The query thread constructs this with raw data (Arc-shared rows, columns, footprint) plus a lazy PersistableEntry::payload_factory. The factory captures the raw data and produces the bincode-serialized bytes only when the worker invokes it — keeping the query thread free of both serialization and I/O.

PersistedFrame

Decoded frame ready for the table-side deserializer.

PersistedHeader

Header of a persisted frame. Read from disk and re-validated before any payload is decoded.

PersistentCacheIdentity

Stable identity bound to a persisted cache frame (REM-002 §18.1). The loader compares the on-disk frame header against this identity and rejects the entry on any mismatch. The logical_generation is the durable table epoch (Table::current_epoch().0) — never a process-local counter — so a valid frame from an older logical state cannot be served after a commit has advanced the epoch. Persistent cache frames are valid only for the exact logical table generation: a frame from a future generation (backup/PITR restore, manual _rcache copy, partial filesystem rollback) can contain rows that do not exist in the restored table and is rejected with the same GenerationMismatch as an older frame.

PersistentResultCacheWriter

The PersistentResultCacheWriter. The query thread calls PersistentResultCacheWriter::enqueue_store / [enqueue_remove] / enqueue_clear; the worker calls [drain_one] in a loop and re-validates the op before publishing.

RealPersistentCacheIo

Real implementation of PersistentCacheIo writing to a temp directory. Each call performs temp write + fsync + rename + parent dir sync, which is the same crash-safe contract that production code already requires.

WorkerConfig

Configuration for spawning a spawn_persistent_cache_worker.

WriterLimits

Capacity knobs for the writer. Held behind Arc so the worker can read without taking the queue lock.

WriterStalenessGuard

A StalenessGuard backed by the writer’s own queue. The implementation locks the queue, checks the per-key generation and the clear generation, and returns whether the op is still the latest. Cheap when uncontended.

Enums

CacheLoadRejection

Why a persisted frame was rejected by decode_persisted_entry. The caller maps each variant to a rejection metric / log line.

DrainOutcome

Summary returned by the worker for one drained op. The writer uses it to maintain the stale-store / errors / abandoned counters.

IoErrorKind

Distinguishes “retryable” from “fatal” I/O errors. Currently a marker; future changes may add fallback paths.

PendingCacheOp

A pending operation for one cache key. Later writes supersede earlier writes; a Remove supersedes a Store. Clear invalidates every older op under the same generation.

PersistenceDisabledReason

Why persistent publication is unavailable for a result cache. Each variant carries a stable PersistenceDisabledReason::label used in query traces and diagnostics.

PersistentPublicationState

Explicit persistent-publication state of a result cache. Replaces the implicit Option<writer> semantics: the query path either enqueues onto the background writer or skips persistent publication (keeping the in-memory entry and incrementing the skip metric). There is no query-thread synchronous write fallback — the persistent cache is disposable optimization state and its loss degrades to a recompute.

Constants

FRAME_FORMAT_VERSION

On-disk format version. Bumped on any layout change.

FRAME_MAGIC

Magic identifying a persistent-cache frame: “MLCP” (MongreLDb Cache Page).

Traits

PersistentCacheIo

I/O abstraction the worker thread uses to publish and reload persistent cache frames. Production code uses RealPersistentCacheIo; tests inject recording/blocking/failing variants.

StalenessGuard

Hook for the worker to validate that a drained op is still current. The production implementation reads the writer’s per-key generation under the queue lock; tests can use a custom implementation to simulate races.

Functions

decode_frame

Decode a frame buffer. Returns None on any structural or CRC failure.

decode_persisted_entry

Decode + validate a frame produced by encode_persisted_entry. On rejection, the caller removes the on-disk file (the cache is disposable). On success, the plaintext payload is returned so the engine can perform the bincode-specific SerializedEntry deserialize.

decrypt_payload

Inverse of encrypt_payload: returns None on key mismatch or tag failure.

encode_frame

Encode a PersistedFrame into a self-describing, checksummed byte buffer. The CRC32C is computed over the fixed prefix and the payload bytes.

encode_persisted_entry

Frame an already-bincode-serialized payload using the MLCP layout. Both the async worker and the synchronous maintenance helper call this — the on-disk format is unified across both paths (REM-002 §18.2). The payload is encrypted (or left as plaintext) before framing; the magic / version / table / schema / generation / key / payload_len / CRC32C layout is the same for both encrypted and plaintext frames.

encrypt_payload

Helper used by the worker to encrypt a payload before it is written. Returns the plaintext bytes untouched when no DEK is present.

read_header_only

Quick header-only peek: returns Some(PersistedHeader) if the buffer is structurally valid and the magic/version match, without verifying the CRC or copying the payload. The CRC is still verified — a fast reject path would re-parse the entire buffer, so we simply delegate to decode_frame.

real_io_final_path

Test-only: read a path under a RealPersistentCacheIo for assertion purposes. The path uses a hex key naming scheme identical to the I/O implementation, so the same load/write_atomic round-trips apply.

recv_completion_with_timeout

Try to receive a worker completion signal with a bounded timeout. Used by shutdown_persistent_cache(deadline) so the engine returns even if the worker is blocked in filesystem I/O. Returns true if the signal was received (worker has exited), false on timeout.

spawn_persistent_cache_worker

Spawn the persistent-cache publication worker. The worker thread exits when the writer is shut down AND the queue is empty.