Module compression 

Transparent block-storage compression for WombatKV.

Real product feature: KV blocks emitted by inference engines are highly compressible, large stretches of near-zero values dominate later attention layers, so transparent zstd shrinks S3 storage cost ~3-4× on typical bench artifacts without changing the C ABI or the BlockMeta schema.

Wire format

Compressed blobs carry a 10-byte header:

    0       4       5       6              10
    +-------+-------+-------+--------------+--------------------+
    | "WBZ1"| algo  | level | u32 raw_len  | compressed payload |
    +-------+-------+-------+--------------+--------------------+
    | magic | u8    | u8    | LE           |                    |
    +-------+-------+-------+--------------+--------------------+

• Magic b"WBZ1" (WombatKV blob zstd v1) is the only signature a decoder needs to detect compression. Anything else is treated as raw uncompressed bytes, old buckets stay readable verbatim.
• algo = 1 for zstd. Reserved 2 = lz4 (future). 0 = none (header only ever used by tests; production never writes a “compressed with none” blob).
• level = the zstd level the producer used. Stored for observability; not consulted on decode.
• raw_len = uncompressed size (u32). Caps a single block at 4 GiB, which is far above any realistic KV block.

Layering

Compression is applied at the object-store boundary inside put_kv / get_kv. The in-memory flat-file and foyer tiers keep uncompressed bytes, they are warm-read caches, decoding once on the cold-from-S3 path is cheap, and skipping it on every cache hit keeps the warm TTFT story intact.

Compatibility

Mixed-state buckets are first-class: every read calls decode_if_compressed, which inspects the magic and falls through to a no-copy Cow::Borrowed when the header is absent or corrupt.

Structs

BlockCompressionConfig

Compression policy resolved at handle construction time.

Enums

CompressAlgo

Compression algorithm tag. Stored as a u8 in the wire header.

CompressionError

Compression pipeline failures.

Constants

COMPRESS_HEADER_SIZE

Total header size: 4 (magic) + 1 (algo) + 1 (level) + 4 (raw_len) = 10.

COMPRESS_MAGIC

Magic prefix for a compressed WombatKV block. ASCII so logs are human-readable.

Functions

decode_if_compressed

Inspect bytes. If the magic header is present and decodes cleanly, return the decompressed payload as Cow::Owned. Otherwise return Cow::Borrowed(bytes) so the no-compression hot path stays allocation-free.

encode_with_header

Encode payload with the configured codec, prepending the 10-byte header. Returns the raw payload (no header) when cfg.algo is None so callers can blindly call this and get back-compat bytes for free.

has_magic

Quick magic check used by the put-path metrics emitter. Cheap enough to call on every blob.