Skip to main content

Module pack

mkit_core

Module pack 

Source
Expand description

Packfile writer / reader — conformant to docs/specs/SPEC-PACKFILE.md.

Layout (SPEC-PACKFILE §1, §2, §3, §8):

[4B  magic            "MKIT"]                       offset 0
[4B  version u32 LE  == 1 or 2]
[4B  entry_count u32 LE     ]
  for each entry:
    [u8  entry_type]           0x00 raw | 0x02 delta | 0x03 zstd-raw | 0x04 zstd-delta
    [u32 LE payload_len]                            length of payload only
    [payload_len bytes payload]
[32B trailer = BLAKE3 of all preceding bytes]

Entry types (SPEC-PACKFILE §3):

  • 0x00 raw — payload is a fully serialised mkit object.
  • 0x01 — RESERVED, MUST be rejected.
  • 0x02 delta — payload is [32B base_hash][SPEC-DELTA stream].
  • 0x03 zstd-raw — v2 only. payload is [4B uncompressed_len LE][zstd frame]; the frame decompresses to exactly what a 0x00 payload would be.
  • 0x04 zstd-delta — v2 only. payload is [32B base_hash][4B uncompressed_len LE][zstd frame]; base_hash stays uncompressed, the frame decompresses to exactly what a 0x02 entry’s post-base-hash bytes would be.

Version selection is writer policy, not caller policy (SPEC-PACKFILE §1): PackWriter emits version = 1 when the finished pack contains no 0x03/0x04 entries, and version = 2 the moment it contains at least one — even in an otherwise-mixed pack. 0x03/0x04 are illegal inside a version = 1 pack; a reader seeing one there rejects with InvalidEntryType exactly as it would for any other unrecognized type (SPEC-PACKFILE §3).

Compression is per-entry, not a whole-pack stream: every 0x03/0x04 entry carries its own independent zstd frame, so existing framing/caps/trailer semantics (§2, §5, §8) are unchanged and decompression memory is bounded to one entry at a time. Decoding a 0x03/0x04 entry is bomb-guarded: the claimed uncompressed_len is checked against MAX_RAW_OBJECT_SIZE before any decompression allocation, decompression itself is capacity-bounded to that claim, and the actual decompressed length is re-checked against the claim afterward (DecompressedSizeMismatch / DecompressedSizeOverCap).

Caps (SPEC-PACKFILE §5, unchanged by v2 — measured on the wire size; the decompressed-side cap above is separate and new):

  • entry_count <= 10_000_000
  • total payload_len sum <= 4 GiB

Delta-base ordering rule (SPEC-PACKFILE §4): every delta entry’s (0x02 or 0x04) base_hash MUST appear earlier in the same pack as a raw entry, OR already exist in the destination object store. 0x04’s base_hash is never compressed, so this never requires decompression to evaluate.

The pack key (SPEC-PACKFILE §7) is packs/<lower-hex BLAKE3 of entire pack>. The trailer is then redundant w.r.t. that key, but it lets a streaming reader detect bit-rot before the whole pack has been hashed end-to-end.

Structs§

PackReader
Streaming-style packfile reader. Verifies header, trailer, entry framing, and the base-before-delta ordering rule. Reconstructs delta targets and writes every resolved object to store.
PackWriter
Builds a packfile, enforcing entry/payload caps and streaming each pushed entry’s frame directly into the final output buffer as it arrives. Self::finish only patches the header’s entry count (unknown up front from a streaming writer) and appends the trailer — it never re-copies the pushed entries into a second, same-sized buffer (issue #647).
UnpackReport
Result of an unpack: which entries were stored, plus a count of delta resolutions vs raw writes. Useful for transport/CLI summaries.

Enums§

PackError
Packfile errors. Distinct from MkitError so callers can match on pack-specific failures (trailer mismatch, base-missing) without catching every object decode error.

Constants§

ENTRY_COUNT_OFFSET
Byte offset of the 4-byte entry_count field within the header — after the 4-byte magic and 4-byte version fields. Found hardcoded as the literal range 8..12 at four call sites during the epic-#634 code review; named here instead, consistent with this file’s existing HEADER_LEN/TRAILER_LEN convention.
ENTRY_FRAME_LEN
Per-entry framing overhead is [1B type][4B payload_len].
HEADER_LEN
Header is [4B magic][4B version][4B entry_count].
MAGIC
ASCII magic (“MKIT”) at the start of every pack, v1 or v2.
MAX_ENTRIES
Hard cap on entries (SPEC-PACKFILE §5).
MAX_TOTAL_PAYLOAD
Hard cap on the sum of payload bytes across all entries.
TRAILER_LEN
Trailer is a 32-byte raw BLAKE3 digest.
VERSION
Packfile version emitted when a pack contains no compressed (0x03/0x04) entries. Also the minimum version any reader accepts.
VERSION_OFFSET
Byte offset of the 4-byte version field within the header — right after the 4-byte magic. PackWriter::finish patches this once the final v1-vs-v2 decision is known (mirrors ENTRY_COUNT_OFFSET below).
VERSION_V2
Packfile version emitted the moment a pack contains at least one compressed (0x03/0x04) entry (SPEC-PACKFILE §1, §9). Readers accept both VERSION and VERSION_V2; only VERSION_V2 packs may contain 0x03/0x04 entries.

Functions§

delta_base_hashes
Collect the base_hash of every 0x02 delta entry in pack_bytes, without resolving or storing anything.
pack_key
Compute the on-disk pack key: BLAKE3 of the entire packfile bytes (including the trailer). SPEC-PACKFILE §7. Returns the bare digest; callers prepend packs/ and lower-hex-encode for the storage path.