Module format 

Binary format helpers shared across AOF and snapshot files.

Provides TLV-style encoding primitives, CRC32 checksums, and magic byte constants. All multi-byte integers are stored in little-endian.

Enums

FormatError

Errors that can occur when reading or writing persistence formats.

Constants

AOF_MAGIC

Magic bytes for the AOF file header.

FORMAT_VERSION

Current unencrypted format version.

FORMAT_VERSION_ENCRYPTED

Format version for encrypted files.

MAX_COLLECTION_COUNT

Maximum element count for collections (lists, sets, hashes, sorted sets) in persistence formats. Prevents corrupt count fields from causing unbounded iteration during deserialization. 100M is well beyond any realistic collection while catching obviously corrupt u32 values.

MAX_FIELD_LEN

Maximum length we’ll allocate when reading a length-prefixed field. 512 MB is generous for any realistic key or value — a corrupt or malicious length prefix won’t cause a multi-gigabyte allocation.

MAX_PERSISTED_VECTOR_COUNT

Maximum element count per vector set in persistence formats. Prevents corrupt count fields from causing unbounded loops.

MAX_PERSISTED_VECTOR_DIMS

Maximum vector dimensions allowed in persistence formats. Matches the protocol-layer cap. Records exceeding this are rejected during deserialization to prevent OOM from corrupt files.

MAX_PERSISTED_VECTOR_TOTAL_FLOATS

Maximum total f32 elements (dim * count) for vector deserialization. Caps total allocation at ~4 GB. Without this, a crafted file with 65536 dims x 10M vectors would attempt ~2.6 TB.

SNAP_MAGIC

Magic bytes for the snapshot file header.

Functions

capped_capacity

Caps pre-allocation to avoid huge allocations from corrupt count fields.

crc32

Computes a CRC32 checksum over a byte slice.

read_bytes

Reads a length-prefixed byte vector: [len: u32][data].

read_f32

Reads an f32 in little-endian.

read_f64

Reads an f64 in little-endian.

read_header

Reads and validates a file header. Returns an error if magic doesn’t match or version is unsupported. Returns the format version.

read_i64

Reads an i64 in little-endian.

read_u8

Reads a u8 from the reader.

read_u16

Reads a u16 in little-endian.

read_u32

Reads a u32 in little-endian.

validate_collection_count

Validates that a deserialized collection count is within bounds. Returns InvalidData if the count exceeds MAX_COLLECTION_COUNT.

validate_vector_total

Validates that the total vector element budget (dim * count) is within bounds. Call after validating dim and count individually.

verify_crc32

Verifies that data matches the expected CRC32 checksum.

verify_crc32_values

Verifies that two CRC32 values match.

write_bytes

Writes a length-prefixed byte slice: [len: u32][data].

write_f32

Writes an f32 in little-endian.

write_f64

Writes an f64 in little-endian.

write_header

Writes a file header: magic bytes + version byte.

write_header_versioned

Writes a file header with an explicit version byte.

write_i64

Writes an i64 in little-endian.

write_len

Writes a collection length as u32, returning an error if it exceeds u32::MAX.

write_u8

Writes a u8 to the writer.

write_u16

Writes a u16 in little-endian.

write_u32

Writes a u32 in little-endian.