Module codec 

Compression codecs. v0.1 ships store, zstd, zstd-with-dictionary and xz (LZMA2); the codec id space is reserved for context mixing, model-predictive, delta and float-split codecs (spec §4.2).

Structs

CodecOptions

Decision

What a sample decided for a block (decide_on_sample).

Encoded

Encoded output plus the parameters the decoder needs.

Enums

CodecChoice

Which codecs the packer may try for a block.

Constants

EFFORT_DEFAULT

Default effort: the full trial.

FILTER_ARM64

FILTER_NONE

FILTER_X86

PPMD

SAMPLE_SLICE

Length of one slice of the codec-choice sample (§2.1 of docs/design/plans/1.1-codec-sampling.md).

STORE

XZ

ZSTD

ZSTD_DELTA

zstd with a raw-content dictionary made of chunks of an earlier version (BlobRecord.d).

ZSTD_DICT

Functions

arm64_alignment

Heuristic: ARM64 code is a stream of 4-byte instructions with frequent BL calls to near targets (imm26 sign-extended, magnitude below 2^20 words). Random words qualify 0.05 % of the time; real code 1–5 %. Blocks start at arbitrary chunk boundaries, so all four word alignments are tried; returns the byte offset of the first aligned instruction word.

arm64_convert

ARM64 branch converter (the arm64 BCJ filter from XZ Utils, 0BSD): converts the immediate of BL and ADRP instructions from PC-relative to absolute form. Symmetric, works on 4-byte words.

compress_best

Try every applicable codec (with and without a branch filter when the block looks like x86 or ARM64 machine code) and keep the smallest output; store if nothing is smaller. Blocks that zstd cannot shrink by at least 3 % are treated as incompressible and the slower codecs are skipped.

decide_on_sample

Decide filter and codec for a block on its sample (sample() of the block; efforts 2..=4). The same heuristics and codecs as the full trial run on the sample, in the same order, and the candidate the full trial would have kept on the sample wins. looks_like_text is judged on the (filtered) sample here, not on the first 64 KiB of the block as in the full trial: a strided sample represents a mixed block better; this is intended.

decompress

decompress_with

filter_id

Filter byte layout: low nibble = filter id, high nibble = filter parameter (ARM64: byte offset of the first aligned instruction word, 0..3).

filter_param

looks_like_text

Heuristic: does the block look like text (worth a PPMd attempt)? Samples up to 64 KiB.

looks_like_x86

Heuristic: x86/x64 machine code has many E8/E9 CALL/JMP opcodes whose rel32 operand has a most-significant byte of 00 or FF (near targets). Random bytes hit that pattern about 0.006 % of the time; real code sections hit it 0.5–2 % of the time.

name

ppmd_params

PPMd model order / memory chosen deterministically from the block size: memory grows with the block (16 MiB..256 MiB), order 16 is a good text default for PPMd H.

sample

The codec-choice sample of a block: the block itself (borrowed) when it is small enough for the full trial, otherwise the concatenation of its sample slices. A pure function of the bytes.

sample_full_trial_max

Largest block that is never sampled: 2 * slices * SAMPLE_SLICE (128 KiB for blocks up to 1 MiB), below which a sampled trial costs about as much as the full one. Every golden archive holds one blob below this size, so efforts 2..=5 reproduce them by construction.

sample_offsets

Byte offsets of the sample slices of a block of len bytes, evenly spaced from 0 to len - SAMPLE_SLICE, each rounded down to a multiple of 4 so that the word alignment arm64_alignment finds on the sample is the block’s. Empty when the block is not sampled.

sample_slices

Number of slices the sample of a block of len bytes has: one per 128 KiB, at least 8 (a 1 MiB block gives 8 slices = 64 KiB) and at most 32 (256 KiB at 4 MiB and above). A block prefix must not be used instead: blocks concatenate chunks of several files and a prefix predicts the wrong codec on a third of the benchmark blocks (measured, see the plan).

samples_block

Does compress_best decide the codec of a block of len bytes on a sample with these options? (Best at efforts 2..=4 and a block above sample_full_trial_max.)

train_dict

Train a zstd dictionary on the unique chunks (needs at least 8 samples).

x86_convert

x86 BCJ branch converter (the classic LZMA SDK x86.c transform, public domain): rewrites the relative targets of E8/E9 CALL/JMP instructions into absolute addresses so that repeated calls to the same function become identical byte sequences. Symmetric: encoding = false restores the original bytes exactly. Returns the number of bytes fully processed.

zstd_compress_with_prefix

zstd with prefix as a raw-content dictionary (the “patch-from” model): matches may reach into the prefix, so a new version of a document costs about the size of its edits.

zstd_decompress_with_prefix