Module measure 

The timing harness for stuffr bench: one format at one level, run over the user’s input, compressed and decompressed, with only codec work inside a timed region, except the temp file’s writes when the output is too large to keep in memory (below).

Compression’s timed regions hold encoder creation, write_all of the input and Sink::finish (which consumes the sink, so its teardown is inside too). Decompression’s hold decoder creation and reading the decoder to its end into a buffer whose bytes are only counted. Building the destination, opening files, reading streamed input, the temp file’s creation and removal, and dropping the decoder are all outside.

Each phase is the sum of several timed pieces, each one pair of clock reads. Compression: encoder creation, then one piece per 64 KiB write_all of the input, then finish. Decompression: one piece per 64 KiB the decoder fills, the decoder’s creation in the first, its own reads of the compressed bytes being part of decoding. In memory and streamed alike; streamed, the input’s reads fall between the pieces. Otherwise nothing runs between them but the progress report, so the pieces cost every format the same: one pair of clock reads per 64 KiB.

Each codec gets the destination its own worst case allows, decided before any timing: memory whenever that worst case (input + 1/8 + 1 KiB, or 2 x input + 1 KiB for compress, LZW) fits under the memory left for it, half of --memory-limit when the input is held in memory, all of it when the input is streamed. Only past that does its output go to one temp file, and then its writes (to the page cache, through the 64 KiB buffer) fall inside its compression regions, so a row that went to disk is marked (Measured::on_disk; the table’s *, the output field of JSON and CSV). With --per-file each file’s stream is decided by its own size. Both destinations have the same shape: a 64 KiB BufWriter over a pre-reserved Vec, or over the temp file, so a codec that writes in small pieces pays the same either way. Decoding reads the same shape back: the Vec as a seekable, sized source, or the temp file through a 64 KiB BufReader, so a decoder that reads a byte at a time (LZW’s) costs a memory copy per byte, not a system call.

Each file’s length is taken once, when the input is loaded; every later read stops there, so a file still being written is benched as it was.

bench ignores STUFFR_THREADS; threads come only from –threads.

With --threads above 1, a multi-threaded encoder’s workers keep running in the untimed gaps between pieces, so its compression time can be under-reported against a single-threaded one. Streamed, the gaps hold the input’s reads; in memory, only the progress report (a clock read, and a redraw at most every 100 ms).

Structs

Plan

The conditions every row runs under.

RealClock

The real clock: nanoseconds since it was made.

Snapshot

A streamed file and its length when the input was loaded: every read of it stops at len, so bytes appended later are never benched.

Enums

Input

The input, in operand order: held in memory (one buffer per file) or streamed from the files.

Mode

One stream of the files concatenated, or one stream per file.

Phase

Which half of a row’s round trip a progress report is about.

Traits

Clock

A monotonic clock in nanoseconds; injected so tests can count its reads.

Functions

governor_for

The encode governor for plan: pack --threads’s mapping, except that no --threads means single-threaded whatever STUFFR_THREADS says (resolved_budget would read it). With a CLI value the variable is already ignored: resolve_workers takes the CLI value first.

load_input

Loads paths into memory when their total size is at most half of memory_limit, leaving room for the compressed copy; otherwise the input is streamed from the files.

measure

Compresses and decompresses input with codec at level (None: the encoder’s own default) under plan. Every error, a rejected level included, is a failed row; nothing panics.

warm

One untimed read of every file, so the first timed run does not pay for a cold cache. Nothing to do for input already in memory. A file that cannot be opened is a usage error naming it, as in load_input.