stuffr — worked examples
Everything below runs against this build as shown. `stuffr <command> --help`
lists the full flag reference for any one command.
COMPRESSING A FILE
stuffr pack notes.txt
Writes notes.txt.gz beside it. The input, notes.txt, is left alone —
unlike gzip(1), stuffr never deletes or replaces what you compressed.
stuffr pack notes.txt -o archive.gz
Choose the output name with -o/--output. Without it, the default is
the input path plus the chosen format's extension (".gz" today).
stuffr pack notes.txt --format gzip -o out.bin
--format names the codec explicitly instead of inferring one from -o's
extension. Every build — even one with no C toolchain — accepts
gzip, zlib, deflate, bzip2, brotli, lz4, snappy, zstd, xz, lzma and
lzip.
stuffr pack notes.txt --format xz -o out.xz
A default build writes xz through `lzma-rust2`, a pure-Rust port of
Tukaani's "XZ for Java" — no C toolchain needed, and unlike zstd's
pure fallback this is a FULL codec, not a weaker one:
`--allow-weak-encoder` is never required for xz. Measured on a 6.5 MB
payload against `--features c-backed`'s real `liblzma`: output sizes
within 0.02% of each other (parity, not a discount), encoding takes
roughly 1.2-1.3x as long, and decoding roughly 2-2.2x as long. Both
backends select CRC64 as xz's integrity check unconditionally, and
every position in a corruption sweep against either backend's own
output was detected. `c-backed` is a speed choice for xz, not a
capability one.
stuffr pack notes.txt --format lzma -o out.lzma
LZMA1, the older "alone" format xz superseded. Every build reads and
writes it, including one with no C toolchain: `--features c-backed`
swaps the pure-Rust `lzma-rust2` backend for the real `liblzma`,
which is faster. The two match byte-for-byte on small ordinary
payloads, but that is not a guarantee across every preset: measured
on a real file at presets 0/3/6/9, only preset 3 came out identical;
the rest differ by roughly 0.02-1.4% in size, parity rather than
identity. Unlike every other codec on this page, `.lzma` defines
NO magic bytes at all: its header's first byte encodes lc/lp/pb and
the next four the dictionary size, not a fixed signature, so this
format can never be detected from bytes alone, on a file or a pipe —
see "DECOMPRESSING" below. It also has no concatenation convention
the way gzip and xz do: unlike carrying a checksum, corruption is
still detected almost everywhere despite that (measured at 4,166 of
4,170 swept byte positions) through the LZMA range coder's own
structural sensitivity, not a designed check.
stuffr pack notes.txt --format lzip -o out.lz
LZIP, a newer container around the same LZMA1 payload `lzma` above
wraps, with two differences that matter: it carries a 4-byte `"LZIP"`
magic (so, unlike `.lzma`, a `.lz` file IS detected automatically —
see "DECOMPRESSING" below), and it defines a real multi-member
concatenation convention with a mandatory CRC32/size trailer per
member, so `cat a.lz b.lz | stuffr cat --format lzip -` decodes both
members' data, not just the first. Only one backend exists for it —
the same pure-Rust `lzma-rust2` xz and lzma use — so `--features
c-backed` makes no difference to this row at all, unlike zstd's or
xz's. Validated directly against the reference `lzip` 1.26 tool in
both directions, including the two-member case.
stuffr pack notes.txt --level 9
--level asks the codec for a tradeoff: gzip accepts 0 (fastest,
largest output) through 9 (slowest, smallest output). xz's range is
the same shape, 0-9, with 6 as its default too, and lzma and lzip
share that exact range too — both wrap an LZMA1 payload under the
hood. bzip2's range is 1-9, not 0-9 — it has no "store" level, so
--level 0 is a usage error (exit 2) rather than meaning "don't
compress". brotli's range is 0-11, and unlike every other codec here
its OWN default is 11 (the slowest setting it has), not something in
the middle of the range. zstd's range is the widest and the only one
with negative numbers — -131072 through 22 on this build — where
negative levels are zstd's "fast" modes, faster than level 1 at the
cost of ratio; its default is 3. lz4 and snappy have no level at all —
neither underlying crate exposes such a knob — so --level is accepted
and silently ignored rather than rejected, unlike every other codec on
this page.
stuffr pack notes.txt --force
Without --force, pack refuses to overwrite an existing output: exit
code 2, the existing file left untouched, nothing written.
COLLECTING FILES INTO AN ARCHIVE
stuffr pack one.txt two.txt -o bundle.tar
pack takes PATHS..., not a single path: an output naming a CONTAINER
(by extension, or with --format tar) collects every path into one
archive, one entry each. Each entry is stored under its path's final
component, so `stuffr pack /etc/hosts -o x.tar` writes an entry named
`hosts` — storing the absolute name would produce an archive stuffr
itself refuses to extract, at exit 7.
stuffr pack notes.txt --format tar
A single path still derives its own output name: notes.txt.tar.
Several paths have no one name to derive from, so they need -o.
stuffr pack one.txt two.txt -o bundle.tar.gz
stuffr pack one.txt two.txt -o bundle.tgz
A container INSIDE a codec, written in one step. -o names the whole
chain: the container comes from the inner extension and the codec
from the outer one, with the compound spellings (.tgz, .txz, .tbz2,
.tzst) read as the aliases they are. The codec's sink IS the
container's destination, so a single pass writes the entries, the
container's trailer and the codec's trailer in that order — nothing
is staged through a temporary file. --format names the CONTAINER
only, so `--format tar -o bundle.tar.gz` resolves to the same chain
rather than to a bare tar under a misleading name. When --format and
the -o name disagree about the CONTAINER (`--format zip -o x.tar.gz`)
pack refuses at exit 2 rather than picking one: the bytes would be
valid gzip, so pack would exit 0 and `stuffr list` on that same path
would then exit 5 calling stuffr's own output corrupt — a reader takes
the outer layer from the magic and the container from the name.
stuffr pack proj -o backup.tar
stuffr pack proj -o backup.tar.gz
A DIRECTORY is walked: every file, subdirectory and symlink under it
becomes an entry, named beneath the directory's own final component
(proj/src/main.rs, not the path as typed). Depth-first, with each
level sorted by name, so the same tree produces the same bytes twice
— readdir order is filesystem-dependent and is not alphabetical. An
empty directory is a real entry and survives the round trip.
Two symlink rules, opposites on purpose, both of them tar's: a link
NAMED on the command line is FOLLOWED (`stuffr pack link-to-notes.txt`
stores the file it points at, because storing the link itself would
produce an archive `stuffr unpack` refuses at exit 7), while a link
found INSIDE the walk is stored AS a link and never followed. That
second rule is also why there is no cycle detection: a loop cannot be
entered in the first place.
What the walk cannot store, it names rather than swallowing: a
socket, fifo or device node; a filename or symlink target that is not
valid UTF-8 (storing a lossy substitute would silently write a
different name, or point a restored link elsewhere); a directory it is
not allowed to LIST, which is stored while its contents are not; and a
file it is not allowed to OPEN, which is not stored at all. Neither
permission case fails the pack: one unreadable file in a home
directory must not cost the whole backup, and losing that one file
beats losing all of them. Nothing here is silent — each is named in
the fidelity report, and --strict-fidelity turns any of them into exit
4. An i/o ERROR part-way through a payload still fails the archive:
there is no honest way to finish an entry whose bytes stopped coming.
A file that merely CHANGED SIZE under the walk is different, and is
now forgiven too — a live system truncates and appends to files while
a backup runs, and the entry's header was written from a `stat` that
happened earlier. A file that shrank is padded with zeros to the
length its header declares; one that grew stops there. Either way the
archive stays correctly framed and valid, the entry is named in a
fidelity warning with both lengths, and --strict-fidelity still
refuses. GNU tar's `file changed as we read it` is the same bargain;
before this, one touched file aborted the pack and left no archive at
all. Hardlinks are reported too: no container here writes a hardlink
entry, so two names for one inode extract as two independent files.
stuffr pack . -o backup.tar
`.` and `..` work, and are named the way every other path is: by the
final component of the directory they RESOLVE to. Run from
/home/me/proj, `pack .` writes proj/src/main.rs — the same entries
`pack ../proj` would write, not a set of unprefixed ones. `tar cf
x.tar .` is what every tutorial teaches, so this is the likeliest
first thing to type at a directory walk; refusing it (which `.` having
no final component made it do) was a bug, not a rule. A trailing slash
is noise: `pack proj/` and `pack proj` are the same command.
`backup.tar` itself is excluded from the walk, so re-running this
exact command with --force does not nest the previous archive inside
the new one — GNU tar's `file is the archive; not dumped` for the
identical shape. It is reported on stderr every run, as a NOTE rather
than a fidelity warning, and --strict-fidelity does NOT fire on it: a
fidelity warning means something you asked for was lost, and stuffr
declining to put a file inside itself is not that. So
`pack . -o backup.tar --force --strict-fidelity` in a nightly job
stays at exit 0 run after run, and still exits 4 the moment anything
real is lost.
stuffr pack proj -o proj.a
`ar` has no directory and no symlink concept at all, so those entries
are DROPPED rather than written as zero-byte regular files — which
would make everything beneath them unextractable, the parent being a
file. The files themselves are stored, and each dropped entry is
named in a fidelity warning. --strict-fidelity turns any of these
into exit 4, on the write side exactly as it does on the read side.
Four things pack refuses, all before anything is written: more than one
path with a codec output (a codec compresses one stream and has nowhere
to put a second); two inputs whose final components collide (the archive
could not then be extracted without --force); a path NAMED on the
command line that is neither a file nor a directory — a fifo or socket
has no entry shape, and naming one is a request, not an incidental find
inside a walk; and a plan that would hold no entries at all.
That last one is exit 2, and it exists because of the exclusion rule two
paragraphs above. `stuffr pack backup.tar -o backup.tar --force` names
only the archive being written, so the walk excludes it, nothing is left,
and pack used to replace a good archive with an empty 1024-byte one at
exit 0 — with --strict-fidelity passing, since a self-exclusion is a note
and not a fidelity loss. The refusal happens while the plan is still
being built, before the destination is opened at all, so an archive
already there is not overwritten, not truncated, not touched. The message
names the cause rather than just saying there was nothing to do.
An empty DIRECTORY is not that case and still packs:
stuffr pack empty-dir -o x.tar # exit 0; x.tar holds empty-dir/
The directory's own entry is real — lose it and the tree does not come
back the shape it went in — so the plan is not empty and nothing refuses.
--strict-fidelity is NOT one of those refusals, and the difference is the
point: a refusal is a pre-flight check that writes nothing, while
--strict-fidelity is a verdict on an archive that HAS been written. The
archive is complete, valid, and LEFT ON DISK; exit 4 says only that
something the walk met could not be stored in it. Deleting it would throw
a whole good backup away over one unreadable file, which is exactly what
making that skip a warning was for — and it is the same contract the read
side keeps, where `unpack -C --strict-fidelity` exits 4 with the extracted
files still there. So read the exit code; do not read the file's absence.
A fifth refusal, for the same reason the extract side refuses flags it
cannot honour: --threads, --turbo and --allow-weak-encoder are refused
(exit 2) only where the resolved chain has NO codec layer to give them
to. A bare container is that case. `-o bundle.tar` compresses nothing,
so there is no worker count to hand over, no CPU cap to lift and no
fallback encoder to consent to, and accepting the flag there would
promise behaviour that never happens.
A composed output is NOT that case and is not refused. `-o backup.tar.xz`
really does reach xz's parallel encoder:
stuffr pack proj -o backup.tar.xz --threads 4 # honoured, exit 0
stuffr pack proj -o backup.tar --threads 4 # refused, exit 2
STUFFR_THREADS is read on that same composed path, and deliberately: the
environment is consulted exactly where the flag is honoured, never where
it is refused. --level reaches the codec layer there too.
What that costs is worth knowing rather than being surprised by. A
parallel xz or lzip encode splits its input per worker, so asking for
threads changes the bytes: `--threads 4` and no --threads at all produce
two different, equally valid archives from the same tree. The
reproducibility promise is the same one `stuffr info` states — same
input + same flags + same environment — and omitting every threading
flag builds no governor at all, so a plain `stuffr pack proj -o
backup.tar.xz` is byte-for-byte repeatable.
stuffr pack one.txt two.txt -o bundle.ar
stuffr pack one.txt two.txt -o bundle.cpio
ar and cpio (newc only) collect files the same way tar does — pack
infers the container from -o's extension (.a or .ar for ar, .cpio for
cpio) or from --format. Both are simpler than tar: neither carries a
trailing index or a per-entry codec, and they differ on directories:
cpio records one through its mode's type bits, ar has no directory
concept at all and drops the entry with a warning (see above). cpio's
per-entry size field is a hard 4 GiB (u32)
ceiling — an entry larger than that is refused outright, naming the
limit, never silently truncated to the wrong length. ar has no such
ceiling.
stuffr pack one.txt two.txt -o bundle.zip
zip collects files the same way, and is the first container here whose
INDEX is at the end of the file rather than beside each entry. Two
consequences worth knowing. Writing: a zip local header has to carry
each entry's CRC and compressed size, neither of which is known until
the entry has been compressed, so stuffr holds one entry in memory and
streams out everything before it — the archive is never buffered
whole, but a single huge entry does cost its own compressed size in
RAM. Reading: see "LISTING AND TESTING" below, because a zip on a
pipe is genuinely worth less than a zip in a file.
Entries are deflate-compressed by default. Directory entries carry
the trailing "/" the format identifies them by, and symlinks are
stored the way Info-ZIP stores them, so `unzip` restores them as
symlinks rather than as files holding a path.
stuffr pack one.txt two.txt -o bundle.zip --level 9
--level reaches a zip's entry compressor. deflate's range here is
1-9, NOT 0-9 like a bare `.gz`: zip's deflate writer has no "store"
level, so `--level 0` is a usage error (exit 2) naming the codec,
refused before the output is touched — the same shape as bzip2's
own missing level 0 under "COMPRESSING A FILE" above.
DECOMPRESSING
stuffr unpack notes.txt.gz
Writes notes.txt, stripping the recognised extension. As with pack,
an existing destination is refused unless --force is given too.
stuffr unpack notes.txt.gz -o out.txt --force
Choose the name and allow overwriting an existing one.
stuffr unpack data.bin --format gzip -o data.txt
--format names the codec explicitly instead of detecting it. Some
formats carry no magic bytes and no conventional extension, so an
explicit --format is the only way to reach them.
stuffr unpack raw.bin --format deflate -o data.txt
Raw deflate has no header, no trailer and no conventional extension,
so it cannot be detected at all — --format is the ONLY way in.
stuffr unpack data.br --format brotli -o data.txt
brotli defines no magic bytes at all, unlike gzip/zlib/bzip2's header
bytes — detection from a file works only by the `.br` extension. A
brotli stream arriving on a pipe, with no filename to go by, cannot be
detected either way, so --format brotli is the only way in there too.
stuffr unpack data.lzma --format lzma -o data.txt
LZMA1 defines no magic bytes either, the same as brotli — detection
from a file works only by the `.lzma` extension, and a `.lzma` stream
arriving on a pipe, with no filename to go by, cannot be detected at
all, so --format lzma is the only way in there too. No special build
is needed — every build has an LZMA1 backend.
stuffr unpack data.lz
LZIP, unlike its older sibling LZMA1 just above, DOES carry a 4-byte
`"LZIP"` magic at offset 0 — so this is detected automatically, on a
file or a pipe, and needs no --format at all.
STREAMING TO STDOUT
curl -s https://example.com/log.gz | stuffr cat - | grep pattern
cat decompresses straight to stdout without ever naming an output
file — the shape for feeding a compressed stream into another tool.
stuffr cat data.bin --format gzip
cat takes --format for the same reason, and --max-ratio to bound a
decode that would otherwise stream an enormous file to stdout.
curl -s https://example.com/logs.tar | stuffr cat - app.log
Naming an entry makes cat entry-aware: it streams that entry's bytes
and nothing else, in archive order, to stdout — over a pipe, with no
seeking and nothing written to disk. Without an entry name cat decodes
its input as a single stream, so `stuffr cat backup.tar` is a usage
error (exit 2) that names what to do instead. --max-ratio bounds this
exactly as it bounds extraction below, and --memory-limit bounds the
codec layer beneath the container (see "EXTRACTING AN ARCHIVE" below).
--format is refused alongside an entry name rather than accepted and
ignored: entry-aware streaming detects the container, and any codec
above it, from the archive itself. `--index N` asks for the same
entry-aware streaming by POSITION rather than by name — see --index
under "EXTRACTING AN ARCHIVE'S ENTRIES" below.
STDIN AND STDOUT VIA `-`
`-` means the stream, wherever a path is expected: stdin for an input,
stdout for an output.
cat notes.txt | stuffr pack - -o notes.txt.gz
Stdin has no path to derive a name from, so -o is mandatory here;
leaving it out is a usage error (exit 2).
cat notes.txt | stuffr pack - -o - | wc -c
Both ends can be `-` at once: stdin in, stdout out.
INSPECTING A STREAM WITHOUT DECODING IT
stuffr info notes.txt.gz
Reports the format, the resolved chain, the fidelity "rung", and the
size when it's known.
stuffr info notes.txt.gz --json
The same report as machine-readable JSON, for scripting.
A plain file reports `rung: exact` — stuffr can seek it, so what it reports
is verified. A pipe (`stuffr info -`) reports `rung: forward-only` — a
stream can only be read once, so this is the tool telling you the format
was inferred from bytes it saw going by, not confirmed by looking back.
The heading is literal: info does not decode, and it does not OPEN an
archive either. So it reports `fidelity: not evaluated` instead of
"nothing approximated" wherever a forward read of that container could
genuinely have lost something and info has not looked. Today that means
zip, the only container here that keeps its index at the END of the
stream, whenever that zip will not be read authoritatively (run `stuffr
test` for the real answer; see "LISTING AND TESTING" below). Two shapes
qualify:
* a zip on a pipe, and
* a zip under a codec — `bundle.zip.gz` — however seekable the FILE
is. The rung reported is the raw input's, and a seekable file says
nothing about a container read out of a gzip decoder: that decoder
is forward-only by construction, and `stuffr test bundle.zip.gz
--strict-fidelity` on the same bytes exits 4 with two warnings.
Everywhere else the claim is true and info still makes it: a seekable
bare zip read its central directory, a tar, ar or cpio has no trailing
index to miss piped, wrapped or not, and a bare codec stream has no
container at all. In `--json` the same distinction is the
`fidelity_evaluated` field, because `"warnings": []` cannot carry it.
One limit worth knowing: on a PIPE with no filename, info resolves only
the outer layer. `cat bundle.zip.gz | stuffr info -` reports `chain:
gzip`, because seeing the zip inside would mean running the decoder,
which info does not do. It reports honestly on what it identified.
LISTING AND TESTING AN ARCHIVE
stuffr list backup.tar
Prints one line per entry — index, size, then name — and extracts
nothing at all: no file is written, no directory created. `ls` is an
alias for `list`, for the muscle memory. Both work on `backup.tar.gz`
and `backup.tgz` exactly as they do on a bare `.tar`: the codec layer
(gzip) is resolved and decoded first, then `list` walks the tar
layer underneath it — on a real file or over a pipe (`stuffr list -`)
alike.
The first column is the entry's position in ARCHIVE ORDER, counting
from 0, and it is what `cat --index` and `unpack --index` select by
(see --index under "EXTRACTING AN ARCHIVE'S ENTRIES" below). One
forward walk
produces both the listing and the numbering, so `list` and `--index`
cannot disagree about which entry is which.
stuffr ls backup.tar.gz --json
`--json` emits a JSON array of `{index, name, kind, size}` objects
instead of one line per entry, for scripting. `kind` is `file`, `dir`,
`symlink` or `other`, and `index` is the same 0-based archive position
the human listing prints.
stuffr list photos.zip --strict-fidelity
`list` reports what the read approximated or lost, on stderr, and
--strict-fidelity turns any such report into exit 4 — the same
contract `test` and `unpack -C` already had, through the same printer.
It matters most on exactly the archive whose LISTING is incomplete: a
zip whose central directory holds 8 records under 6 distinct names
lists six entries, and the two shadowed records are unreachable. Before
this, `stuffr test` said so and `stuffr list` — the verb you reach for
first — printed those six lines and stayed silent.
stuffr test backup.tar
Reads every entry's data all the way through — not just its header —
verifying integrity without writing anything to disk. A `test` that
only walked headers would pass on an archive whose payloads are
corrupt; this build actually pulls the bytes, into nowhere. Exits 5 on
a corrupt or truncated archive, the same code `unpack`/`cat` use for a
damaged codec stream, so a script branches on "corrupt" the same way
regardless of which command found it.
stuffr test backup.tar --strict-fidelity
Fails with exit 4 if the read approximated anything at all. A tar over
a pipe is NOT such a case: it lands on the forward-only rung, but tar
has no trailing index for a forward read to skip, so nothing is lost
and nothing is reported. The flag gates on what was lost, never on
the rung.
stuffr list plain.txt.gz
stuffr list plain.txt
Both fail with exit 2, because both are the same mistake: you pointed
an entry-aware verb at something that has no entries. The first is a
bare codec stream, and the error names what the input actually
resolved to (here, `gzip`) rather than a bare refusal, so "oh, that's
not a tar, it's just gzip" is diagnosable without inspecting the file
by hand. The second is a file of no recognised format at all, and the
error prints its leading bytes. A file whose format is AMBIGUOUS is
exit 2 as well, and its message names the candidates and tells you to
pass --format.
stuffr list bundle.ar
stuffr test bundle.cpio
list, test, cat and unpack -C all work identically on ar and cpio:
each is a plain forward read, the same shape as tar's own. Neither
format has a trailing index to lose on a pipe, so `--strict-fidelity`
passes a piped ar or cpio archive for the same reason it passes a
piped tar.
stuffr list legacy.cpio
Exit 3, not 5, if that archive is `odc` (magic 070707) or the CRC
variant of `newc` (magic 070702). Both are perfectly valid cpio
archives that this build cannot read — the crate it uses implements
`newc` (070701) alone — so they are recognised by magic and refused
as a capability limit, naming the variant and how to convert it.
Calling an intact file corrupt would be a lie; exit 5 is reserved for
damage. Bytes that are not cpio at all are still exit 5.
stuffr test photos.zip --strict-fidelity
cat photos.zip | stuffr test - --strict-fidelity
The first succeeds and the second fails with exit 4, and that is the
whole point of the flag. A zip keeps its authoritative metadata in a
central directory at the END of the file: given a real file stuffr
reads it (`stuffr info` says rung "exact" — random access to any
entry, unix modes present, symlinks reported as symlinks), and given
a pipe it cannot, so it reads the inline local headers instead. That
yields the same DATA and a strict subset of the metadata — no modes,
no entry count, and a symlink indistinguishable from a small file
holding its target. Both losses are reported by name, which is why
--strict-fidelity fails here and passes on a piped tar: it gates on
what was lost, never on the rung. (`list`/`ls`, `test` and `unpack`
all take the flag; `cat` does not, because its output IS the data and
a caller piping it somewhere wants the bytes or an error, not a
verdict on metadata.)
stuffr test archive.zip
Verifies every entry's CRC32 — zip carries one per entry, so
corruption is really detected rather than inferred. Exit 5 for a
damaged or truncated archive, including one cut anywhere inside its
central directory: a forward read stops at the index rather than
reading it, so stuffr checks that the index is whole itself. Bytes
appended AFTER a complete archive are ignored, as every zip tool
ignores them.
stuffr test encrypted.zip
Exit 3, not 5: an encrypted entry is a capability this build does
not have, not a damaged file, and the message says which. Exit 3 is
"this build cannot do that" throughout — the same code you get for
naming a format the build was compiled without. The same answer
covers a zstd-compressed zip ENTRY on a default build: the one entry
codec that needs a C toolchain, so the refusal names `--features
c-backed`. Note the asymmetry there — a standalone .zst file reads
fine on every build; it is only zstd INSIDE a zip that needs the
flag. An LZMA zip entry is the reverse, readable everywhere and
writable nowhere, and reports as exit 3 too.
stuffr list bomb.tar.gz --max-ratio 100
stuffr test bomb.tar.gz --max-ratio 100
`list` and `test` both resolve the codec layer beneath the container
(gzip, for `.tar.gz`/`.tgz`) before the container ever sees a byte, so
that layer is bounded the same way `unpack`/`cat` already bound a
bare codec stream: past this multiple of the compressed input, the
read is refused at exit 6, naming the ratio, the limit and the flag.
Defaults to 10000, same as `unpack`/`cat` — raise it if a legitimate
archive is this compressible; a bare `.tar` (no codec layer at all)
is never affected, since nothing there is expanding.
`test` bounds each ENTRY's decoded payload too, exactly as `cat` and
`unpack -C` do — the same per-entry and running-total pair, charged
from bytes actually read. All four entry-aware verbs refuse the same
bomb at the same ratio; the verb you reach for to inspect an
untrusted archive is not the lenient one.
stuffr list bundle.tar.lz --memory-limit 512M
stuffr test bundle.tar.lz --memory-limit 512M
--memory-limit is the OTHER bound, and --max-ratio cannot stand in
for it: the pure xz, lzma and lzip decoders size a dictionary from a
value the file declares in its own header, before producing a single
byte of output, and --max-ratio counts output bytes. So a crafted
131-byte .tar.lz declaring a 512 MiB dictionary sails past any ratio
and is refused at exit 6 by this bound alone. On by default at 25% of
available RAM (cgroup-aware) for `list`, `test`, `cat` and
`unpack -C` alike — "reads nothing and extracts nothing" was never
true of the codec layer above the container.
EXTRACTING AN ARCHIVE'S ENTRIES
stuffr unpack backup.tar -C out/
-C/--directory extracts every entry into out/, creating it if it does
not exist yet. -C is what asks for entry-aware extraction: without it
unpack decodes a single stream, so pointing it at a container is a
usage error (exit 2) naming this flag. It is a flag rather than
something inferred from the input because the decision has to be made
before a byte is read — a pipe cannot be probed and then reconsidered.
stuffr unpack backup.tar.gz -C out/ docs notes.txt
PATTERNS after the input select what to extract: an exact entry name,
or a directory selecting everything beneath it (`docs` takes
`docs/a.txt` with it). Deliberately no globbing — the shell already
expands `*`, and a pattern language nobody asked for is a pattern
language to get subtly wrong. A pattern matching no entry at all is a
usage error (exit 2), never a silent success.
stuffr cat notion.zip --index 2 > notion-api.sh
stuffr unpack notion.zip -C out/ --index 2
--index selects an entry by POSITION instead of by name — the 0-based
number `stuffr list` prints in its first column, counting in archive
order. It is repeatable (`--index 0 --index 3`), and it is how you
reach an entry whose name you cannot usefully type:
* a zip whose central directory holds two records under one name, so
naming it can only ever reach one of them;
* `Makefile` and `makefile` from a case-sensitive filesystem, on a
case-insensitive one;
* any name your shell would fight you over.
Entries come out in archive order however the flags were ordered, and
a repeated index selects its entry once — the same contract PATTERNS
have. An index past the end is a usage error (exit 2) naming the
archive's real entry count, never a silent empty result.
--index and PATTERNS are two ways of saying the same thing, so passing
both is a usage error (exit 2) rather than a guess at whether you
meant the union or the intersection. On `unpack`, --index needs -C for
exactly the reason a pattern does: it selects entries, and without -C
there is nowhere for entries to go.
It works over a pipe too (`cat notion.zip | stuffr cat - --index 2`),
because the position is counted during the ordinary forward walk. On a
real file a container that HAS an index — zip is the only one here —
jumps straight to the entry instead; tar, ar and cpio are sequential
formats with no index to jump with, so they always count. Both routes
deliver the same bytes, which is the property the tests pin.
stuffr unpack backup.tar -C out/ --force
An entry whose target path already holds a file or a symlink is
refused (exit 2) unless --force is given — the same contract pack and
unpack apply to a single output file. --force REPLACES what is in the
way rather than writing through it: a symlink already sitting at an
entry's path would otherwise redirect that entry's bytes wherever it
points. An existing directory is never removed either way; entries
are written into it.
Path containment is not optional and cannot be switched off. An entry
naming an absolute path, traversing above the destination with `..`, or a
symlink whose target resolves outside it, is REFUSED at exit 7 — never
silently sanitised into something safe, because a caller handed a
cleaned-up path has no way to know it was handed one. The refusal names
the offending entry verbatim and stops the extraction there, so a later
entry written "through" a hostile symlink is never reached. An entry
named `.` or `./` IS accepted: it names the destination directory itself,
and it is the first entry `tar cf x.tar .` emits.
A symlink whose target resolves TO the destination is accepted too, for
the same reason: `sub/top -> ..` points at the extraction root, and the
root is inside the root. One step further out (`sub/top -> ../..`) is an
escape and is refused. Note the asymmetry with entry names, which is
deliberate: an entry NAMED `sub/..` is refused, because no real archiver
emits one and a name that pops back to the destination is more likely an
attack than an intent.
An entry whose path runs THROUGH a symlink is refused too — a directory
component that is a link, whether an earlier entry created it or somebody
planted it in the destination beforehand. The two refusals above resolve
a name component-wise, which is what keeps them immune to a symlink
appearing between the check and the write; the operating system resolves
the same name through the link instead, and that difference is an escape
on its own.
stuffr unpack backup.tar -C out/ --strict-fidelity
Extraction restores each entry's mode and mtime — a directory's after
its children are written, since creating one bumps its parent's mtime
back to now — and reports whatever it could NOT put back.
--strict-fidelity turns any such report into exit 4 instead of a
warning, for a caller that needs "everything, exactly" rather than
"the data". `stuffr test --strict-fidelity` gates the same way on a
read. Three things extraction deliberately does not reproduce, each
reported per entry rather than passed over in silence:
* setuid, setgid and the sticky bit. An archive is untrusted input and
honouring its setuid bit hands over a privilege escalation for free,
so only the 0o777 permission bits are restored — the same default
tar and bsdtar apply for a non-root extraction.
* a symlink's own mode and mtime, which cannot be set without
following the link on this platform. Expect this one: it means
--strict-fidelity fails on essentially any tarball containing a
symlink at all. The gate firing there is honest rather than a bug —
the link's mtime genuinely was not reproduced — but it is why the
flag is opt-in, and why a caller who only wants "the data arrived"
should not reach for it.
* a device node, fifo, socket or hardlink. There is no shape on disk
for one here, so the entry is SKIPPED and named, never materialised
as a regular file holding its supposed contents.
Ownership (uid/gid) is not restored and not reported: reproducing it
needs privileges an extraction does not have, and warning per entry
would fail --strict-fidelity for every archive in existence, which
would make the flag useless rather than strict.
stuffr unpack bomb.tar.gz -C out/ --max-ratio 100
--max-ratio bounds extraction twice over — per entry, and against the
archive's running total — charged from the bytes actually read rather
than the size an entry's header declares, since a header that
under-declares would otherwise walk straight through the check. Past
the limit the extraction is refused at exit 6, naming the entry. The
identical bound applies to `list`, `test` and `cat`, so the verb you
reach for to INSPECT an untrusted archive is no less bounded than the
one that writes it to disk.
--no-sync is accepted and does nothing — extraction writes entries
straight to their final paths, so there is no fsync for it to skip.
Extraction is NOT atomic, and this is the one place the two paths
differ. A single-stream decode (`stuffr unpack a.gz -o out`) writes to
a temp file and renames it into place, so a refused decode leaves no
partial file at all. An entry-aware extraction writes each entry
straight to its final path, so a refusal partway through — a bomb, an
unsafe path, a corrupt entry — leaves whatever entries already
completed, plus a partially-written one. Extract untrusted archives
into a fresh directory you can delete, and check the exit code before
trusting the contents.
stuffr unpack bundle.tar.lz -C out/ --memory-limit 512M
--memory-limit bounds the codec layer BENEATH the container, which
--max-ratio cannot see: the pure xz, lzma and lzip decoders size a
dictionary from a value the file declares in its own header, before
producing any output, and --max-ratio counts output bytes. A crafted
131-byte .tar.lz declaring a 512 MiB dictionary is refused at exit 6
rather than allocating for it. Defaults to 25% of available RAM
(cgroup-aware) on `list`, `test`, `cat` and `unpack -C` alike — the
bound is on by default, since a bound that defaults off closes
nothing. Raise it to decode a file that genuinely needs more.
WHAT THIS BUILD CAN DO
stuffr formats
Lists the formats compiled into THIS binary, not everything stuffr will
ever support. A pure build (the default) has fifteen rows: the eleven
codecs gzip, zlib, deflate, bzip2, brotli, lz4, snappy, zstd, xz, lzma
and lzip, plus the tar, ar, cpio and zip containers — the KIND column
tells codecs and containers apart. zstd's row differs by which backend is
compiled in: a build with the `zstd-c` feature (needs a C toolchain)
shows `yes` in WRITE, using the real `libzstd`; a pure-Rust build
falls back to `ruzstd` instead, and shows `weak` in WRITE — see
"USING A WEAK FALLBACK ENCODER" below.
Either way the format still reads `.zst` unconditionally; only writing
it without the C backend needs the flag. xz's and lzma's rows do NOT
have this shape: both backends show `yes` in WRITE, never `weak` —
`--features c-backed` swaps the pure-Rust `lzma-rust2` backend for the
real `liblzma`, which is faster (see "COMPRESSING A FILE" above) but not
more capable. lzip's row is unaffected by `c-backed` entirely — it has
only ever had the one, pure-Rust `lzma-rust2` backend, so this feature
changes nothing about it, in either direction. All three tiers (`pure`,
`pure` plus `c-backed`, and any granular combination) list all
fifteen formats — the ONLY capability difference the `c-backed`
feature makes anywhere in this matrix is zstd's WRITE column, `weak`
against `yes`.
The PARALLEL column is truthful per build too, and independent of
the WRITE column above: on the default (`pure`) tier it reads `yes`
for `xz` and `lzip` and `-` for every other format, including
`zstd` — `ruzstd` has no multi-threaded encoder to hand a worker
count to. Adding `--features c-backed` flips `zstd` to `yes` as
well, alongside its WRITE column becoming `yes`; it changes nothing
else in this column, on any tier. All four containers — tar, ar, cpio
and zip — always show `-` here: none of them parallelises anything, on
any tier.
GUARDING AGAINST DECOMPRESSION BOMBS
stuffr unpack bomb.gz --max-ratio 100
A decompression bomb is a small compressed file that expands to an
enormous one. --max-ratio refuses a decode once output has grown
past this multiple of the compressed input (default 10000). gzip's
structural ceiling is roughly 1032:1, so it cannot reach that default
alone — here the flag mostly guards oversized input, or lets a
cautious caller set a tighter bound by hand. It earns its keep once
later phases add codecs and containers that CAN reach 10000:1.
SKIPPING THE DURABILITY FSYNC
stuffr pack big.txt --no-sync
By default, pack and unpack fsync the output before renaming it into
place, so a crash cannot leave a truncated file under the real name.
--no-sync trades that guarantee for speed on bulk or scratch work.
USING A WEAK FALLBACK ENCODER
stuffr pack big.dat --format zstd --allow-weak-encoder
A build with no C toolchain writes `.zst` through `ruzstd`, a
pure-Rust fallback, instead of the reference `libzstd`. The output is
valid zstd any tool can read, but markedly larger — measured at 3.53x
on an 11.7 MB corpus where the C backend's default reaches 6.21x
(files 76% larger) — and slower (969ms against 118ms in that same
measurement), and it buffers the whole input in memory rather than
streaming it, unsuitable for large files. pack refuses such an
encoder unless this flag is given; run `stuffr formats` to see which
codecs in THIS build are affected (WRITE shows `weak` rather than
`yes`). Rebuild with `--features c-backed` to remove the limitation.
xz does not have this shape: its pure-Rust backend (`lzma-rust2`) is a
full codec at ratio parity with `liblzma`, not a weaker fallback, so
`--allow-weak-encoder` never applies to it, in any build. lzma (the
older LZMA1 "alone" format) shares xz's shape now too: its pure-Rust
backend is the same full-ratio `lzma-rust2`, so `--features c-backed`
is purely a speed choice for it as well, never a capability one.
lzip does not have this shape either, for a simpler reason still: it
has exactly one backend, period — there is no second, C-backed
implementation for it to differ from, weakly or otherwise, so
`--features c-backed` changes nothing about its row at all.
PARALLEL ENCODE FLAGS
stuffr pack big.dat --format xz --threads 4
Encode with 4 worker threads instead of stuffr's default of one. Only
THREE of this build's eleven codecs actually parallelise their
encoder: xz and lzip on every build, and zstd only with --features
c-backed (the default ruzstd encoder has no multi-threaded path).
Counted by implementation rather than format this is four
(zstd-c, xz-c, xz-pure, lzip), because xz has two interchangeable
backends — but xz-c and xz-pure register the same format and never
both compile in, so no single build ever shows more than three
PARALLEL rows in `stuffr formats`. The other eight (nine counting only
one xz backend) accept --threads without error but encode
single-threaded regardless, since their crates expose no MT encoder
to hand a worker count to. --threads is deliberately opt-in rather
than auto-detected: a multi-threaded encoder splits its input across
workers, so the very same file compresses to different bytes
depending on how many workers ran. Omitting --threads (and --turbo,
and leaving STUFFR_THREADS unset) keeps pack single-threaded, so a
given input, given the same flags and the same environment, always
produces the same output bytes — "same machine" is not part of the
promise once --threads 0 is in play, because auto-detection
resolves against cgroup CPU quotas, not just core count.
--memory-limit does not affect this: it bounds allocation and the
worker count the governor can grant, never which bytes a worker
count already granted produces. Passing --threads 0 asks stuffr to
auto-detect a worker count
instead, trading that byte-for-byte reproducibility for speed.
--threads exists only on pack; multi-threaded decode is out of scope
for this phase, and unpack/cat refuse the flag outright rather than
silently accepting and ignoring it — see
"threads_is_not_offered_on_decode_subcommands" if you want the
enforcement in test form.
stuffr pack big.dat --format xz --threads 4 --turbo
--turbo lifts the CPU cap only, letting stuffr use the full detected
core budget instead of a conservative default worker count.
--memory-limit (below) still binds regardless of --turbo: turbo
means "use my cores," not "ignore the OOM killer."
stuffr pack big.dat --format xz --memory-limit 512M
stuffr unpack archive.xz --memory-limit 512M
--memory-limit caps the memory stuffr will ask for. It accepts a plain
byte count or a binary, 1024-based K/M/G suffix (e.g. 512M or 2G) —
matching what `xz --memlimit` and every other tool in this space
mean by "M", not the decimal 1000-based reading. A malformed value
(512MB, a decimal unit, anything else this parser doesn't recognise)
is a usage error naming the accepted forms, never a silent fall back
to the default — a user who typed 512MB and got 25% of RAM instead
would have no way to notice. On pack this bounds the worker count;
unpack, cat and info accept it too, since a decode can also need to
fit inside a memory ceiling even with no threads to bound. Defaults
to 25% of available RAM, honouring cgroup limits; `stuffr info` reports
the resolved figure, humanized (e.g. "256 MiB"), on every build.
The bound has teeth: xz and lzip cost roughly 128 MiB per worker
below preset 7 and roughly 896 MiB per worker at preset 7 and above
(the match finder, not the block buffer, dominates). On the 256 MiB
floor this build defaults to when it cannot read available memory
(macOS, with no /proc/meminfo), that means `stuffr pack big.dat --format
xz --threads 8 --level 9` silently runs with exactly ONE worker —
the governor protecting the machine, not a bug — unless
--memory-limit is raised. A 4 GiB budget gets 4 workers at that same
level, or 32 (then CPU-capped) at the default level 6.
--threads, --turbo and --memory-limit build the thread governor that
pack's encode path consumes: --threads and --turbo resolve to a worker
count via the same precedence STUFFR_THREADS and a future config file will
follow, and --memory-limit resolves to the byte cap that count is
clamped against — divided by the codec's own measured per-worker demand,
never a flat division. Omitting --threads builds no governor at all —
the codec sees no worker count and encodes single-threaded, which is
what keeps output byte-reproducible by default. unpack, cat and info
accept --memory-limit and validate it the same way. info only reports
the resolved figure (it never decodes), but unpack and cat pass it
through to the decoder: a handful of pure codecs size a dictionary
buffer from a value the file declares in its own header, before
producing any output, so an unbounded decode of a hostile file can
allocate far more than the archive's own size would suggest — this is
what closed the xz-pure/lzip/lzma-pure decode denial-of-service, each
refusing at exit 6 (see "EXIT CODES" below) naming the declared size,
the limit and the flag, never the exit 5 a genuinely corrupt archive
gets. Omitting the flag on unpack/cat still binds — it resolves to the same
25%-of-RAM default as pack — because a bound that defaults off closes
nothing.
EXIT CODES
0 success
1 general failure (i/o error, or anything not listed below)
2 usage error (bad flag, missing -o, existing output without --force,
list/test pointed at something that resolved to a bare codec stream
rather than an archive, an input whose format could not be detected
at all or was ambiguous, unpack/cat pointed at a container with
neither -C nor an entry name, or a pattern matching no entry) — every
one of these is "you pointed the command at the wrong thing", and
they all share the one code
3 format not available in this build
4 fidelity degraded under strict mode
5 archive or stream is corrupt
6 resource limit exceeded (--max-ratio or --memory-limit tripped, or
archive nesting exceeded the depth stuffr will peel through)
7 unsafe path refused
NOT YET
A pure build (the default) has eleven codecs and four containers: gzip,
zlib, deflate, bzip2, brotli, lz4, snappy, zstd, xz, lzma and lzip, plus
tar, ar, cpio and zip. zstd's pure backend (`ruzstd`) is a weak fallback (see
"USING A WEAK FALLBACK ENCODER" above); xz's and lzma's pure backends are
not — both read and write full-ratio output with no C toolchain, just
slower than `--features c-backed`'s `liblzma` (roughly 1.2-1.3x to
encode, 2-2.2x to decode, measured on a 6.5 MB payload). `--features
c-backed` upgrades all three: zstd to the real `libzstd`, and xz/lzma to
`liblzma` for speed. lzip has no such upgrade path at all — it has
exactly one backend (the same pure-Rust `lzma-rust2` xz and lzma use), so
`--features c-backed` changes nothing about it, and there is no
weak-fallback distinction for it either.
All four containers read and write, and all four entry-aware
subcommands work on each of them identically: `list` and `test` read
entries without extracting, `unpack -C` writes them to disk under the
containment rules above, and `pack PATHS...` collects files into one.
`.tar.gz` and `.tgz` resolve as tar over gzip, which `stuffr info` shows
as the chain — ar, cpio and zip have no such codec-wrapped convention of
their own, though any of them wrapped in a codec still resolves the same
way from its extensions (`bundle.zip.gz`). `ar` is the simplest of the
four: no trailing index, no directory concept, no multi-member
concatenation convention. `zip` is the only one with a trailing index,
which is why it is the only one whose forward read approximates anything
— see "IDENTIFYING A STREAM" above. `cpio` is `newc` (SVR4 "new ASCII")
only: the crate this build uses does not implement the older `odc`
format (magic 070707) or the CRC variant of `newc` (magic 070702).
Pointing stuffr at either is exit 3, not 5 — both are recognised by
magic and refused with a message naming the variant and how to convert
it, because an intact archive in a variant this build cannot read is a
capability limit, not a damaged file. cpio's per-entry size field is
also a hard 4 GiB (u32) ceiling: an entry larger than that is refused
with a typed error naming the limit, never silently written with the
wrong framing. Walking a directory tree and writing a container inside a
codec (`bundle.tar.gz`) both work in a single command now — see
"COLLECTING FILES INTO AN ARCHIVE" above. `stuffr formats` always tells
you the truth about the build you are actually running, whatever this
page says.