stuffr — worked examples
Everything below runs against this build as shown. `stuffr <command> --help`
lists the full flag reference for any one command.
COMPRESSING A FILE
stuffr pack notes.txt
Writes notes.txt.gz beside it. The input, notes.txt, is left alone —
unlike gzip(1), stuffr never deletes or replaces what you compressed.
stuffr pack notes.txt -o archive.gz
Choose the output name with -o/--output. Without it, the default is
the input path plus the chosen format's extension (".gz" today).
stuffr pack notes.txt --format gzip -o out.bin
--format names the codec explicitly instead of inferring one from -o's
extension. Every build — even one with no C toolchain — accepts
gzip, zlib, deflate, bzip2, brotli, lz4, snappy, zstd, xz, lzma and
lzip.
stuffr pack notes.txt --format xz -o out.xz
A default build writes xz through `lzma-rust2`, a pure-Rust port of
Tukaani's "XZ for Java" — no C toolchain needed, and unlike zstd's
pure fallback this is a FULL codec, not a weaker one:
`--allow-weak-encoder` is never required for xz. Measured on a 6.5 MB
payload against `--features c-backed`'s real `liblzma`: output sizes
within 0.02% of each other (parity, not a discount), encoding takes
roughly 1.2-1.3x as long, and decoding roughly 2-2.2x as long. Both
backends select CRC64 as xz's integrity check unconditionally, and
every position in a corruption sweep against either backend's own
output was detected. `c-backed` is a speed choice for xz, not a
capability one.
stuffr pack notes.txt --format lzma -o out.lzma
LZMA1, the older "alone" format xz superseded. Every build reads and
writes it, including one with no C toolchain: `--features c-backed`
swaps the pure-Rust `lzma-rust2` backend for the real `liblzma`,
which is faster. The two match byte-for-byte on small ordinary
payloads, but that is not a guarantee across every preset: measured
on a real file at presets 0/3/6/9, only preset 3 came out identical;
the rest differ by roughly 0.02-1.4% in size, parity rather than
identity. Unlike every other codec on this page, `.lzma` defines
NO magic bytes at all: its header's first byte encodes lc/lp/pb and
the next four the dictionary size, not a fixed signature, so this
format can never be detected from bytes alone, on a file or a pipe —
see "DECOMPRESSING" below. It also has no concatenation convention
the way gzip and xz do: unlike carrying a checksum, corruption is
still detected almost everywhere despite that (measured at 4,166 of
4,170 swept byte positions) through the LZMA range coder's own
structural sensitivity, not a designed check.
stuffr pack notes.txt --format lzip -o out.lz
LZIP, a newer container around the same LZMA1 payload `lzma` above
wraps, with two differences that matter: it carries a 4-byte `"LZIP"`
magic (so, unlike `.lzma`, a `.lz` file IS detected automatically —
see "DECOMPRESSING" below), and it defines a real multi-member
concatenation convention with a mandatory CRC32/size trailer per
member, so `cat a.lz b.lz | stuffr cat --format lzip -` decodes both
members' data, not just the first. Only one backend exists for it —
the same pure-Rust `lzma-rust2` xz and lzma use — so `--features
c-backed` makes no difference to this row at all, unlike zstd's or
xz's. Validated directly against the reference `lzip` 1.26 tool in
both directions, including the two-member case.
stuffr pack notes.txt --level 9
--level asks the codec for a tradeoff: gzip accepts 0 (fastest,
largest output) through 9 (slowest, smallest output). xz's range is
the same shape, 0-9, with 6 as its default too, and lzma and lzip
share that exact range too — both wrap an LZMA1 payload under the
hood. bzip2's range is 1-9, not 0-9 — it has no "store" level, so
--level 0 is a usage error (exit 2) rather than meaning "don't
compress". brotli's range is 0-11, and unlike every other codec here
its OWN default is 11 (the slowest setting it has), not something in
the middle of the range. zstd's range is the widest and the only one
with negative numbers — -131072 through 22 on this build — where
negative levels are zstd's "fast" modes, faster than level 1 at the
cost of ratio; its default is 3. lz4 and snappy have no level at all —
neither underlying crate exposes such a knob — so --level is accepted
and silently ignored rather than rejected, unlike every other codec on
this page. `compress` (see "LEGACY FORMATS" below — one of the three
legacy formats that writes) is the one codec whose range is not
0-something: `9..=16`, `compress`(1)'s own `-b` (`maxbits`) flag, 16 by
default; a value outside that range is the same usage error (exit 2)
as bzip2's `--level 0` above.
stuffr pack notes.txt --force
Without --force, pack refuses to overwrite an existing output: exit
code 2, the existing file left untouched, nothing written.
COLLECTING FILES INTO AN ARCHIVE
stuffr pack one.txt two.txt -o bundle.tar
pack takes PATHS..., not a single path: an output naming a CONTAINER
(by extension, or with --format tar) collects every path into one
archive, one entry each. Each entry is stored under its path's final
component, so `stuffr pack /etc/hosts -o x.tar` writes an entry named
`hosts` — storing the absolute name would produce an archive stuffr
itself refuses to extract, at exit 7.
stuffr pack notes.txt --format tar
A single path still derives its own output name: notes.txt.tar.
Several paths have no one name to derive from, so they need -o.
stuffr pack one.txt two.txt -o bundle.tar.gz
stuffr pack one.txt two.txt -o bundle.tgz
A container INSIDE a codec, written in one step. -o names the whole
chain: the container comes from the inner extension and the codec
from the outer one, with the compound spellings (.tgz, .txz, .tbz2,
.tzst) read as the aliases they are. The codec's sink IS the
container's destination, so a single pass writes the entries, the
container's trailer and the codec's trailer in that order — nothing
is staged through a temporary file. --format names the CONTAINER
only, so `--format tar -o bundle.tar.gz` resolves to the same chain
rather than to a bare tar under a misleading name. When --format and
the -o name disagree about the CONTAINER (`--format zip -o x.tar.gz`)
pack refuses at exit 2 rather than picking one: the bytes would be
valid gzip, so pack would exit 0 and `stuffr list` on that same path
would then exit 5 calling stuffr's own output corrupt — a reader takes
the outer layer from the magic and the container from the name.
stuffr pack proj -o backup.tar
stuffr pack proj -o backup.tar.gz
A DIRECTORY is walked: every file, subdirectory and symlink under it
becomes an entry, named beneath the directory's own final component
(proj/src/main.rs, not the path as typed). Depth-first, with each
level sorted by name, so the same tree produces the same bytes twice
— readdir order is filesystem-dependent and is not alphabetical. An
empty directory is a real entry and survives the round trip.
Two symlink rules, opposites on purpose, both of them tar's: a link
NAMED on the command line is FOLLOWED (`stuffr pack link-to-notes.txt`
stores the file it points at, because storing the link itself would
produce an archive `stuffr unpack` refuses at exit 7), while a link
found INSIDE the walk is stored AS a link and never followed. That
second rule is also why there is no cycle detection: a loop cannot be
entered in the first place.
What the walk cannot store, it names rather than swallowing: a
socket, fifo or device node; a filename or symlink target that is not
valid UTF-8 (storing a lossy substitute would silently write a
different name, or point a restored link elsewhere); a directory it is
not allowed to LIST, which is stored while its contents are not; and a
file it is not allowed to OPEN, which is not stored at all. Neither
permission case fails the pack: one unreadable file in a home
directory must not cost the whole backup, and losing that one file
beats losing all of them. Nothing here is silent — each is named in
the fidelity report, and --strict-fidelity turns any of them into exit
4. An i/o ERROR part-way through a payload still fails the archive:
there is no honest way to finish an entry whose bytes stopped coming.
A file that merely CHANGED SIZE under the walk is different, and is
now forgiven too — a live system truncates and appends to files while
a backup runs, and the entry's header was written from a `stat` that
happened earlier. A file that shrank is padded with zeros to the
length its header declares; one that grew stops there. Either way the
archive stays correctly framed and valid, the entry is named in a
fidelity warning with both lengths, and --strict-fidelity still
refuses. GNU tar's `file changed as we read it` is the same bargain;
before this, one touched file aborted the pack and left no archive at
all. Hardlinks are reported too: pack never writes a hardlink entry,
so two names for one inode extract as two independent files. (convert
into a tar does keep the links an archive already holds; see
CONVERTING.)
stuffr pack . -o backup.tar
`.` and `..` work, and are named the way every other path is: by the
final component of the directory they RESOLVE to. Run from
/home/me/proj, `pack .` writes proj/src/main.rs — the same entries
`pack ../proj` would write, not a set of unprefixed ones. `tar cf
x.tar .` is what every tutorial teaches, so this is the likeliest
first thing to type at a directory walk; refusing it (which `.` having
no final component made it do) was a bug, not a rule. A trailing slash
is noise: `pack proj/` and `pack proj` are the same command.
`backup.tar` itself is excluded from the walk, so re-running this
exact command with --force does not nest the previous archive inside
the new one — GNU tar's `file is the archive; not dumped` for the
identical shape. It is reported on stderr every run, as a NOTE rather
than a fidelity warning, and --strict-fidelity does NOT fire on it: a
fidelity warning means something you asked for was lost, and stuffr
declining to put a file inside itself is not that. So
`pack . -o backup.tar --force --strict-fidelity` in a nightly job
stays at exit 0 run after run, and still exits 4 the moment anything
real is lost.
stuffr pack proj -o proj.a
`ar` has no directory and no symlink concept at all, so those entries
are DROPPED rather than written as zero-byte regular files — which
would make everything beneath them unextractable, the parent being a
file. The files themselves are stored, and each dropped entry is
named in a fidelity warning. --strict-fidelity turns any of these
into exit 4, on the write side exactly as it does on the read side.
Four things pack refuses, all before anything is written: more than one
path with a codec output (a codec compresses one stream and has nowhere
to put a second); two inputs whose final components collide (the archive
could not then be extracted without --force); a path NAMED on the
command line that is neither a file nor a directory — a fifo or socket
has no entry shape, and naming one is a request, not an incidental find
inside a walk; and a plan that would hold no entries at all.
That last one is exit 2, and it exists because of the exclusion rule two
paragraphs above. `stuffr pack backup.tar -o backup.tar --force` names
only the archive being written, so the walk excludes it, nothing is left,
and pack used to replace a good archive with an empty 1024-byte one at
exit 0 — with --strict-fidelity passing, since a self-exclusion is a note
and not a fidelity loss. The refusal happens while the plan is still
being built, before the destination is opened at all, so an archive
already there is not overwritten, not truncated, not touched. The message
names the cause rather than just saying there was nothing to do.
An empty DIRECTORY is not that case and still packs:
stuffr pack empty-dir -o x.tar # exit 0; x.tar holds empty-dir/
The directory's own entry is real — lose it and the tree does not come
back the shape it went in — so the plan is not empty and nothing refuses.
--strict-fidelity is NOT one of those refusals, and the difference is the
point: a refusal is a pre-flight check that writes nothing, while
--strict-fidelity is a verdict on an archive that HAS been written. The
archive is complete, valid, and LEFT ON DISK; exit 4 says only that
something the walk met could not be stored in it. Deleting it would throw
a whole good backup away over one unreadable file, which is exactly what
making that skip a warning was for — and it is the same contract the read
side keeps, where `unpack -C --strict-fidelity` exits 4 with the extracted
files still there. So read the exit code; do not read the file's absence.
A fifth refusal, for the same reason the extract side refuses flags it
cannot honour: --threads, --turbo and --allow-weak-encoder are refused
(exit 2) only where the resolved chain has NO codec layer to give them
to. A bare container is that case. `-o bundle.tar` compresses nothing,
so there is no worker count to hand over, no CPU cap to lift and no
fallback encoder to consent to, and accepting the flag there would
promise behaviour that never happens.
A composed output is NOT that case and is not refused. `-o backup.tar.xz`
really does reach xz's parallel encoder:
stuffr pack proj -o backup.tar.xz --threads 4 # honoured, exit 0
stuffr pack proj -o backup.tar --threads 4 # refused, exit 2
STUFFR_THREADS is read on that same composed path, and deliberately: the
environment is consulted exactly where the flag is honoured, never where
it is refused. --level reaches the codec layer there too.
What that costs is worth knowing rather than being surprised by. A
parallel xz or lzip encode splits its input per worker, so asking for
threads changes the bytes: `--threads 4` and no --threads at all produce
two different, equally valid archives from the same tree. The
reproducibility promise is the same one `stuffr info` states — same
input + same flags + same environment — and omitting every threading
flag builds no governor at all, so a plain `stuffr pack proj -o
backup.tar.xz` is byte-for-byte repeatable.
stuffr pack one.txt two.txt -o bundle.ar
stuffr pack one.txt two.txt -o bundle.cpio
ar and cpio (newc only) collect files the same way tar does — pack
infers the container from -o's extension (.a or .ar for ar, .cpio for
cpio) or from --format. Both are simpler than tar: neither carries a
trailing index or a per-entry codec, and they differ on directories:
cpio records one through its mode's type bits, ar has no directory
concept at all and drops the entry with a warning (see above). cpio's
per-entry size field is a hard 4 GiB (u32)
ceiling — an entry larger than that is refused outright, naming the
limit, never silently truncated to the wrong length. ar has no such
ceiling.
stuffr pack one.txt two.txt -o bundle.zip
zip collects files the same way, and is the first container here whose
INDEX is at the end of the file rather than beside each entry. Two
consequences worth knowing. Writing: a zip local header has to carry
each entry's CRC and compressed size, neither of which is known until
the entry has been compressed, so stuffr holds one entry in memory and
streams out everything before it — the archive is never buffered
whole, but a single huge entry does cost its own compressed size in
RAM. Reading: see "LISTING AND TESTING" below, because a zip on a
pipe is genuinely worth less than a zip in a file.
Entries are deflate-compressed by default. Directory entries carry
the trailing "/" the format identifies them by, and symlinks are
stored the way Info-ZIP stores them, so `unzip` restores them as
symlinks rather than as files holding a path.
stuffr pack one.txt two.txt -o bundle.zip --level 9
--level reaches a zip's entry compressor. deflate's range here is
1-9, NOT 0-9 like a bare `.gz`: zip's deflate writer has no "store"
level, so `--level 0` is a usage error (exit 2) naming the codec,
refused before the output is touched — the same shape as bzip2's
own missing level 0 under "COMPRESSING A FILE" above.
DECOMPRESSING
stuffr unpack notes.txt.gz
Writes notes.txt, stripping the recognised extension. As with pack,
an existing destination is refused unless --force is given too.
stuffr unpack notes.txt.gz -o out.txt --force
Choose the name and allow overwriting an existing one.
stuffr unpack data.bin --format gzip -o data.txt
--format names the codec explicitly instead of detecting it. Some
formats carry no magic bytes and no conventional extension, so an
explicit --format is the only way to reach them.
stuffr unpack raw.bin --format deflate -o data.txt
Raw deflate has no header, no trailer and no conventional extension,
so it cannot be detected at all — --format is the ONLY way in.
stuffr unpack data.br --format brotli -o data.txt
brotli defines no magic bytes at all, unlike gzip/zlib/bzip2's header
bytes — detection from a file works only by the `.br` extension. A
brotli stream arriving on a pipe, with no filename to go by, cannot be
detected either way, so --format brotli is the only way in there too.
stuffr unpack data.lzma --format lzma -o data.txt
LZMA1 defines no magic bytes either, the same as brotli — detection
from a file works only by the `.lzma` extension, and a `.lzma` stream
arriving on a pipe, with no filename to go by, cannot be detected at
all, so --format lzma is the only way in there too. No special build
is needed — every build has an LZMA1 backend.
stuffr unpack data.lz
LZIP, unlike its older sibling LZMA1 just above, DOES carry a 4-byte
`"LZIP"` magic at offset 0 — so this is detected automatically, on a
file or a pipe, and needs no --format at all.
STREAMING TO STDOUT
curl -s https://example.com/log.gz | stuffr cat - | grep pattern
cat decompresses straight to stdout without ever naming an output
file — the shape for feeding a compressed stream into another tool.
stuffr cat data.bin --format gzip
cat takes --format for the same reason, and --max-ratio to bound a
decode that would otherwise stream an enormous file to stdout.
curl -s https://example.com/logs.tar | stuffr cat - app.log
Naming an entry makes cat entry-aware: it streams that entry's bytes
and nothing else, in archive order, to stdout — over a pipe, with no
seeking and nothing written to disk. Without an entry name cat decodes
its input as a single stream, so `stuffr cat backup.tar` is a usage
error (exit 2) that names what to do instead. --max-ratio bounds this
exactly as it bounds extraction below, and --memory-limit bounds the
codec layer beneath the container (see "EXTRACTING AN ARCHIVE" below).
--format is refused alongside an entry name rather than accepted and
ignored: entry-aware streaming detects the container, and any codec
above it, from the archive itself. `--index N` asks for the same
entry-aware streaming by POSITION rather than by name — see --index
under "EXTRACTING AN ARCHIVE'S ENTRIES" below.
CONVERTING BETWEEN FORMATS
stuffr convert IN -o OUT [flags]
stuffr convert IN OUT [flags]
Two spellings of one command: the destination is either -o OUT or a
second positional OUT. Giving both, or neither, is a usage error
(exit 2). Nothing is unpacked to disk on the way.
stuffr convert backup.tar.gz -o backup.tar.zst
Codec mode: the container is kept byte for byte and only the
compression around it changes. Every codec layer of the input is
peeled (a gzip over an xz decodes fully) and at most one is written,
as with pack. Nothing is parsed inside the tar, so nothing can be
lost and the conversion is exact. IN's chain is detected from its
CONTENT, never its name: a `.tar.gz` that is really gzip over xz
over tar converts correctly. The target chain comes from OUT's name
(compound spellings like `.tgz` and `.tbz` included) or from
--format. On a build whose zstd encoder is the weak fallback, a zstd
target needs --allow-weak-encoder, exactly as pack does.
stuffr convert notes.txt.gz notes.txt.xz
stuffr convert backup.tar.xz backup.tar
stuffr convert notes.txt.gz notes.txt
The same mode on a plain stream, and with no codec at all on the
target side: the last two only remove one. The summary on stderr
names the TARGET chain — `-> tar over zstd`, `-> tar`, `-> raw`.
How the two chains decide what happens:
same container (or plain on both sides), codec changes codec mode
identical chain, e.g. .tar.gz -> .tar.gz exit 2,
"nothing to convert" — unless --level, --threads or --turbo asks
for a re-encode, which makes it codec mode again. A chain with no
codec (.tar -> .tar) is always refused: there is nothing to
re-encode
plain stream <-> container (.gz -> .zip, .tar.gz -> .gz) exit 2,
naming both chains: a plain stream has no entries to put in an
archive, and an archive's entries do not fit in a plain stream.
`stuffr pack` / `stuffr unpack -C` are the verbs for those
different containers (.tar.gz -> .zip) entry mode
stuffr convert backup.tar.gz -o backup.zip
stuffr convert release.zip release.tar.xz
curl -s https://example.org/src.zip | stuffr convert - -o src.tar.xz
Entry mode: the container changes, so every entry is read from IN
and written into OUT — nothing is unpacked to disk. A target that
cannot hold something behaves exactly as pack does: a directory in
an ar, or a symlink in a container without symlink entries, is
skipped with a warning, a device or fifo is never stored, and the
rest is written (exit 0; exit 4 with --strict-fidelity, OUT still
written). A zip records no owner, so converting one into a
container that stores uid/gid reports a missing-uid_gid warning for
every entry (the entries are written owned by 0:0). Converting into
a zip keeps the FIRST entry of a repeated name and skips the rest
with a warning; other targets keep repeats in order. Names are
copied verbatim — convert extracts nothing, so a `../x` stays
`../x` and unpack refuses it later, as it would from IN. A zip on a
pipe is spooled (memory, then a temp file) so entries another tool
streamed with data descriptors come out whole; an entry of unknown
size past the spill limits is exit 6. A source entry that delivers
fewer or more bytes than its header declared is exit 5 and leaves no
OUT. --strict-fidelity judges the warnings, never how IN was read: a
tar piped on stdin reads forward-only at full fidelity, and a
conversion that lost nothing is exit 0. For a damaged IN, run
`stuffr salvage` first, then convert what it recovered.
Hard links in IN (tar and cpio carry them) are kept as links when
OUT is a tar. Into a container without link entries each link is
written as a full copy of its target's bytes, which loses nothing:
for a tar source, from a memory window of the most recent payloads
(64 MiB); for a cpio source, which announces its linked files, from
exactly those. A link to a symlink is written as that symlink when
OUT stores symlinks. A link whose copy is not available (its target
past the window, never written, or a symlink OUT cannot store) is
skipped with a warning naming it and why — exit 4 under
--strict-fidelity.
stuffr convert backup.tar.gz backup.tar.gz --level 9 --force
Refused (exit 2), --force or not: OUT is IN. So is the same file reached
as `./backup.tar.gz`, through a symlink, or through a hard link —
convert reads its input while it writes, so it never converts a
file onto itself. Write to a new name and rename afterwards.
stuffr convert notes.txt.gz -o - --format xz > notes.txt.xz
cat backup.tar.gz | stuffr convert - -o backup.tar.xz
`-` works at both ends. Stdout has no name to take a target from, so
-o - needs --format; stdin is read exactly once. --format names a
codec or a container, and one OUT's name contradicts is refused
(exit 2) as pack refuses it: `convert a.tar.gz -o b.tar.xz --format
zstd` would write a zstd file under a name promising xz. A codec
--format against a stdout target writes a plain stream, so it fits
only a plain input. For a container to stdout, strip the codec with
convert and put the new one on with pack:
stuffr convert a.tar.gz -o - --format tar | stuffr pack - -o - --format xz
stuffr convert big.tar.gz -o big.tar.xz --threads 0 --memory-limit 2G
--threads, --turbo, --level and --allow-weak-encoder mean what they
mean for pack and act on the target codec. Where the target has no
codec (`-o backup.tar`, `-o notes.txt`), --threads, --turbo and
--allow-weak-encoder are refused (exit 2) rather than ignored;
--level is accepted and inert there, as `pack -o x.tar --level 9`
accepts it. --max-ratio and --memory-limit bound the read side as
they do for unpack (exit 6), and --memory-limit also bounds the
encoder's workers. --force replaces an existing OUT, --no-sync skips
the fsync before it is published, and --strict-fidelity exits 4 on
any fidelity loss — which codec mode, copying the container
untouched, never has (entry mode can; see above).
Unrecognised input — bytes no probe identifies, such as a plain text
file — is a usage error (exit 2): convert changes a format, and to
compress a plain file you want `stuffr pack`. A corrupt input is exit 5
and leaves no OUT behind: everything is decoded before OUT is
published, so a gzip trailer that fails its CRC on the last byte still
costs you nothing on disk. That promise is a PATH destination's only:
with -o - the bytes already written to stdout cannot be taken back, so
a late failure leaves a truncated stream downstream and only the exit
status says so. Check it.
A codec with no magic bytes (brotli, raw deflate) nested INSIDE another
codec cannot be detected by content, so it is not peeled and stays in
the output: gzip over brotli named `x.gz`, converted to `x.xz`, is xz
over brotli. unpack and cat behave the same way, and so does entry
mode: such a layer beneath the container is not peeled either. And whatever sits
beneath it is invisible too — gzip over brotli over tar reads as a plain
gzip stream, so converting it to `.tar.xz` is the plain -> container
refusal above. As the OUTERMOST layer such a codec is still found by
IN's extension (`.br`).
STDIN AND STDOUT VIA `-`
`-` means the stream, wherever a path is expected: stdin for an input,
stdout for an output.
cat notes.txt | stuffr pack - -o notes.txt.gz
Stdin has no path to derive a name from, so -o is mandatory here;
leaving it out is a usage error (exit 2).
cat notes.txt | stuffr pack - -o - | wc -c
Both ends can be `-` at once: stdin in, stdout out.
INSPECTING A STREAM WITHOUT DECODING IT
stuffr info notes.txt.gz
Reports the format, the resolved chain, the fidelity "rung", and the
size when it's known.
stuffr info notes.txt.gz --json
The same report as machine-readable JSON, for scripting.
A plain file reports `rung: exact` — stuffr can seek it, so what it reports
is verified. A pipe (`stuffr info -`) reports `rung: forward-only` — a
stream can only be read once, so this is the tool telling you the format
was inferred from bytes it saw going by, not confirmed by looking back.
The heading is literal: info does not decode, and it does not OPEN an
archive either. So it reports `fidelity: not evaluated` instead of
"nothing approximated" wherever a forward read of that container could
genuinely have lost something and info has not looked. Today that means
zip, the only container here that keeps its index at the END of the
stream, whenever that zip will not be read authoritatively (run `stuffr
test` for the real answer; see "LISTING AND TESTING" below). Two shapes
qualify:
* a zip on a pipe, and
* a zip under a codec — `bundle.zip.gz` — however seekable the FILE
is. The rung reported is the raw input's, and a seekable file says
nothing about a container read out of a gzip decoder: that decoder
is forward-only by construction, and `stuffr test bundle.zip.gz
--strict-fidelity` on the same bytes exits 4 with two warnings.
Everywhere else the claim is true and info still makes it: a seekable
bare zip read its central directory, a tar, ar or cpio has no trailing
index to miss piped, wrapped or not, and a bare codec stream has no
container at all. In `--json` the same distinction is the
`fidelity_evaluated` field, because `"warnings": []` cannot carry it.
One limit worth knowing: on a PIPE with no filename, info resolves only
the outer layer. `cat bundle.zip.gz | stuffr info -` reports `chain:
gzip`, because seeing the zip inside would mean running the decoder,
which info does not do. It reports honestly on what it identified.
LISTING AND TESTING AN ARCHIVE
stuffr list backup.tar
Prints one line per entry — index, size, then name — and extracts
nothing at all: no file is written, no directory created. `ls` is an
alias for `list`, for the muscle memory. Both work on `backup.tar.gz`
and `backup.tgz` exactly as they do on a bare `.tar`: the codec layer
(gzip) is resolved and decoded first, then `list` walks the tar
layer underneath it — on a real file or over a pipe (`stuffr list -`)
alike.
The first column is the entry's position in ARCHIVE ORDER, counting
from 0, and it is what `cat --index` and `unpack --index` select by
(see --index under "EXTRACTING AN ARCHIVE'S ENTRIES" below). One
forward walk
produces both the listing and the numbering, so `list` and `--index`
cannot disagree about which entry is which.
stuffr ls backup.tar.gz --json
`--json` emits a JSON array of `{index, name, kind, size}` objects
instead of one line per entry, for scripting. `kind` is `file`, `dir`,
`symlink` or `other`, and `index` is the same 0-based archive position
the human listing prints.
stuffr list photos.zip --strict-fidelity
`list` reports what the read approximated or lost, on stderr, and
--strict-fidelity turns any such report into exit 4 — the same
contract `test` and `unpack -C` already had, through the same printer.
It matters most on exactly the archive whose LISTING is incomplete: a
zip whose central directory holds 8 records under 6 distinct names
lists six entries, and the two shadowed records are unreachable. Before
this, `stuffr test` said so and `stuffr list` — the verb you reach for
first — printed those six lines and stayed silent.
stuffr test backup.tar
Reads every entry's data all the way through — not just its header —
verifying integrity without writing anything to disk. A `test` that
only walked headers would pass on an archive whose payloads are
corrupt; this build actually pulls the bytes, into nowhere. Exits 5 on
a corrupt or truncated archive, the same code `unpack`/`cat` use for a
damaged codec stream, so a script branches on "corrupt" the same way
regardless of which command found it.
stuffr test backup.tar --strict-fidelity
Fails with exit 4 if the read approximated anything at all. A tar over
a pipe is NOT such a case: it lands on the forward-only rung, but tar
has no trailing index for a forward read to skip, so nothing is lost
and nothing is reported. The flag gates on what was lost, never on
the rung.
stuffr list plain.txt.gz
stuffr list plain.txt
Both fail with exit 2, because both are the same mistake: you pointed
an entry-aware verb at something that has no entries. The first is a
bare codec stream, and the error names what the input actually
resolved to (here, `gzip`) rather than a bare refusal, so "oh, that's
not a tar, it's just gzip" is diagnosable without inspecting the file
by hand. The second is a file of no recognised format at all, and the
error prints its leading bytes. A file whose format is AMBIGUOUS is
exit 2 as well, and its message names the candidates and tells you to
pass --format.
stuffr list bundle.ar
stuffr test bundle.cpio
list, test, cat and unpack -C all work identically on ar and cpio:
each is a plain forward read, the same shape as tar's own. Neither
format has a trailing index to lose on a pipe, so `--strict-fidelity`
passes a piped ar or cpio archive for the same reason it passes a
piped tar.
stuffr list legacy.cpio
Exit 3, not 5, if that archive is `odc` (magic 070707) or the CRC
variant of `newc` (magic 070702). Both are perfectly valid cpio
archives that this build cannot read — the crate it uses implements
`newc` (070701) alone — so they are recognised by magic and refused
as a capability limit, naming the variant and how to convert it.
Calling an intact file corrupt would be a lie; exit 5 is reserved for
damage. Bytes that are not cpio at all are still exit 5.
stuffr test photos.zip --strict-fidelity
cat photos.zip | stuffr test - --strict-fidelity
The first succeeds and the second fails with exit 4, and that is the
whole point of the flag. A zip keeps its authoritative metadata in a
central directory at the END of the file: given a real file stuffr
reads it (`stuffr info` says rung "exact" — random access to any
entry, unix modes present, symlinks reported as symlinks), and given
a pipe it cannot, so it reads the inline local headers instead. That
yields the same DATA and a strict subset of the metadata — no modes,
no entry count, and a symlink indistinguishable from a small file
holding its target. Both losses are reported by name, which is why
--strict-fidelity fails here and passes on a piped tar: it gates on
what was lost, never on the rung. (`list`/`ls`, `test` and `unpack`
all take the flag; `cat` does not, because its output IS the data and
a caller piping it somewhere wants the bytes or an error, not a
verdict on metadata.)
stuffr test archive.zip
Verifies every entry's CRC32 — zip carries one per entry, so
corruption is really detected rather than inferred. Exit 5 for a
damaged or truncated archive, including one cut anywhere inside its
central directory: a forward read stops at the index rather than
reading it, so stuffr checks that the index is whole itself. Bytes
appended AFTER a complete archive are ignored, as every zip tool
ignores them.
stuffr test encrypted.zip
Exit 3, not 5: an encrypted entry is a capability this build does
not have, not a damaged file, and the message says which. Exit 3 is
"this build cannot do that" throughout — the same code you get for
naming a format the build was compiled without. The same answer
covers a zstd-compressed zip ENTRY on a default build: the one entry
codec that needs a C toolchain, so the refusal names `--features
c-backed`. Note the asymmetry there — a standalone .zst file reads
fine on every build; it is only zstd INSIDE a zip that needs the
flag. An LZMA zip entry is the reverse, readable everywhere and
writable nowhere, and reports as exit 3 too.
stuffr list bomb.tar.gz --max-ratio 100
stuffr test bomb.tar.gz --max-ratio 100
`list` and `test` both resolve the codec layer beneath the container
(gzip, for `.tar.gz`/`.tgz`) before the container ever sees a byte, so
that layer is bounded the same way `unpack`/`cat` already bound a
bare codec stream: past this multiple of the compressed input, the
read is refused at exit 6, naming the ratio, the limit and the flag.
Defaults to 10000, same as `unpack`/`cat` — raise it if a legitimate
archive is this compressible; a bare `.tar` (no codec layer at all)
is never affected, since nothing there is expanding.
`test` bounds each ENTRY's decoded payload too, exactly as `cat` and
`unpack -C` do — the same per-entry and running-total pair, charged
from bytes actually read. All four entry-aware verbs refuse the same
bomb at the same ratio; the verb you reach for to inspect an
untrusted archive is not the lenient one.
stuffr list bundle.tar.lz --memory-limit 512M
stuffr test bundle.tar.lz --memory-limit 512M
--memory-limit is the OTHER bound, and --max-ratio cannot stand in
for it: the pure xz, lzma and lzip decoders size a dictionary from a
value the file declares in its own header, before producing a single
byte of output, and --max-ratio counts output bytes. So a crafted
131-byte .tar.lz declaring a 512 MiB dictionary sails past any ratio
and is refused at exit 6 by this bound alone. On by default at 25% of
available RAM (cgroup-aware) for `list`, `test`, `cat` and
`unpack -C` alike — "reads nothing and extracts nothing" was never
true of the codec layer above the container.
EXTRACTING AN ARCHIVE'S ENTRIES
stuffr unpack backup.tar -C out/
-C/--directory extracts every entry into out/, creating it if it does
not exist yet. -C is what asks for entry-aware extraction: without it
unpack decodes a single stream, so pointing it at a container is a
usage error (exit 2) naming this flag. It is a flag rather than
something inferred from the input because the decision has to be made
before a byte is read — a pipe cannot be probed and then reconsidered.
stuffr unpack backup.tar.gz -C out/ docs notes.txt
PATTERNS after the input select what to extract: an exact entry name,
or a directory selecting everything beneath it (`docs` takes
`docs/a.txt` with it). Deliberately no globbing — the shell already
expands `*`, and a pattern language nobody asked for is a pattern
language to get subtly wrong. A pattern matching no entry at all is a
usage error (exit 2), never a silent success.
stuffr cat notion.zip --index 2 > notion-api.sh
stuffr unpack notion.zip -C out/ --index 2
--index selects an entry by POSITION instead of by name — the 0-based
number `stuffr list` prints in its first column, counting in archive
order. It is repeatable (`--index 0 --index 3`), and it is how you
reach an entry whose name you cannot usefully type:
* a zip whose central directory holds two records under one name, so
naming it can only ever reach one of them;
* `Makefile` and `makefile` from a case-sensitive filesystem, on a
case-insensitive one;
* any name your shell would fight you over.
Entries come out in archive order however the flags were ordered, and
a repeated index selects its entry once — the same contract PATTERNS
have. An index past the end is a usage error (exit 2) naming the
archive's real entry count, never a silent empty result.
--index and PATTERNS are two ways of saying the same thing, so passing
both is a usage error (exit 2) rather than a guess at whether you
meant the union or the intersection. On `unpack`, --index needs -C for
exactly the reason a pattern does: it selects entries, and without -C
there is nowhere for entries to go.
It works over a pipe too (`cat notion.zip | stuffr cat - --index 2`),
because the position is counted during the ordinary forward walk. On a
real file a container that HAS an index — zip is the only one here —
jumps straight to the entry instead; tar, ar and cpio are sequential
formats with no index to jump with, so they always count. Both routes
deliver the same bytes, which is the property the tests pin.
stuffr unpack backup.tar -C out/ --force
An entry whose target path already holds a file or a symlink is
refused (exit 2) unless --force is given — the same contract pack and
unpack apply to a single output file. --force REPLACES what is in the
way rather than writing through it: a symlink already sitting at an
entry's path would otherwise redirect that entry's bytes wherever it
points. An existing directory is never removed either way; entries
are written into it, and a FILE entry naming one is refused (exit 2),
--force or not. So is an entry beneath a file the destination
already held.
An archive can also contradict ITSELF on disk: a directory `x` and
later a file `x`, or a file `a` and later `a/b`. The later entry is
SKIPPED and named ("a directory `x` from earlier in this archive is
in the way", "`a`, earlier in this archive, is not a directory"),
every entry beneath a skipped path is skipped the same way, and the
rest is extracted: exit 0, or 4 under --strict-fidelity. --force
changes nothing here. A file named twice is not a contradiction; the
rule above applies, and with --force the later one wins. An entry
whose path is too long for the destination's filesystem is skipped
and named the same way.
Path containment is not optional and cannot be switched off. An entry
naming an absolute path, traversing above the destination with `..`, or a
symlink whose target resolves outside it, is REFUSED at exit 7 — never
silently sanitised into something safe, because a caller handed a
cleaned-up path has no way to know it was handed one. The refusal names
the offending entry verbatim and stops the extraction there, so a later
entry written "through" a hostile symlink is never reached. An entry
named `.` or `./` IS accepted: it names the destination directory itself,
and it is the first entry `tar cf x.tar .` emits.
An entry name (or symlink target) containing a NUL byte is refused there
too, at the same exit 7, and it is worth knowing because it is not an
attack — it is what a zeroed run inside an ordinary name looks like after
corruption. It used to reach the filesystem instead, which rejects an
interior NUL, and `unpack` then ended at exit 1: the code reserved for
stuffr itself having failed, on a name the archive chose. `list` and
`test` still exit 0 on such an archive, because neither turns a name into
a path; `salvage` skips just that entry and recovers the rest.
A symlink whose target resolves TO the destination is accepted too, for
the same reason: `sub/top -> ..` points at the extraction root, and the
root is inside the root. One step further out (`sub/top -> ../..`) is an
escape and is refused. Note the asymmetry with entry names, which is
deliberate: an entry NAMED `sub/..` is refused, because no real archiver
emits one and a name that pops back to the destination is more likely an
attack than an intent. A symlink target may use `..` only as a leading
run (`../../a/b`), never after a name (`x/s2/..`, `a/../b`): the system
resolves `x/s2` first, and if that is itself a link the `..` climbs from
wherever it points. Such a link, and one with an EMPTY target, is
SKIPPED rather than refused: it is not created, a warning names the
entry, the rest of the archive is extracted, and the run exits 0 (4
under --strict-fidelity). GNU tar's `--transform 's,^,pkg-1.0/,'`
rewrites symlink targets too, so `bin/foo -> ../lib/foo` arrives as
`pkg-1.0/../lib/foo` — that link is skipped, everything else lands. A
target that climbs out of the destination is still refused at exit 7,
and the refusal names the ENTRY, as `stuffr list` shows it.
An entry whose path runs THROUGH a symlink is refused too — a directory
component that is a link, whether an earlier entry created it or somebody
planted it in the destination beforehand. The two refusals above resolve
a name component-wise, which is what keeps them immune to a symlink
appearing between the check and the write; the operating system resolves
the same name through the link instead, and that difference is an escape
on its own.
stuffr unpack backup.tar -C out/ --strict-fidelity
Extraction restores each entry's mode and mtime — a directory's after
its children are written, since creating one bumps its parent's mtime
back to now — and reports whatever it could NOT put back.
--strict-fidelity turns any such report into exit 4 instead of a
warning, for a caller that needs "everything, exactly" rather than
"the data". `stuffr test --strict-fidelity` gates the same way on a
read. Three things extraction deliberately does not reproduce, each
reported per entry rather than passed over in silence:
* setuid, setgid and the sticky bit. An archive is untrusted input and
honouring its setuid bit hands over a privilege escalation for free,
so only the 0o777 permission bits are restored — the same default
tar and bsdtar apply for a non-root extraction.
* a symlink's own mode and mtime, which cannot be set without
following the link on this platform. Expect this one: it means
--strict-fidelity fails on essentially any tarball containing a
symlink at all. The gate firing there is honest rather than a bug —
the link's mtime genuinely was not reproduced — but it is why the
flag is opt-in, and why a caller who only wants "the data arrived"
should not reach for it.
* a device node, fifo or socket. There is no shape on disk for one
here, so the entry is SKIPPED and named, never materialised as a
regular file holding its supposed contents.
* a hard link whose target this run did not extract (left out by
--index or a pattern, or skipped) or whose target is a directory.
It is SKIPPED and named. A hard link to a file this run extracted
is recreated as a real link (or, where the filesystem refuses one,
a copy), and a hard link to a symlink becomes a symlink; neither is
a loss.
Ownership (uid/gid) is not restored and not reported: reproducing it
needs privileges an extraction does not have, and warning per entry
would fail --strict-fidelity for every archive in existence, which
would make the flag useless rather than strict.
stuffr unpack bomb.tar.gz -C out/ --max-ratio 100
--max-ratio bounds extraction twice over — per entry, and against the
archive's running total — charged from the bytes actually read rather
than the size an entry's header declares, since a header that
under-declares would otherwise walk straight through the check. Past
the limit the extraction is refused at exit 6, naming the entry. The
identical bound applies to `list`, `test` and `cat`, so the verb you
reach for to INSPECT an untrusted archive is no less bounded than the
one that writes it to disk.
--no-sync is accepted and does nothing — extraction writes entries
straight to their final paths, so there is no fsync for it to skip.
Extraction is NOT atomic, and this is the one place the two paths
differ. A single-stream decode (`stuffr unpack a.gz -o out`) writes to
a temp file and renames it into place, so a refused decode leaves no
partial file at all. An entry-aware extraction writes each entry
straight to its final path, so a refusal partway through — a bomb, an
unsafe path, a corrupt entry — leaves whatever entries already
completed, plus a partially-written one. Extract untrusted archives
into a fresh directory you can delete, and check the exit code before
trusting the contents.
stuffr unpack bundle.tar.lz -C out/ --memory-limit 512M
--memory-limit bounds the codec layer BENEATH the container, which
--max-ratio cannot see: the pure xz, lzma and lzip decoders size a
dictionary from a value the file declares in its own header, before
producing any output, and --max-ratio counts output bytes. A crafted
131-byte .tar.lz declaring a 512 MiB dictionary is refused at exit 6
rather than allocating for it. Defaults to 25% of available RAM
(cgroup-aware) on `list`, `test`, `cat` and `unpack -C` alike — the
bound is on by default, since a bound that defaults off closes
nothing. Raise it to decode a file that genuinely needs more.
RECOVERING A DAMAGED ARCHIVE (SALVAGE)
`salvage` is the one recovery-biased verb in this tool. `list`, `cat`,
`unpack` and `test` all stay exactly as uncompromising as they already
are — it is salvage's willingness to accept less-than-proven content
that buys them that freedom. Naming with --format a format this build
has no scanner for is exit 3, a statement about this build rather than
about the archive. `stuffr formats` shows which ones, and how far each
can be trusted, in its SALVAGE column.
Scanners exist for: zip, arc, zoo, lha, arj, tar, cpio, ar.
HOW MUCH TO TRUST AN `Intact` ROW IS NOT THE SAME ANSWER FOR EACH.
`Intact` always means the same thing mechanically — this build decoded
the entry's payload and it agreed with the checksum the archive carries.
What differs is how much evidence outside this project stands behind the
code that read it, and `salvage` exists to tell you which recovered
bytes you can rely on, so the difference belongs here rather than only
in the source:
* arc and zoo — strongest. Their test archives are a borrowed corpus,
and the checksum in each archive's own header was computed by a
DOS-era archiver decades ago with no knowledge of stuffr. Agreeing
with that number is genuinely independent evidence.
* lha — strong. Checked against `lhasa`, a live external decoder
sharing no code with the reader used here.
* arj — stuffr alone. No `arj` or `unarj` binary is obtainable on any
platform in reach, so the test archive, the reader, the writer, the
salvage scanner and the checksum comparison are all stuffr's reading
of an unofficial specification plus one Rust crate's parser. A
misreading shared between those two leaves every test in this
project green. It has never been checked against an implementation
that was not ours. Treat an `Intact` arj row as this build's best
reading rather than a second opinion, and keep the damaged original
where the bytes matter.
zip is not in that list because it needs no such caveat: it is checked
against Info-ZIP on this machine and in CI, and the whole world writes
it.
tar records no checksum over an entry's CONTENT, only over each
512-byte header — so a tar entry is never `Intact`. One whose header
checks out and whose every declared byte is present is `Complete`, and
that is as far as the format's own evidence goes: a payload byte
flipped after the archive was written is invisible to it, exactly as
it is to `tar` itself. A tar header whose magic field holds anything
but `ustar` or zeros, or a pre-POSIX (v7) header whose checksum field
is spelled unlike any known writer's, is one the scan cannot tell from
a real header seen a byte late; it refuses it, and a run that recovers
nothing else says so at exit 3 rather than "nothing recoverable" —
`stuffr list` and `stuffr unpack` still read those headers. A run that
DOES recover other entries names every such header on stderr, with its
offset (`salvage -> this build's salvage scanner saw 1 tar header(s)
whose magic field ... at offset(s) 0`), and exits 4, not 0: the header
is an entry `list` may read that this run did not recover, so `salvage
-C out && rm damaged.tar` must not go ahead. It is never a row. A run
narrowed with --index is about the positions it named and keeps its
own code; the note still prints.
cpio (`newc`) records NEITHER checksum — nothing over the content and
nothing over the header — so every cpio entry salvage recovers is
`Unattested`: written under its real name, tagged on its `--list` row,
and the run exits 4, because nothing in the format can prove a
recovered record is a real entry rather than a coincidental header
match. The scan's gate is plausibility alone (the `070701` magic,
thirteen hex-only fields, a name under 64 KiB that is NUL-terminated
UTF-8): against random or compressed bytes it essentially never
fires, but a real cpio stored inside a damaged entry's payload is
reported entry by entry, and nothing can tell. --strict declines every
such entry. A header of the odc (070707) or newc-crc (070702) variant,
which this build does not read, is never a row either: a run that
recovers nothing else says so at exit 3, and a run that recovers
something names each one on stderr with its offset.
ar records no checksum either, so every member salvage recovers is
`Unattested` too — and ar salvage has a LIMIT the others do not. A
member header's only fixed marker is the two bytes "`\n", far too weak
to search a damaged archive for (it would report noise as members), so
ar salvage WALKS the archive from its start, member by member, exactly
as `stuffr list` does, and stops at the first header it cannot read.
That recovers a truncated tail fully: every member before the cut, and
the cut one's surviving bytes as NAME.partial. It cannot cross a hole
in the middle. When the walk stops early the run says so on stderr,
--list, -C and -o alike, with the offset. Five members, the third's
header destroyed:
$ stuffr salvage --list damaged.a
salvage -> this build's ar salvage scanner walks the archive in order from its start, as `stuffr list`
does, and stopped at offset 142 with 200 bytes still to go: Invalid
timestamp field in entry header ("XXXXXXXXXXXX"). ... any entries in
those 200 bytes are unreachable by construction — not absent, and not
counted in this report
0 Unattested one.txt [no checksum and no header self-check ...]
1 Unattested two.txt [no checksum and no header self-check ...]
Two members of five means the other three were never reached, not that
they are gone: a hex editor at that offset is the next step. A file
that ends partway through a member's header, its long name or a GNU
long-name table is said to be cut short instead — the file ended inside
that member, so nothing is missing from the report. The exit code is
the ordinary one: 4 when any member was recovered (they are
`Unattested`), 5 when none was. Two healthy shapes this build cannot
read are exit 3, naming them: a GNU thin archive (`!<thin>`, whose
members live in other files), and a symbol table whose header the ar
reader cannot parse. GNU `ar` on macOS writes its `__.SYMDEF` with a
blank mode field (plain `ar rc`, `ar rcs` and GNU `ranlib` all do);
that one is read normally, by `list` and `salvage` alike. Its mode is
unknown: `list` and `list --json` print no mode for it (as Apple's
`ar t` prints none), and `unpack` writes the default 0644.
LHA and ARJ are the two where the ordinary reader has no fallback at
all, and ARJ's is the sharper of them. LHA carries no index, no entry
count and no trailer, so `list` reaches entry N only by having parsed
entries 1..N and ONE damaged header ends the archive for everything
BEHIND it. An ARJ archive opens with a main header the reader must
parse before it will hand back a single entry, so two damaged bytes
THERE cost the whole archive — `stuffr list` exits 5 over a file whose
every entry is intact. `salvage` finds each header on its own in both.
For LHA, header levels 0, 1 and 2 are scanned.
Two header shapes are not, and `salvage` now SAYS SO rather than
reporting an empty archive: header level 3 (it re-sizes every length
field to 32 bits and needs a different parser), and a level-2 header
carrying no "common" extension header (that header holds the CRC-16
this scan gates on, and without it there is nothing to tell a real
header from five coincidental bytes). Meeting either is exit 3 with a
message naming the level — a statement about this build, never about
your archive, which `list` and `unpack` still read normally. Exit 5,
"the scan found nothing recoverable", is reserved for an archive that
really does hold nothing this build can recover.
One more LHA limit: an entry using a compression method this build has
no decoder for (`-lhx-`) is reported but never written, as
`[unverified: undecodable method]` at exit 3.
ARJ has three limits of its own, all deliberate:
* An entry whose `file type` is 2 is never reported. That value
means "comment header" in an entry and "must equal 2" in the
ARCHIVE's own main header, and the two headers are otherwise
byte-for-byte the same shape — so nothing can tell them apart in a
damaged archive. Reporting a main header as an entry would claim a
payload of whatever the archive's timestamp field happens to
encode, at exit 0, on every ARJ ever written; the cost of refusing
that is that a genuine comment header is not recovered either.
Comment headers are rare and carry no file content.
* Methods 8 and 9 ("no data") have no decoder here, so an entry
using one is reported and never written, as
`[unverified: undecodable method]` at exit 3 — the `-lhx-` case in
ARJ's own spelling.
* A method 4 ("compressed fastest") entry whose compressed stream
opens with a back-reference is refused before decoding, and the
row reads `Partial (decode failed)`. The refusal is right — such a
stream would copy from a history that does not exist — and the
word is now right too: nothing is missing from the FILE, so the
row no longer says `truncated` and no longer sends you looking for
a longer copy of an archive that is not short. Nothing is
invented either way.
Three surface forms, one of which is required:
stuffr salvage broken.zip --list
Reports on the scan with NO destination at all — listing is not
recovering, and this is the diagnostic half of the feature: seeing
what an archive holds without extracting anything, not even into a
throwaway directory. Prints every record the scan found, one per
line: SCAN POSITION, status (Intact, Complete, Unattested, Partial
with its cause, or Unverified with its cause), and name. `Complete`
means "every declared byte was there and the header self-verified,
but the format offers no checksum to check the CONTENT against" —
tar, and only tar, reaches it here. `Unattested` means "the format
offers neither a checksum nor a self-verifying header, so nothing
attests this is an entry at all" — cpio and ar reach it. This can
show MORE
rows than `stuffr list` on the same archive: a zip whose central
directory declares two records under one name (`zip.rs`'s own
last-record-wins collapse — see "WHAT THIS BUILD CAN DO" below) makes
`list` show one of them and `salvage --list` show both.
A record repeating an earlier record's name is annotated one of two
ways, and they mean different things. `[shadowed: dup of #N]` is a
MEASUREMENT: name, declared length and checksum all agree with record
N, so that record is a copy and writing it again would add nothing.
`[name collision: #N uses this name too]` is the weaker fact: the
name repeats but the content was NOT proven identical, so both
records are real and both are worth recovering — `-C` writes the
second as `NAME.salvaged-N` (see below) rather than over the first.
stuffr salvage broken.zip -C out/
Scans broken.zip for zip local file headers directly, rather than
trusting its central directory (which may be truncated, zeroed, or
simply WRONG — see above), and recovers what it can prove into out/,
created if it does not exist. Default policy (--partial=keep): an
entry whose payload ran out early, or whose CRC-32 disagreed after a
full decode, is still written, but under `NAME.partial`, never its
real name — a truncated file under its real name would be
indistinguishable from a whole one to every tool downstream. A
shadowed record — one MEASURED to be a byte-identical copy of an
earlier one — is never written a second time.
A record whose name an earlier one already took, without being proven
a copy of it, is a different case and is recovered too: it lands as
`NAME.salvaged-N`, where N is its own scan position (the number in
the first column of `--list`). The same principle as `.partial`, for
the same reason — recover by default, and let the filesystem carry
the distinction. The run exits 4 and the summary's last field counts
how many names had to be changed, so a scripted
`stuffr salvage broken.zip -C out/ && rm broken.zip` is told rather
than quietly losing a record to an overwrite.
The default (non---list) report — counts only — goes to stderr, the
same destination `unpack`'s and `test`'s own summary lines use.
stuffr salvage broken.zip -o recovered.bin --index 4
Recovers exactly ONE entry's bytes to a named file — no `-C`
directory at all. Requires exactly one `--index`; refused (exit 2)
with zero, with more than one, or together with `-C`. A `Partial`
entry still lands under `recovered.bin.partial`, never the literal
name given — the same rule `-C` follows, for the same reason.
stuffr salvage broken.zip -C out/ --index 6
--index selects by SCAN POSITION — 0-based, counting every record the
raw scan found, shadowed ones included — and this is a DIFFERENT
numbering from `list`'s own index (see "LISTING AND TESTING AN
ARCHIVE" above), which never counts a shadowed record at all. On an
archive where a name is duplicated, `list --index 6` and
`salvage --index 6` can name different entries, or `list`'s may not
exist at all. Every message naming this numbering says "scan
position", never "index", so it cannot be misread as `list`'s.
Repeatable with `-C` (`-o` takes exactly one); an entirely
out-of-range request is a usage error (exit 2) naming how many scan
positions the archive actually has — the same leniency `unpack
--index`/`cat --index` already give a mix of a real position and a
typo, silently recovering the real one.
`--index` gates what is WRITTEN, not merely what is reported: an
unselected scan position is skipped before anything is ever opened
for writing, so `-C out --index 4` leaves ONLY entry 4's file in
out/, never the rest of the archive too. Combined with `--list`, the
listing still shows the FULL scan regardless — `--index` narrows the
write, never the diagnostic — with every position it excluded tagged
`[not selected]` (a shadowed position keeps its own
`[shadowed: ...]` tag instead, since a shadow is a permanent fact
about that entry, not something selection changes). The printed
summary and the exit code, unlike `--list`, DO narrow to the
selection: `--index 6` on a shadow reports `1 scanned, 1 skipped`
and exits 4 even though the rest of the archive is fine, because
entry 6 is the only one asked about.
stuffr salvage broken.zip -C out/ --partial=skip
Recovers nothing for a Partial entry instead of writing it as
`NAME.partial` — its cause (truncated, decode failed, or checksum
mismatch) is still named under `--list`, exactly as a WRITTEN
partial's is. `--partial=
ask` has no terminal to ask from at this layer, so it is read
conservatively as `skip` rather than silently upgraded to `keep`.
stuffr salvage broken.zip -C out/ --max-entry 8G
Raises the structural ceiling on a single entry's declared length for
an archive known to hold something genuinely that large. Refused
before allocation either way — this is the same "never size a buffer
from an untrusted, possibly corrupt declaration" discipline
--memory-limit and container entry limits already apply elsewhere on
this page.
An entry past the ceiling is ONE ENTRY'S problem: it is listed as
`Unverified (over the ceiling)`, nothing is written for it (not
even a 0-byte `.partial` — nobody read a byte of it, so there is no
surviving prefix to claim), and everything else in the archive is
recovered as usual, at exit 3. It used to abort the whole run at exit
6 instead, which meant one oversized declaration cost you every other
entry — in the one verb whose job is not losing entries. A container
that decodes an entry WHOLE narrows this ceiling further with its own
(ARC's is a fixed 256 MiB; see "READ-ONLY LEGACY FORMATS"), so
--max-entry can lower the figure for such a format but never raise it
past what that format will buffer.
stuffr salvage broken.zip -C out/ --strict
Demands proof: every Partial entry is skipped regardless of
--partial, and so is every Unattested one — an entry from a format
with no checksum and no header self-check, which is written under
its real name by default and which nothing can prove. Those two are
ALL it does, stated precisely because the wording here used to add
"and nothing this build cannot verify is ever written" — true, but
not this flag's doing: an Unverified entry is never written under any
policy, with or without --strict. Nor does --strict fix the ceiling;
--max-entry is honoured either way.
Salvage's own strict path, applied on request rather than by default
— unlike the rest of this tool, where uncompromising IS the default
and salvage is the one exception.
Exit codes for a COMPLETED salvage run (see "EXIT CODES" below for the
full table), computed over whatever `--index` selected (the whole scan
when it was omitted): 0 — every entry Intact or Complete, and no header
seen that the scan could not gate. 3 — any entry
Unverified (an undecodable compression method, a data-descriptor entry
with no length to reconcile against a central directory, or an entry past
the ceiling --max-entry sets) — outranks 4 deliberately, since it usually
names its own remedy (a rebuild, --features c-backed, or a higher
--max-entry) where 4 only says "degraded". 4 — any entry written
as `.partial`, any entry written under a disambiguated
`NAME.salvaged-N`, or any entry skipped for any other reason (a shadow,
--partial=skip, a kind this build has no shape for, or a name this
filesystem will not hold), or any entry whose status is Unattested (the
format carries neither a checksum nor a self-verifying header, so the
entry IS written under its real name and the run says so — there is no
flag to opt out of, and no rebuild that would change it, which is why it
is 4 and not 3), or — when no --index narrowed the run — any header the
scan saw and could not gate (named on stderr; `stuffr list` may read
it, and this run did not recover it). 5 — the scan found
nothing recoverable in the archive at all, regardless of `--index`. 7 —
any entry refused by containment (a hostile path escaping -C or -o, an
empty entry name, a symlinked ancestor), which OUTRANKS every code above
it. Exit 6 no longer reaches `salvage` at all: an entry over --max-entry
is reported and skipped like any other, never allowed to end the run.
An entry whose NAME the destination filesystem refuses — a component past
its limit (255 bytes nearly everywhere), a name colliding with something
it cannot replace — is skipped the same way. Every such entry is named
on stderr as the run happens (`salvage -> N entries could not be
written:`, then one line each), and tagged on its own --list row as
`[not written: ...]`. It used to end the run at exit 1, losing every
entry after it, on a name the ARCHIVE chose.
A name that would escape -C is refused too, and NOTHING is ever written
under it — but since the final fix wave it is refused the same way: that
one entry is skipped with `[refused: unsafe entry path: ...]` on its row
and on stderr, every later record is still recovered, and the run's exit
code is 7. This text said the opposite for a whole phase ("aborts the run
outright, exactly as it does for `unpack -C`"), and that claim was made by
analogy rather than measured. `unpack` aborting is right, because it has
no per-entry vocabulary to report a refusal with. `salvage` does. On a
228-byte zip whose MIDDLE record simply has an empty name — one corrupted
length byte, not an attack — the old behaviour wrote the first entry,
stopped at exit 7, never attempted the third, and printed no summary at
all, while `--index 0 --index 2` recovered both. Not being able to write a
file here is still not the same as not being allowed to: the two are told
apart by the row's own wording and by the exit code.
WHAT THIS BUILD CAN DO
stuffr formats
Lists the formats compiled into THIS binary, not everything stuffr will
ever support. A default build has twenty rows: twelve
round-trip codecs — gzip, zlib, deflate, bzip2, brotli, lz4, snappy,
zstd, xz, lzma, lzip and `compress` — and eight containers, six of
them round-trip (tar, ar, cpio, zip, lha, arj) and two read-only
(arc, zoo). Counted the other way, that is the eleven pure-tier
codecs and four pure-tier containers, plus
the five legacy formats — the `compress` codec
(read AND write, as of Phase 3c Task 5), the `lha` and `arj`
containers (read AND write, as of Phase 3c Tasks 6 and 7 — but see
the WARNING under "LEGACY FORMATS" below: `lha` READS seven
compression methods and WRITES one, and `arj` READS five and
WRITES only the uncompressed one) and the READ-ONLY
`arc`/`zoo` containers, added to `default`
alongside `pure` so a plain `cargo install`/`cargo build` shows all
twenty, not just fifteen — see "LEGACY FORMATS" below.
The KIND
column tells codecs and containers apart. `--features legacy` (bundled
into `full`) still exists, but only narrows a build now: it is what
`--no-default-features --features pure` would otherwise lack, not
something a default build needs to opt into. zstd's row
differs by which backend is
compiled in: a build with the `zstd-c` feature (needs a C toolchain)
shows `yes` in WRITE, using the real `libzstd`; a pure-Rust build
falls back to `ruzstd` instead, and shows `weak` in WRITE — see
"USING A WEAK FALLBACK ENCODER" below.
Either way the format still reads `.zst` unconditionally; only writing
it without the C backend needs the flag. xz's and lzma's rows do NOT
have this shape: both backends show `yes` in WRITE, never `weak` —
`--features c-backed` swaps the pure-Rust `lzma-rust2` backend for the
real `liblzma`, which is faster (see "COMPRESSING A FILE" above) but not
more capable. lzip's row is unaffected by `c-backed` entirely — it has
only ever had the one, pure-Rust `lzma-rust2` backend, so this feature
changes nothing about it, in either direction. Every build lists all
twenty of those rows unless `--no-default-features` narrows it — the
ONLY capability difference `c-backed` makes anywhere in this matrix is
zstd's WRITE column, `weak` against `yes`; the five legacy rows sit
outside that comparison entirely, `c-backed` changing nothing about any
of them — `compress`, `lha` and `arj` show `yes` in WRITE on every
tier (there is only ever the one implementation of each), and
`arc`/`zoo` stay `-`,
read-only on every tier — see "LEGACY FORMATS" below.
The PARALLEL column is truthful per build too, and independent of
the WRITE column above: on the default (`pure`) tier it reads `yes`
for `xz` and `lzip` and `-` for every other format, including
`zstd` — `ruzstd` has no multi-threaded encoder to hand a worker
count to. Adding `--features c-backed` flips `zstd` to `yes` as
well, alongside its WRITE column becoming `yes`; it changes nothing
else in this column, on any tier. All eight containers — tar, ar,
cpio, zip, lha, arj, arc and zoo — always show `-` here: none of
them parallelises anything, on any tier (`arc` and
`zoo` trivially so — neither has an encoder to parallelise
in the first place; `lha` and `arj` have one each and both are
single-threaded — and `arj`'s does not compress at all).
The SALVAGE column says what `stuffr salvage` can do with a damaged
archive of that format, so the answer no longer has to be found by
running it and reading an exit 3. `yes` means salvage scans it,
checks each recovered entry against a checksum the archive itself
carries (an entry can reach `Intact`), and writes what it recovers.
`weak` means it scans and writes, but the format offers nothing to
verify an entry's content against: tar checksums only each header
(its best is `Complete`), and cpio and ar checksum nothing at all
(`Unattested`, exit 4) — see "RECOVERING A DAMAGED ARCHIVE" above.
`-` means no scanner, which is every codec: salvage works on archives,
and a bare `.gz` is refused at exit 2 as not one.
SALVAGE reads yes for: zip, arc, zoo, lha, arj.
SALVAGE reads weak for: tar, cpio, ar.
GUARDING AGAINST DECOMPRESSION BOMBS
stuffr unpack bomb.gz --max-ratio 100
A decompression bomb is a small compressed file that expands to an
enormous one. --max-ratio refuses a decode once output has grown
past this multiple of the compressed input (default 10000). gzip's
structural ceiling is roughly 1032:1, so it cannot reach that default
alone — here the flag mostly guards oversized input, or lets a
cautious caller set a tighter bound by hand. It earns its keep once
later phases add codecs and containers that CAN reach 10000:1.
SKIPPING THE DURABILITY FSYNC
stuffr pack big.txt --no-sync
By default, pack and unpack fsync the output before renaming it into
place, so a crash cannot leave a truncated file under the real name.
--no-sync trades that guarantee for speed on bulk or scratch work.
USING A WEAK FALLBACK ENCODER
stuffr pack big.dat --format zstd --allow-weak-encoder
A build with no C toolchain writes `.zst` through `ruzstd`, a
pure-Rust fallback, instead of the reference `libzstd`. The output is
valid zstd any tool can read, but markedly larger — measured at 3.53x
on an 11.7 MB corpus where the C backend's default reaches 6.21x
(files 76% larger) — and slower (969ms against 118ms in that same
measurement), and it buffers the whole input in memory rather than
streaming it, unsuitable for large files. pack refuses such an
encoder unless this flag is given; run `stuffr formats` to see which
codecs in THIS build are affected (WRITE shows `weak` rather than
`yes`). Rebuild with `--features c-backed` to remove the limitation.
xz does not have this shape: its pure-Rust backend (`lzma-rust2`) is a
full codec at ratio parity with `liblzma`, not a weaker fallback, so
`--allow-weak-encoder` never applies to it, in any build. lzma (the
older LZMA1 "alone" format) shares xz's shape now too: its pure-Rust
backend is the same full-ratio `lzma-rust2`, so `--features c-backed`
is purely a speed choice for it as well, never a capability one.
lzip does not have this shape either, for a simpler reason still: it
has exactly one backend, period — there is no second, C-backed
implementation for it to differ from, weakly or otherwise, so
`--features c-backed` changes nothing about its row at all.
PARALLEL ENCODE FLAGS
stuffr pack big.dat --format xz --threads 4
Encode with 4 worker threads instead of stuffr's default of one. Only
THREE of this build's twelve round-trip codecs actually parallelise
their encoder: xz and lzip on every build, and zstd only with --features
c-backed (the default ruzstd encoder has no multi-threaded path).
Counted by implementation rather than format this is four
(zstd-c, xz-c, xz-pure, lzip), because xz has two interchangeable
backends — but xz-c and xz-pure register the same format and never
both compile in, so no single build ever shows more than three
PARALLEL rows in `stuffr formats`. The other eight (nine counting only
one xz backend) accept --threads without error but encode
single-threaded regardless, since their crates expose no MT encoder
to hand a worker count to. --threads is deliberately opt-in rather
than auto-detected: a multi-threaded encoder splits its input across
workers, so the very same file compresses to different bytes
depending on how many workers ran. Omitting --threads (and --turbo,
and leaving STUFFR_THREADS unset) keeps pack single-threaded, so a
given input, given the same flags and the same environment, always
produces the same output bytes — "same machine" is not part of the
promise once --threads 0 is in play, because auto-detection
resolves against cgroup CPU quotas, not just core count.
--memory-limit does not affect this: it bounds allocation and the
worker count the governor can grant, never which bytes a worker
count already granted produces. Passing --threads 0 asks stuffr to
auto-detect a worker count
instead, trading that byte-for-byte reproducibility for speed.
--threads exists only on pack; multi-threaded decode is out of scope
for this phase, and unpack/cat refuse the flag outright rather than
silently accepting and ignoring it — see
"threads_is_not_offered_on_decode_subcommands" if you want the
enforcement in test form.
stuffr pack big.dat --format xz --threads 4 --turbo
--turbo lifts the CPU cap only, letting stuffr use the full detected
core budget instead of a conservative default worker count.
--memory-limit (below) still binds regardless of --turbo: turbo
means "use my cores," not "ignore the OOM killer."
stuffr pack big.dat --format xz --memory-limit 512M
stuffr unpack archive.xz --memory-limit 512M
--memory-limit caps the memory stuffr will ask for. It accepts a plain
byte count or a binary, 1024-based K/M/G suffix (e.g. 512M or 2G) —
matching what `xz --memlimit` and every other tool in this space
mean by "M", not the decimal 1000-based reading. A malformed value
(512MB, a decimal unit, anything else this parser doesn't recognise)
is a usage error naming the accepted forms, never a silent fall back
to the default — a user who typed 512MB and got 25% of RAM instead
would have no way to notice. On pack this bounds the worker count;
unpack, cat and info accept it too, since a decode can also need to
fit inside a memory ceiling even with no threads to bound. Defaults
to 25% of available RAM, honouring cgroup limits; `stuffr info` reports
the resolved figure, humanized (e.g. "256 MiB"), on every build.
The bound has teeth: xz and lzip cost roughly 128 MiB per worker
below preset 7 and roughly 896 MiB per worker at preset 7 and above
(the match finder, not the block buffer, dominates). On the 256 MiB
floor this build defaults to when it cannot read available memory
(macOS, with no /proc/meminfo), that means `stuffr pack big.dat --format
xz --threads 8 --level 9` silently runs with exactly ONE worker —
the governor protecting the machine, not a bug — unless
--memory-limit is raised. A 4 GiB budget gets 4 workers at that same
level, or 32 (then CPU-capped) at the default level 6.
--threads, --turbo and --memory-limit build the thread governor that
pack's encode path consumes: --threads and --turbo resolve to a worker
count via the same precedence STUFFR_THREADS and a future config file will
follow, and --memory-limit resolves to the byte cap that count is
clamped against — divided by the codec's own measured per-worker demand,
never a flat division. Omitting --threads builds no governor at all —
the codec sees no worker count and encodes single-threaded, which is
what keeps output byte-reproducible by default. unpack, cat and info
accept --memory-limit and validate it the same way. info only reports
the resolved figure (it never decodes), but unpack and cat pass it
through to the decoder: a handful of pure codecs size a dictionary
buffer from a value the file declares in its own header, before
producing any output, so an unbounded decode of a hostile file can
allocate far more than the archive's own size would suggest — this is
what closed the xz-pure/lzip/lzma-pure decode denial-of-service, each
refusing at exit 6 (see "EXIT CODES" below) naming the declared size,
the limit and the flag, never the exit 5 a genuinely corrupt archive
gets. Omitting the flag on unpack/cat still binds — it resolves to the same
25%-of-RAM default as pack — because a bound that defaults off closes
nothing.
EXIT CODES
0 success
1 general failure (i/o error, or anything not listed below)
2 usage error (bad flag, missing -o, existing output without --force,
list/test pointed at something that resolved to a bare codec stream
rather than an archive, an input whose format could not be detected
at all or was ambiguous, unpack/cat pointed at a container with
neither -C nor an entry name, or a pattern matching no entry) — every
one of these is "you pointed the command at the wrong thing", and
they all share the one code
3 format not available in this build, or a `salvage` run found an entry
it could not verify (an undecodable compression method, or no
declared length to bound the payload against)
4 fidelity degraded under strict mode, or a `salvage` run left something
less than fully recovered (an entry written only as `NAME.partial`,
an entry written under a disambiguated `NAME.salvaged-N` because an
earlier record already took its name, or skipped for any reason — a
shadow, --partial=skip, a kind this build has no shape for)
5 archive or stream is corrupt, or a `salvage` scan found nothing
recoverable in the archive at all
6 resource limit exceeded: --max-ratio or --memory-limit tripped, archive
nesting deeper than the depth stuffr will peel through, or a container's
fixed metadata ceiling (not configurable): a tar pax header or GNU long
name/link over 16 MiB, an ar GNU name table over 16 MiB or BSD name over
64 KiB, a cpio name or symlink target over 64 KiB or more than 65536
held hard-link names, a zip central-directory name over 64 KiB, an
ARC/ARJ/ZOO entry over 256 MiB (a zip symlink target over 64 KiB is
currently refused as corrupt, exit 5, not 6). NOT a `salvage` entry
over --max-entry: that is reported per entry as `Unverified (over the
ceiling)` and lands in 3, because one oversized declaration must never
cost a caller the rest of the archive. Nor any of `salvage`'s own scan
work caps (tar's per-header and scan-wide extension-chain budgets; the
scan-wide verdict budget of zip, zoo, lha, arj and cpio): what a cap
skips is reported as a sighting, and a work cap never gives exit 6: the
run exits by its ordinary rule (4, 3, 7 or 5 as the entries and the
shapes it could not gate dictate)
7 unsafe path refused. In a `salvage` run it is the entry that is
refused, never the run: the rest of the archive is still recovered
and the summary still prints, and 7 outranks 3 and 4 so a caller
scripting against a hostile archive still sees it
NOT YET
A `--no-default-features --features pure` build has eleven pure-tier
codecs and four pure-tier
containers: gzip, zlib, deflate, bzip2, brotli, lz4, snappy, zstd, xz, lzma
and lzip, plus tar, ar, cpio and zip. (It is no longer what a plain
`cargo build` gives you — `legacy` joined `default`, so the default build
adds the five legacy formats on top; see "LEGACY FORMATS" above.) zstd's pure backend (`ruzstd`) is a weak fallback (see
"USING A WEAK FALLBACK ENCODER" above); xz's and lzma's pure backends are
not — both read and write full-ratio output with no C toolchain, just
slower than `--features c-backed`'s `liblzma` (roughly 1.2-1.3x to
encode, 2-2.2x to decode, measured on a 6.5 MB payload). `--features
c-backed` upgrades all three: zstd to the real `libzstd`, and xz/lzma to
`liblzma` for speed. lzip has no such upgrade path at all — it has
exactly one backend (the same pure-Rust `lzma-rust2` xz and lzma use), so
`--features c-backed` changes nothing about it, and there is no
weak-fallback distinction for it either.
All four pure-tier containers read and write, and all four entry-aware
subcommands work on each of them identically: `list` and `test` read
entries without extracting, `unpack -C` writes them to disk under the
containment rules above, and `pack PATHS...` collects files into one.
`.tar.gz` and `.tgz` resolve as tar over gzip, which `stuffr info` shows
as the chain — ar, cpio and zip have no such codec-wrapped convention of
their own, though any of them wrapped in a codec still resolves the same
way from its extensions (`bundle.zip.gz`). `ar` is the simplest of the
four: no trailing index, no directory concept, no multi-member
concatenation convention. `zip` is the only one with a trailing index,
which is why it is the only one whose forward read approximates anything
— see "IDENTIFYING A STREAM" above. `cpio` is `newc` (SVR4 "new ASCII")
only: the crate this build uses does not implement the older `odc`
format (magic 070707) or the CRC variant of `newc` (magic 070702).
Pointing stuffr at either is exit 3, not 5 — both are recognised by
magic and refused with a message naming the variant and how to convert
it, because an intact archive in a variant this build cannot read is a
capability limit, not a damaged file. cpio's per-entry size field is
also a hard 4 GiB (u32) ceiling: an entry larger than that is refused
with a typed error naming the limit, never silently written with the
wrong framing. Walking a directory tree and writing a container inside a
codec (`bundle.tar.gz`) both work in a single command now — see
"COLLECTING FILES INTO AN ARCHIVE" above. `stuffr formats` always tells
you the truth about the build you are actually running, whatever this
page says.
LEGACY FORMATS
The set from the 1980s and early 1990s. `compress`, `lha` and `arj` read
AND write (Phase 3c Tasks 5, 6 and 7); `arc` and `zoo` are READ-ONLY and
refuse `pack --format <name>` with one shared sentence at exit 3.
This section was called "READ-ONLY LEGACY FORMATS" while all of them were.
stuffr unpack backup.Z
Unix `compress`, gzip's predecessor. Behind its own `--features
compress` (bundled into `--features legacy`, itself in `default`
alongside `pure`, and also in `full`/`--all-features`) — only a build
that explicitly excludes it, e.g. `--no-default-features --features
pure`, reports `--format compress` the same as any other name that
build does not have: "unknown format `compress`; this build has: ..."
at exit 2, not a corruption error. Where it IS built in, `.Z` carries its
own 2-byte magic (`1f 9d`) at offset 0, so — like `.lz` above, unlike
`.lzma` — it is detected automatically from a file or a pipe, with no
--format needed.
This codec now WRITES too (Phase 3c Task 5): `stuffr pack notes.txt
--format compress -o notes.txt.Z` produces a real `.Z` file the actual
`compress`(1)/`uncompress`(1) tools read back byte-exactly. `--level`
maps to `compress`'s own `-b` flag (`maxbits`, `9..=16`, not the 0-9
shape every codec above uses) — an out-of-range value is a `Usage`
error (exit 2), before any destination is touched. The default
(no `--level`) is 16, the best ratio the format has. **Avoid `--level 9`
unless you specifically need it**: once its 512-entry dictionary
fills, GNU `ncompress`'s own `-b 9` output cannot even be decoded by
that same GNU `ncompress` — a genuine defect in that one encoder, not
a second legal dialect — so this build writes the BSD-shaped bytes at
that level instead, which every reference decoder handles correctly.
The `arc` and `zoo` containers below are still
READ-ONLY, and refuse `pack --format <name>` with
`Error::CapabilityUnavailable` (exit 3) from the same capability gate
`compress` itself used to answer through before this phase. `lha`
left that list in Phase 3c Task 6 and `arj` in Task 7; both write
now, each with the method caveat its own entry below spells out.
Unlike every OTHER codec in the pure/c-backed tiers above, `.Z` also
carries no length field, checksum or end-of-stream marker of its own:
a `.Z` file truncated part-way through decodes to a silently SHORTER
prefix of the original data rather than an error, on this build's
decoder AND on the real `compress`/`gzip -dc` tools alike — measured
directly, not assumed. `stuffr formats` reports it accurately (KIND
`codec`, WRITE `yes`), the same promise "WHAT THIS BUILD CAN DO" above
makes for every other row.
stuffr pack tree -o backup.lzh
stuffr list backup.lzh
LHA/LZH, behind its own `--features lha` (bundled into `--features
legacy`, itself in `default`/`full`/`--all-features`, same as
`compress` above). It was the first read-only CONTAINER, not codec,
and gained an encoder in Phase 3c Task 6. Unlike `compress`, `lha`
genuinely STREAMS: the crate behind it (`delharc`) parses forward off
a pipe natively, with no `Seek` anywhere in its decode path, so
`list`/`test`/`cat`/`unpack -C` and `--max-ratio` all work exactly the
same over a real file or a pipe — no bound-before-allocation guard
needed, unlike the ARJ container below, which needs one. Its magic
sits at offset 2, not 0 (`-lh` for the LHarc family, `-lz` for the
older LArc one this build also reads), so both `.lzh` and `.lha` are
detected automatically.
IT READS SEVEN COMPRESSION METHODS AND WRITES ONE, and that
asymmetry is the thing to know before using `pack --format lha`.
The reader accepts `-lh0-` through `-lh7-` (plus the older LArc
`-lz*-` family); the writer emits `-lh5-` — 8 KB window, static
Huffman, the method every LHA tool in existence supports — and
nothing else. So unpacking an `-lh7-` archive and packing it again
gives you a valid `.lzh` whose entries are `-lh5-`: a larger file,
no data loss, and a method an older tool is more likely to read.
`--level` is accepted and ignored, because in LHA the method letter
IS the level and this build writes one method.
A directory is stored as an `-lhd-` entry and listed with a trailing
`/`, the same convention zip follows. Symlinks are NOT stored: `pack`
warns and skips one rather than writing something a reader would
restore as an ordinary file.
ENTRY NAMES ARE NOW REPORTED EXACTLY AS THE ARCHIVE STORES THEM, and
that changed in Phase 3c Task 6 in a way that can break a script.
The crate behind the reader strips `..`, `.` and empty components
from every name; stuffr no longer does. So an archive whose entry is
`../../etc/passwd` used to extract to `<dest>/etc/passwd` at exit 0,
reported as exact fidelity, and is now refused at exit 7 with nothing
written — that is the point of the change. Two smaller differences
come with it, and both matter because entry names are matched
EXACTLY: a non-printable byte escapes as `%XX` in UPPER-case hex
where it used to be lower (`%c3` becomes `%C3`), and a literal `%`
in a stored name is now escaped as well (`%1f` becomes `%251f`, so a
stored literal is no longer indistinguishable from an escaped byte).
A `stuffr cat`/`--pattern` argument written against the old spelling
answers `EntryNotFound` at exit 2; run `stuffr list` and copy the
name from there. `\` is still split as a path separator, so a
DOS-era `dos\sub\file.txt` still reads as `dos/sub/file.txt`.
One related change is NOT specific to LHA: a FILE entry named `.`
resolves to the destination directory itself, and `unpack` used to
try to create a file over it — `i/o error: Is a directory`, exit 1,
the code that means stuffr failed. Any such entry, in any container,
is now refused at exit 7. A DIRECTORY entry named `.`, which is what
`tar cf x.tar .` writes first, is unaffected.
An entry using a compression method this build's
`delharc` feature set cannot decode (compiled with `std`, `lh1`, `lz`
— no `lhx`) is also `Error::Unsupported`, not corruption: the archive
is fine, this build just cannot decode that one method. A directory
entry (`-lhd-`) is the one case where an unsupported decoder is NOT an
error — it has no payload to decode in the first place, and reads back
as an empty directory. Every plain entry carries its own CRC-16, and a
corrupted one is reported as `Error::Corrupt` (exit 5) once its payload
has been fully read, the same as any other checksum-carrying format
here.
stuffr pack tree -o backup.arj
stuffr list backup.arj
ARJ, behind its own `--features arj` (bundled into `--features
legacy`, itself in `default`/`full`/`--all-features`, same as
`compress` and `lha` above). A default build therefore has eight
containers in total — tar, ar, cpio, zip, lha, arj, arc and zoo — not
four; only `--no-default-features --features pure` (or narrower) is
back to four.
Its magic (`0x60 0xEA`) sits at
offset 0.
Unlike `lha`, ARJ is the CONSTRAINED case: the crate behind it
(`unarj-rs`) needs `Seek` to read an archive at all, and decodes each
entry WHOLE — there is no per-entry streaming reader. `list`/`test`
still work over a pipe (the ladder spools it to a temp file first,
same as `zip` forced onto its indexed path), but the read is never
forward-only the way `lha`'s is. Because an entry is decoded whole,
this build refuses an entry whose header declares an implausible size
— past a fixed 256 MiB ceiling — BEFORE allocating for it, at exit 6
(`ResourceLimit`), never exit 5: the archive is not necessarily
damaged, this build simply will not allocate that much for one entry
it cannot stream. This is not the same knob as `--memory-limit`: that
flag binds a CODEC's own dictionary allocation (see "GUARDING AGAINST
DECOMPRESSION BOMBS" and `--memory-limit`'s own entry above); a
container opens with no such flag to read, so this ceiling is fixed
rather than configurable.
This container WRITES too, as of Phase 3c Task 7 — and with TWO
warnings, both wider than the `yes` in `stuffr formats`' WRITE
column. **First: it does not compress.** The reader accepts ARJ's
five methods; the writer emits method 0, `Stored`, and nothing else,
so `stuffr pack tree -o backup.arj` produces a real, valid ARJ
archive that is LARGER than the input, not smaller. Re-packing a
compressed `.arj` through `unpack` then `pack` loses no data and no
names — it loses the compression. Put a codec under it if you want
the size back: `-o backup.arj.gz` works, and composes exactly like
`bundle.tar.gz` above.
**Second, and it is the honest reason this entry reads differently
from `lha`'s: nothing outside stuffr has ever read an ARJ archive
stuffr wrote.** `.Z` is checked against the real `compress`(1) and
`.lzh` against `lhasa`; no `arj`/`unarj` binary is obtainable, so
here the archive is only ever judged by the same project that wrote
it. What is done about that is a byte-level test asserting the
published specification's own constraints — the ones the ARJ reader
this build uses parses and then ignores — against the bytes the
encoder emits. That is real, and it is not a second implementation.
If you have `arj` or `unarj` on a machine somewhere, a report either
way is worth more than anything in this repository.
A symlink is SKIPPED with a fidelity warning — not stored, and not
refused. ARJ's own entry-type table has no value for a link, so
writing one would put the target text in a regular file's contents;
`pack` names the entry it skipped, writes the rest, and exits 0.
--strict-fidelity turns that into exit 4, with the archive still on
disk. Exactly what `lha` says above, and what `ar` says about
directories: a kind a container cannot store is a fidelity loss, never
a refusal. (This paragraph said "refused rather than stored (exit 3)"
until Phase 3c's final review measured the binary and found no path to
exit 3 at all. It matters more than an ordinary typo because this page
ships INSIDE the binary, and `cli.rs`'s
`a_symlink_in_a_walked_tree_is_never_refused_and_the_page_may_not_say_it_is`
now runs the case on every writable container rather than trusting the
sentence.) Directories ARE stored, as themselves. Every entry carries its own CRC-32, checked once
its payload has been decoded, and a mismatch is `Error::Corrupt` (exit
5) — the same promise every other checksum-carrying format here
makes. `by_index` has exactly one answer on this container, unlike
`lha`'s two: ARJ carries no entry index of its own on any source
shape it is ever opened over, so `Error::Unsupported` routes straight
to a forward count instead.
stuffr list old.arc
ARC (and its PAK descendant — both `.arc` and `.pak` are read by this
one container), behind its own `--features arc`, bundled into
`--features legacy` exactly like the three above. The oldest format
stuffr reads, and the only one whose decoders are written from
scratch here rather than delegated to a crate: nothing publishable
wraps it that this project can depend on. Like `lha` and unlike
`arj`, it parses forward off a pipe — every entry's size is in its
own header — so `cat old.arc | stuffr list -` is a genuine forward
walk.
Four of ARC's compression methods are decoded: Stored (1 and 2),
Packed/RLE90 (3), Squeezed (4) and Crunched (8), plus PAK's Squashed
(9), which came free with Crunched. An entry using Crushed (10),
Distilled (11) or one of the pre-8 Crunched variants (5, 6, 7) is
`Error::Unsupported` (exit 3), naming the method — the archive is
fine, this build has no decoder for it. Note that a name is not a
method: `license_crushed.pak` really is Crushed (10), a different
algorithm from Crunched (8), not a spelling of it.
This container is READ-ONLY: `stuffr pack notes.txt --format arc`
fails with `Error::CapabilityUnavailable` (exit 3), the same one
sentence `lha` above used to answer with, before it and `arj` both
gained encoders. Every
entry carries its own CRC-16, checked before a single decoded byte
is handed back, so a mismatch is `Error::Corrupt` (exit 5) rather
than silently wrong data. Like `arj`, an entry decodes whole, so a
header declaring an implausible size is refused before the
allocation it would size, at exit 6 (`ResourceLimit`) against the
same fixed 256 MiB ceiling — again not `--memory-limit`, which binds
a codec rather than a container. Damage is never scanned past:
a byte that is not ARC's `0x1A` marker where a header must begin is
exit 5, not a search for the next plausible header.
**`stuffr list` CAN FAIL on this container, where on the other six it
cannot.** Everywhere else `list` reads no payload at all, so an entry
that cannot be decoded — an unsupported method, a bad CRC — is only
visible to a verb that reads one (`cat`, `test`, `unpack`). Here an
entry decodes WHOLE, and the decode is what makes the declared-size
comparison possible, so `stuffr list old.arc` on an archive using
Crushed or Distilled exits 3, and on one with a damaged entry exits 5.
A deliberate divergence, not an oversight; `zoo` below behaves the
same way. If you want a listing that cannot fail, there is none —
read the exit code.
stuffr list old.zoo
ZOO, behind its own `--features zoo`, bundled into `--features
legacy` exactly like the four above. Like `arc` its framing is
decoded from scratch here; unlike `arc` the format could be checked
against the ORIGINAL implementation — zoo 2.10's own C source — so
every structural constant this reader uses cites the macro it came
from rather than a modern reading of the layout.
Three packing methods are decoded, which is all the format ever
assigned: Stored (0), zoo's own 13-bit LZW (1), and LH5 (2), that
last one through the same `delharc` this build already carries for
`lha`. A method byte above 2 is `Error::Unsupported` (exit 3),
naming it — the archive is fine, the byte is outside the format.
ZOO's directory is a LINKED LIST of absolute file offsets, not a
sequence of headers, so this container needs `Seek`: a piped
`cat old.zoo | stuffr list -` still works, but the ladder spools it
to a temp file first (the same rung `arj` and an indexed `zip` take)
rather than parsing forward the way `lha` and `arc` do. A chain that
does not advance — a self-referential or cyclic `next` link — is
`Error::Corrupt` (exit 5), which is zoo's own verdict on the same
shape. A deleted entry stays in the chain and is skipped by `list`,
`cat`, `test` and `unpack`, neither listed nor extracted, again
matching zoo — but NOT by `salvage`, which is the one verb that
recovers it. Its payload is still whole in the file and the flag is
a single byte, so in a damaged archive a flipped bit turns a live
entry into one no other verb will ever give back. The row says so:
`[deleted: the archive marks this entry removed]`. The exit code
does not change for it — recovering a deleted record is salvage
doing its job, not a fidelity loss.
This container is READ-ONLY: `stuffr pack notes.txt --format zoo`
fails with `Error::CapabilityUnavailable` (exit 3), the same one
sentence `arc` above answers with.
Every entry carries its own CRC-16, checked before a single decoded
byte is handed back. An entry decodes whole, so a directory entry
declaring an implausible size is refused before the allocation it
would size, at exit 6 (`ResourceLimit`) against the same fixed
256 MiB ceiling `arc` and `arj` use — again not `--memory-limit`,
which binds a codec rather than a container.
**`stuffr list` CAN FAIL here too**, for the same reason and with the
same codes as `arc` above: an entry decodes whole, so an unsupported
method is exit 3 and a bad CRC-16 exit 5 from the listing verb itself,
where every container that streams an entry only ever reports those
from a verb that reads a payload.