ferrosys/lib.rs
1//! `ferrosys` — pure-Rust, userspace filesystem tooling.
2//!
3//! The crate creates and reads filesystem images entirely in userspace, over ordinary byte
4//! streams, in self-contained, safe Rust. On-disk types serialize through explicit
5//! little-endian byte accessors, so the byte layout is spelled out at every field.
6//!
7//! # Structure
8//!
9//! The crate root holds what is true of a filesystem whatever family it is, and each family
10//! lives in a module of its own behind a feature of its own.
11//!
12//! **Finding one, and opening it.**
13//!
14//! - [`detect`](fn@detect) reads an image and reports the [`Filesystem`] family it holds, and
15//! [`detect_with`] does the same at an offset within the source, for a partition or a
16//! region a carver located; [`DetectError`] tells an unreadable source from an
17//! unrecognized one.
18//! - [`OpenOptions`], [`ReadPolicy`], and [`Limits`] say where to look, how strictly, and
19//! within what bounds a read may allocate.
20// The bullet belongs to the builds that have the item: with no family compiled in there is
21// nothing an image could be opened as, so `open` and `FsReader` are not there to link.
22# detects and hands back the matching family's reader, as an \
31[`FsReader`] — an enum of concrete readers rather than a common interface, so a caller that \
32has matched its way to one has that family's whole surface."
33)]
34//!
35//! **Describing a tree, and draining one.** A directory tree is not any family's concept, so
36//! the vocabulary for one is here and every family consumes it unchanged.
37//!
38//! - [`Source`], [`SourceEntry`], [`EntryKind`], [`Metadata`], [`FileContent`], [`Xattr`],
39//! and [`Timestamp`] describe what to write; [`TreeBuilder`] and [`LayeredSource`] build
40//! one programmatically.
41//! - [`FsTree`] is the other direction, and the one behavioural trait the families share:
42//! walk the names, stat one, stream a file's bytes, resolve a link, find a path, list a
43//! directory, and report the room. It is what lets a sink drain any family without knowing
44//! which.
45//!
46//! **Saying what was lost, and what was found.**
47//!
48//! - [`FidelityReport`] names every property a format could not hold and every value a read
49//! had to invent, and [`Synthesis`] is what a read invents them from — with conservative
50//! defaults, because a tree extracted from a format with no permission bits must not land
51//! world-writable because nothing was named. A build that would lose a property fails
52//! until the caller names it in an [`AcceptedLoss`], so nothing is dropped unacknowledged.
53//! - [`Finding`] is what a scan reports, carrying a [`Severity`], a byte offset, the
54//! [`Family`] that found it, and that family's own words for the rest. Each family keeps
55//! its own typed taxonomy and projects into this, so there is one document shape and one
56//! severity scale however many families a build carries.
57//! - [`printable`](fn@printable) renders a name that came off an image as text a terminal
58//! will not act on, and [`push_json_string`] does the same into a JSON document. Every
59//! message and every finding this crate produces has already been through them; they are
60//! public for a caller whose own output names the same untrusted bytes.
61//! - [`crc32c`](fn@crc32c) is the reflected CRC-32C primitive filesystem metadata checksums
62//! are built from.
63#![cfg_attr(
64 feature = "ext",
65 doc = "\n The [`ext`] module implements the ext2/ext3/ext4 family — the \
66[`format`](ext::format) writer, the [`Reader`](ext::Reader) opened over any `Read + Seek` \
67source, and the byte-exact on-disk structures. Its images are \
68byte-reproducible: the UUID, hash seed, and timestamps are inputs, never read from the \
69clock or a random source. It re-exports the root vocabulary above, so a caller formatting \
70an ext image names one namespace rather than two."
71)]
72#![cfg_attr(
73 feature = "fat",
74 doc = "\n The [`fat`] module implements the FAT12/FAT16/FAT32 family — the \
75[`format`](fat::format) writer, the [`Reader`](fat::Reader), the \
76[`plan_layout`](fat::plan_layout) geometry planner, and the byte-exact on-disk structures \
77under [`fat::ondisk`]. Which of the three a volume is follows from its cluster count and \
78from nothing else, so the type is derived rather than chosen, and the arithmetic that \
79derives it is the format's real contract. Its images are byte-reproducible: the volume \
80serial number and the times a directory entry carries are inputs, and the date conversion \
81is UTC, so nothing about the machine that wrote an image reaches it. The reader reads any \
82conformant volume with one or two allocation tables, whatever wrote it, and takes one input \
83no other family has — \
84[`ShortNameCharset`](fat::ShortNameCharset), since nothing in a FAT volume records the code \
85page its short names are written in and this crate does not guess one."
86)]
87#![cfg_attr(
88 feature = "exfat",
89 doc = "\n The [`exfat`] module implements the exFAT family — the \
90[`format`](exfat::format) writer, the [`Reader`](exfat::Reader), the \
91[`plan_layout`](exfat::plan_layout) geometry planner, and the byte-exact on-disk structures \
92under [`exfat::ondisk`]. It shares a name with FAT and \
93no bytes: a different boot region, a different directory entry format, a different name \
94encoding, and an allocation bitmap FAT has no equivalent of. Three of its structures carry a \
95checksum and a fourth carries a hash, and every one is recomputed rather than copied. Its \
96images are byte-reproducible, and cost one input to be so — the volume serial number, since \
97an empty exFAT volume records no time anywhere."
98)]
99#![cfg_attr(
100 feature = "btrfs",
101 doc = "\n The [`btrfs`] module implements the btrfs family, over the two layers the format \
102has. A [`Volume`](btrfs::Volume) opens one over any `Read + Seek` source and gives the lower \
103one: the superblock and every mirror of it the device holds, the chunk map that turns a \
104logical address into a place on the device, and a [`Tree`](btrfs::Tree) over any of the \
105filesystem's B-trees — searched by the key tuple, iterated in key order, with every block's \
106checksum verified as it is read. A [`Reader`](btrfs::Reader) is the filesystem view built on \
107that, and [`FormatPlan`](btrfs::FormatPlan) writes one, subvolumes included. Reading a file \
108whose bytes are compressed takes the decoder for its algorithm; verifying one takes none, the \
109checksums covering the bytes on the volume. The byte-exact on-disk structures are under \
110[`btrfs::ondisk`]."
111)]
112#![cfg_attr(
113 feature = "apfs",
114 doc = "\n The [`apfs`] module reads the APFS family, over the two layers the format has. A \
115[`Container`](apfs::Container) opens one over any `Read + Seek` source at its newest whole \
116checkpoint and lists the volumes it holds, and a [`Reader`](apfs::Reader) is one of those \
117volumes read as files, or one of its snapshots. Every object's checksum is verified and every \
118object is held to the reference that reached it; a compressed file reads through its compression \
119header, its content after the header or in its resource fork. [`format`](apfs::format) writes a \
120container as `newfs_apfs` writes one into a file, its first volume populated from a source. The \
121byte-exact on-disk structures are under [`apfs::ondisk`]."
122)]
123//!
124//! # Features
125//!
126//! A build takes the filesystem families it names. The default build carries four — `ext`,
127//! `fat`, `exfat` and `btrfs` — and the three decoders they read, so it reads what a Linux
128//! kernel mounts, a compressed btrfs included. The fifth family, `apfs`, is named beside the
129//! decoders a Mac's compressed files use, as `features = ["apfs", "lzvn", "lzfse", "lzbitmap"]`.
130//! `apfs`, `lzvn` and `lzfse` depend on this project's two crates under licenses of their own,
131//! the Unicode License v3 and BSD-3-Clause, so the default build compiles this project's code
132//! under MIT OR Apache-2.0 alone. A build that turns off
133//! every family compiles the root substrate and no family code at all, and
134//! [`detect`](fn@detect) then recognizes nothing. Granularity is per *family* rather than per
135//! format — `ext` is ext2, ext3, and ext4 together, and `fat` is FAT12, FAT16, and FAT32
136//! together, since each set is one lineage sharing its on-disk structures.
137//!
138//! Cargo unifies features across a dependency graph, so selecting a subset is a property of
139//! a leaf application rather than of a library deep in someone's tree: anything else in the
140//! build that pulls this crate with a family turns that family on for everyone in it —
141//! including the answers [`detect`](fn@detect) then gives.
142//!
143//! Each feature stands alone but `lzfse`, which turns `lzvn` on, since an LZFSE stream carries
144//! LZVN blocks among its own. The two ends of a tree are the root's vocabulary, so a source
145//! feeds whichever family is being written and a sink drains whichever one was opened, and a
146//! decoder undoes an encoding whichever family stored a run of bytes in it. The seven the
147//! default build carries:
148//!
149//! - **`ext`** adds the ext2/ext3/ext4 family. It has no dependencies of its own.
150//! - **`fat`** adds the FAT12/FAT16/FAT32 family. It has no dependencies of its own.
151//! - **`exfat`** adds the exFAT family, which shares a name with the one above and none of
152//! its structures. It has no dependencies of its own.
153//! - **`btrfs`** adds the btrfs family, over a format built out of B-trees and a logical
154//! address space. It has no dependencies of its own.
155//! - **`zlib`**, **`lzo`** and **`zstd`** each add a decoder, so that a file whose extents
156//! are stored in that encoding reads as the file rather than as a refusal naming the
157//! algorithm. btrfs stores extents in all three, and APFS keeps a compressed file's content
158//! in `zlib` as well. A decoder reaches bytes only through a family that stores some, so a
159//! build naming its features names one beside such a family, as `--features btrfs,zstd`;
160//! alone it compiles its dependency and decodes nothing, since no reachable read stores runs
161//! that way. `lzo` takes no
162//! dependency, its decoder being in this crate; the other two depend on `miniz_oxide`
163//! and `ruzstd`. None of them is needed to *verify* a filesystem: the checksums it
164//! records cover the bytes it stored, so a compressed extent is checked without being
165//! expanded.
166//!
167//! The four named beside APFS, two of which depend on a crate under a license of its own:
168//!
169//! - **`apfs`** adds the APFS family: a container of volumes under a checkpoint, each volume a
170//! tree of records found through an object map, read, and written from a source tree. It
171//! depends on `ferrosys-ucd`, the Unicode Character Database tables a volume's names are
172//! compared with, which are under the Unicode License v3.
173//! - **`lzvn`**, **`lzfse`** and **`lzbitmap`** add the decoders an APFS compressed file's
174//! chunks are stored with beside `zlib`, so that such a file reads as its content rather than
175//! as a refusal naming its compression type. `lzvn` and `lzfse` depend on `ferrosys-lzfse`,
176//! whose decoders are derived from Apple's lzfse project and are under its BSD-3-Clause
177//! license, and `lzfse` turns `lzvn` on, since an LZFSE stream carries LZVN blocks among its
178//! own. `lzbitmap` takes no dependency, its decoder being in this crate.
179//!
180//! Three more name no family and are off by default, so the default build's dependencies are
181//! `thiserror` and the two decoders':
182//! - **`tar`** adds the tar/PAX archive source and sink: a filesystem built from an
183//! archive, and one written back out as one. It depends on `tar`.
184//! - **`dir`** adds the host-directory source and sink: a filesystem built by walking a
185//! tree on this machine, and one written back out as a tree, with modes, ownership,
186//! times, hard links, special files, and extended attributes. It depends on `rustix`
187//! for the directory, node, ownership and extended-attribute calls the standard library
188//! has no equivalent of, and is present on Linux.
189//! - **`serde`** adds `Serialize` to the findings taxonomy, each family's scan report and
190//! planned geometry, the fidelity report, `Usage`, and the ext feature model, for embedding
191//! them in a document of your own. It depends on `serde`.
192// The crate is safe by construction: on-disk types serialize through explicit
193// little-endian byte accessors, never transmutes or `zerocopy`.
194#![forbid(unsafe_code)]
195#![warn(missing_docs)]
196
197/// The crate's README, compiled as a doctest so the example on its front page is the
198/// example this crate's API actually accepts. The item exists only while rustdoc is
199/// collecting doctests, so it is neither part of the crate nor part of its documentation.
200#[cfg(doctest)]
201#[doc = include_str!("../README.md")]
202struct Readme;
203
204// ── Generic substrate: the crate root ──
205//
206// Present regardless of which family backends are enabled. The root exposes only what a
207// detector or a scan produces independent of the family that answers it.
208
209// The reflected CRC-32C primitive. The module is pure and has no family dependencies.
210mod crc32c;
211pub use crc32c::crc32c;
212
213// Undoing the encodings a filesystem stores a run of bytes in fewer of them. Here beside
214// `crc32c` and for the same reason: an encoding is a property of a run of bytes rather than
215// of the format around it, and the same algorithms appear in more than one filesystem. How a
216// format frames them stays with the format. Compiled where a family that stores runs this way
217// is; the decoders inside it are each a feature of their own, so a build with that family and
218// none of them still names the encoding it is declining.
219#[cfg(any(feature = "btrfs", feature = "apfs"))]
220mod compress;
221
222// The digests a filesystem may checksum with besides crc32c, here beside it and for the same
223// reason: a digest is a property of a run of bytes. Compiled where btrfs, the family that
224// checksums with them, is.
225#[cfg(feature = "btrfs")]
226mod digest;
227
228// The little-endian byte accessors every family's on-disk layer serializes through. Two
229// families compute them identically with nothing interpreted, which is what makes them a
230// shared primitive rather than a shared seam. Not compiled behind a family: the POSIX ACL
231// boundary form is a fixed little-endian record that the family-agnostic substrate parses,
232// so a build carrying no family still reads fields out of a buffer.
233mod bytes;
234
235// The blocks a reader returns to, held so the source is asked for each once. Which blocks
236// are worth holding is the family's; how many are held and which goes first is the same
237// answer for every reader that holds any.
238#[cfg(any(feature = "ext", feature = "btrfs", feature = "apfs"))]
239mod cache;
240
241// The byte boundary, in both directions. Deciding what a byte is, and finding the structure
242// that holds it, belong to the family; seeking to an offset and moving exactly that many
243// bytes is the same operation for every one of them, and so is the checked arithmetic that
244// names the offset. Both are here and the layout logic stays with the family. This also
245// holds what an i/o failure records and the conversion every error type carrying one is
246// written by, which is why the module is not compiled behind a family: `DetectError` carries
247// one in every build.
248mod io;
249
250// Rendering bytes that came off an image, an archive, or a host tree as text. Every name
251// every family reads is a byte string that may hold whatever a terminal acts on, and a
252// message or a finding that interpolates one raw hands those bytes on — so the escaping
253// happens where the name enters the text, which is the last point anything can tell it apart
254// from the words around it. Both renderings are public because a caller building its own
255// output faces the same problem with the same names, and a second implementation of these
256// rules is a second place for them to drift.
257mod escape;
258pub use escape::{hex, printable, push_json_string};
259
260// Building a JSON document, as against escaping the strings that go into one. Comma
261// placement and value formatting are the same rules whatever document is being written, so
262// they live here and every document this crate or a consumer emits is built through them —
263// the same move `escape` is, one level up from the characters.
264pub mod json;
265
266// The depth-first walk every family's reader is driven by: one frontier, one cycle check,
267// one set of bounds, with each family supplying what sits on the frontier and how a name's
268// children are read. Compiled where a family is, since with none there is no tree to walk.
269#[cfg(any(
270 feature = "ext",
271 feature = "fat",
272 feature = "exfat",
273 feature = "btrfs",
274 feature = "apfs"
275))]
276mod walk;
277
278// The path resolution every family's reader answers a lookup with: one component loop, one
279// ascent on `..`, one hop budget for symbolic links, with each family supplying only how a
280// name is found in a directory. Compiled where a family is, for the same reason the walk is.
281#[cfg(any(
282 feature = "ext",
283 feature = "fat",
284 feature = "exfat",
285 feature = "btrfs",
286 feature = "apfs"
287))]
288mod resolve;
289
290// One shape for a newtype over a field of on-disk flag bits: the set operations every one of
291// them needs, so no two of them have different ones. Compiled where a family is, since with
292// none there is no on-disk flag word.
293#[cfg(any(feature = "ext", feature = "fat", feature = "exfat", feature = "btrfs"))]
294mod flags;
295
296// One table per named choice, generating the word a variant is written as, the list of
297// them, and the variant a word means. Every set of choices a caller names goes through it,
298// so a vocabulary cannot be spelled twice and drift. `NamedChoice` is public because the
299// same three questions are what a consumer's own argument parser asks, and asking them
300// generically is what keeps its list of choices from being a second copy of this one.
301mod naming;
302pub use naming::NamedChoice;
303
304// Image detection and the family selector it returns. `Filesystem` and `DetectError` are
305// always present; `detect` dispatches across whichever families are compiled in.
306mod detect;
307pub use detect::{DetectError, DetectOptions, Filesystem, detect, detect_with};
308
309// What a path is made of, and which of its components a directory can hold. Both questions
310// are asked by every family and on both sides of each of them — a source keying entries, a
311// model placing them, a reader resolving one, a sink creating one — so both are answered
312// here. A second answer that drifted would not fail; it would key two entries apart that the
313// model considers one path, or refuse a name where it is read and accept it where it is
314// written.
315mod path;
316
317// A wall-clock instant, and one extended attribute. Both are boundary vocabulary rather
318// than storage: how an instant splits across fields and how an attribute's name compresses
319// are the family's business, and each family's on-disk layer carries its own conversion.
320//
321// The DOS date is where that rule has an exception, and it is here because FAT and exFAT
322// carry one encoding rather than two — the same packed words, epoch, and granularity,
323// inherited from the same ancestor. Where each format puts those words in an entry stays in
324// that family's own layer, so this is compiled only where a family that stores it is.
325mod time;
326#[cfg(any(feature = "fat", feature = "exfat"))]
327pub use time::DosTimestamp;
328pub use time::{Civil, Timestamp};
329mod xattr;
330pub use xattr::Xattr;
331
332// What populates a filesystem. A directory tree is not an ext concept, so the vocabulary
333// that describes one lives here and every family consumes it — what a family can
334// *represent* of an entry is that family's business, not a reason to give each family a
335// source vocabulary of its own.
336mod source;
337pub use source::{
338 CompressedFile, Darwin, EntryKind, FileContent, FileRange, LayeredSource, Metadata,
339 RemovalError, Source, SourceEntry, TreeBuilder,
340};
341
342// What a scan found, and what renders it. A family keeps its own typed taxonomy and
343// projects into `Finding`, so there is one document shape and one severity scale however
344// many families a build carries.
345mod finding;
346pub use finding::{
347 Coordinate, Deviation, FINDINGS_SCHEMA_VERSION, Family, Finding, FindingReport, ScanReport,
348 Severity,
349};
350
351// Where a filesystem begins, how strictly it is read, and what one read may allocate — the
352// same three settings whatever family answers them.
353mod policy;
354#[cfg(any(
355 feature = "ext",
356 feature = "fat",
357 feature = "exfat",
358 feature = "btrfs",
359 feature = "apfs"
360))]
361pub use policy::MAX_SYMLINK_HOPS;
362pub use policy::{Limits, OpenOptions, ReadPolicy};
363
364// Opening an image without naming its family. Compiled only where a family is: `FsReader`
365// is an enum of concrete readers, one per family, so a build carrying none has nothing for
366// it to hold.
367#[cfg(any(
368 feature = "ext",
369 feature = "fat",
370 feature = "exfat",
371 feature = "btrfs",
372 feature = "apfs"
373))]
374mod open;
375#[cfg(any(
376 feature = "ext",
377 feature = "fat",
378 feature = "exfat",
379 feature = "btrfs",
380 feature = "apfs"
381))]
382pub use open::{FsReader, OpenError, open, open_with};
383
384// What a format could not hold and what a read had to invent, both directions in one
385// report, plus the values a read fills a missing field from.
386mod fidelity;
387pub use fidelity::{AcceptedLoss, Direction, FidelityRecord, FidelityReport, Property, Synthesis};
388
389// The extraction surface: the four operations a sink needs of any family's reader, and the
390// one behavioural trait the families share.
391mod tree;
392pub use tree::{Attributes, Bound, Child, FsTree, NodeKind, TreeEntry, TreeError, Usage};
393
394// A tree this crate reads, handed back to the write side as a source.
395mod image_source;
396pub use image_source::{ImageFile, ImageSource};
397
398// Room to leave when a filesystem is sized to its contents rather than told a size. What
399// every family's search shares is the input and the arithmetic over it; the search itself
400// is each family's, because a probe is that family's writer. Compiled where a family is:
401// with none there is nothing to size.
402#[cfg(any(
403 feature = "ext",
404 feature = "fat",
405 feature = "exfat",
406 feature = "btrfs",
407 feature = "apfs"
408))]
409mod sizing;
410#[cfg(any(
411 feature = "ext",
412 feature = "fat",
413 feature = "exfat",
414 feature = "btrfs",
415 feature = "apfs"
416))]
417pub use sizing::Slack;
418
419// A POSIX access control list, which is the value one extended attribute carries. Like
420// `Xattr` itself it is boundary vocabulary: how tightly a family packs one is that family's
421// business, and the value is not.
422mod acl;
423pub use acl::{Acl, AclEntry, AclError, AclQualifier};
424
425// Where a tree comes from and where one goes: an archive at one end, a host directory at the
426// other, each in both directions. They are concrete implementations of the root's own
427// vocabulary rather than any family's — a sink drains whatever `open` returns, and a source
428// feeds whichever family is being written — so they are named here and nowhere else.
429#[cfg(feature = "tar")]
430mod archive;
431#[cfg(feature = "tar")]
432pub use archive::{ArchiveError, ArchiveSink, ArchiveSource};
433#[cfg(all(feature = "dir", any(target_os = "linux", target_os = "android")))]
434mod host;
435#[cfg(all(feature = "dir", any(target_os = "linux", target_os = "android")))]
436pub use host::{DirectorySink, DirectorySource, ExtractReport, HostError};
437
438// ── The ext family: the `ext` module, behind the `ext` feature ──
439//
440// The family's modules live at the crate root as private modules — so their cross-module
441// paths stay `crate::` — and are presented to callers only through [`ext`]. The feature
442// model, the checksum seam, and the scan taxonomy are ext concepts: they are reached
443// through `ext`, not the crate root.
444#[cfg(feature = "ext")]
445mod alloc;
446#[cfg(feature = "ext")]
447mod csum;
448#[cfg(feature = "ext")]
449mod dir;
450#[cfg(feature = "ext")]
451mod extent;
452#[cfg(feature = "ext")]
453mod feature;
454#[cfg(feature = "ext")]
455mod fit;
456#[cfg(feature = "ext")]
457mod geometry;
458#[cfg(feature = "ext")]
459mod hash;
460#[cfg(feature = "ext")]
461mod identity;
462#[cfg(feature = "ext")]
463mod journal;
464#[cfg(feature = "ext")]
465mod materialize;
466#[cfg(feature = "ext")]
467mod model;
468#[cfg(feature = "ext")]
469mod ondisk;
470#[cfg(feature = "ext")]
471mod read;
472#[cfg(feature = "ext")]
473mod replay;
474#[cfg(feature = "ext")]
475mod sealed;
476
477#[cfg(feature = "ext")]
478pub mod ext;
479
480// ── The FAT family: the `fat` module, behind the `fat` feature ──
481//
482// A family of its own, sharing only the root substrate with ext. Its layers live under the
483// module rather than at the crate root, so nothing about a FAT boot sector is reachable
484// without naming the family it belongs to.
485#[cfg(feature = "fat")]
486pub mod fat;
487
488// ── The exFAT family: the `exfat` module, behind the `exfat` feature ──
489//
490// A family of its own, and not a member of the one it shares a name with: the two have a
491// different boot region, a different directory entry format, a different name encoding, and
492// an allocation model FAT has no equivalent of. Same arrangement as `fat` — the layers live
493// under the module, so nothing about an exFAT boot sector is reachable without naming the
494// family it belongs to.
495#[cfg(feature = "exfat")]
496pub mod exfat;
497
498// ── The btrfs family: the `btrfs` module, behind the `btrfs` feature ──
499//
500// Same arrangement again, over a format built differently from all three above: B-trees over
501// a logical address space that a chunk tree maps onto the device, with a checksum on every
502// metadata block. The layers live under the module, so nothing about a chunk map is reachable
503// without naming the family it belongs to.
504#[cfg(feature = "btrfs")]
505pub mod btrfs;
506
507// ── The APFS family: the `apfs` module, behind the `apfs` feature ──
508//
509// A container of volumes rather than one filesystem: a pool of blocks under a checkpoint and
510// an object map, listing up to a hundred volumes that each have a tree of their own. The
511// layers live under the module, so nothing about a checkpoint is reachable without naming
512// the family it belongs to.
513#[cfg(feature = "apfs")]
514pub mod apfs;