Expand description
§Hermit
Hermit runs Linux programs with controlled thread
schedules, time and randomness for reproducible testing and record/replay.
The Cargo package is hermit-run; its main executable is hermit.
Read the 2022 introduction, Hermit: Deterministic Linux for Controlled Testing and Software Bug-finding for the ideas behind the project, and the current user guide for supported commands and execution guarantees.
§Install on Ubuntu 24.04
Use x86_64 Linux and a current stable Rust toolchain from rustup. The stock Ubuntu 24.04 Linux 6.8 kernel is the oldest kernel qualified for the 0.4.1 ptrace release. Linux user, PID and mount namespaces, parent-child ptrace and seccomp filters must be available.
Hermit uses Reverie’s explicit ptracer-thread mode on older kernels. The generic
Safeptrace interface for sibling-thread waits requires PIDFD_THREAD, introduced
in Linux 6.9, and reports an error when it is unavailable. Linux 5.15 is a design
target, with no qualification claim yet; see the
compatibility follow-up.
Install the C compiler/linker and native development libraries before Cargo:
sudo apt-get update
sudo apt-get install -y build-essential pkg-config libunwind-dev liblzma-dev
cargo +stable install hermit-run --version 0.5.0 --locked
hermit --version
hermit run -- /bin/echo helloNo extra LZMA linker flags or nightly Rust are required. On other distributions,
install the equivalent C toolchain, pkg-config, libunwind and LZMA development
packages. The installer also provides hermit-dap and verification-report.
The registry package builds the Rust library and executables; libhermit.so
is a separate hermit-cdylib source build.
The default installation includes ptrace and KVM. Optional SaBRe support
and e9patch preprocessing require explicit Cargo features and separately
supplied runtime artifacts. LiteInst, in-guest-trap, native-exec and DBT are
not available in the 0.5.0 registry release, including with
--all-features; their source-development implementations remain on
main. KVM requires read/write access to /dev/kvm, compatible CPU features
and performance counters; ptrace is the default execution path.
§Ubuntu AppArmor and user namespaces
Ubuntu 24.04 restricts unprivileged user namespaces through AppArmor. A first
run can fail with MapUid: EPERM or Setting UID map failed even when
kernel.unprivileged_userns_clone=1. Ask the system administrator to allow user
namespaces for the installed Hermit executable with a scoped AppArmor profile.
Keep the system-wide user-namespace restriction enabled.
Find the executable’s absolute path with readlink -f "$(command -v hermit)".
Have the administrator create /etc/apparmor.d/hermit-local with this content,
replacing /home/YOUR_USER/.cargo/bin/hermit with that exact path:
abi <abi/4.0>,
include <tunables/global>
profile hermit /home/YOUR_USER/.cargo/bin/hermit flags=(unconfined) {
userns,
}Load it and retry:
sudo apparmor_parser -r /etc/apparmor.d/hermit-local
hermit run -- /bin/echo helloThis profile grants the installed Hermit executable user-namespace access while
keeping kernel.apparmor_restrict_unprivileged_userns=1. Review the executable
path and local security policy before applying it. Reinstalling to a different
path requires updating the profile.
§Performance counters
Precise preemption uses CPU performance counters. Host security policy and virtual machines can restrict them. Hermit reports when it disables its PMU timeslice; a successful small example in that mode does not qualify CPU-bound or threaded workloads for precise preemption. Check the user guide before changing the host’s perf policy.
§Run and verify
hermit run -- /bin/echo hello
hermit run --strict -- /bin/echo hello
hermit run --strict --verify --verify-strict -- /bin/echo reproducible < /dev/null--strict selects strict launch behavior. --verify-strict selects the full
canonical observation comparison; ordinary --verify uses a weaker diagnostic
comparison. Hermit does not make a changing filesystem or external network
reproducible; provide stable inputs for repeatability checks.
Virtual time starts at the current host time captured once for the invocation.
Hermit reports the exact epoch and reproduce with --epoch=... on stderr, or in
the selected --log-file. Reuse that epoch and the same seed for time-dependent
reproduction. Explicit --epoch and HERMIT_EPOCH values are preserved;
--seed defaults to 0. See the
virtual-time behavior notes.
§Record and replay
hermit record start -- /bin/echo recorded
hermit record list
hermit replay --autopilotAutopilot replays the latest recording without GDB. For an interactive replay,
install GDB and use hermit replay <record-id>. Run hermit --help and the
subcommand’s --help for options, output locations and verification reports.
§License
BSD-3-Clause. See LICENSE.
Re-exports§
Modules§
- build_
info - Producer values for the shared machine-readable build record.
- canonical_
verdict - Machine-readable verification reports shared by the Hermit producer and its consumers.
- cpu_
profile_ admission - Whether a run may use
--cpu-profile x86-64-v3, and the host facts it binds when it may (https://github.com/rrnewton/hermit/issues/4243). - e9patch
- Content-addressed offline rewriting for the experimental e9patch backend.
- happens_
before - Debug-info resolution for happens-before anchors.
- host_
input_ change - Name a host file that changed during one verification run.
- instruction_
map - Offline discovery of nondeterministic x86 instructions in ELF binaries.
- liteinst_
bootstrap - The shared CLI logging filter policy and the tool identity, configuration schema and effective logging directives carried by the LiteInst bootstrap. The payload is schema-validated and its config fingerprint is compared; it is not cryptographically authenticated. The host resolves the environment once; the guest consumes the same accepted directives without reading its own environment or reformatting field matchers.
- liteinst_
record - Bounded complete records using stock tracing field formatting.
- logdiff_
report - Typed machine-readable reports produced by
hermit log-diff --json. - nonwaiting_
write - Diagnostic writes that never wait for a reader; they live in detcore, whose own stderr writer reads the same flag. Diagnostic writes that never wait for the reader of a descriptor.
- physical_
exit_ watch - Observes guest processes’ physical exits through pidfds, for backends whose processes leave the host some time after their scheduler-granted exit.
- preemption_
admission - Refuse timer requests that the selected runtime cannot implement.
- proc_
mount - Report the proc mount’s capabilities after container setup.
- xstate_
probe - The startup qualification of
--cpu-profile x86-64-v3(https://github.com/rrnewton/hermit/issues/4243).
Structs§
- Backend
RunFailure - The backend failed while running the guest, after Hermit launched it: the
DBT evidence check, a KVM execution failure, a SaBRe guest that never
reached Detcore.
run --verifyrecords it asrun_failedwithbackend_failureinstead of leaving thenot_runstamp. - Backend
Unavailable - A requested backend could not start because its required runtime support is absent.
- Command
- A builder for spawning a process.
- DetConfig
- Configuration options for detcore.
- Detcore
- The detcore tool and its per-process state.
- Guest
Timed Out - The guest outlived the wall-clock bound the caller asked hermit to enforce.
- Hermit
Cleanup Unconfirmed - Sendable diagnostic identifying an operation retained on its original thread.
- Hermit
Data - Holds the context necessary to run high-level hermit functions.
- Hermit
Ptrace Failure - A completed failed ptrace run, with its original nonclone primary cause. Secondary backend/cleanup errors never determine its failure classification.
- Id
- A recording ID.
- Liteinst
InGuest Refusal - In-guest LiteInst refused a run before starting its guest, because the runtime cannot honour the request or could not monitor the program.
- Mount
- A mount.
- Mount
Provenance - What a recording takes from the completed recording container’s mount namespace, which a caller without a Hermit container leaves empty.
- Namespace
- A namespace that may be unshared with
Command::unshare. - Output
- The output of a finished process.
- Recording
- The result of recording a command.
- Serializable
Error - ⚠️ THE
kindFIELD IS THE THIRD AND LAST OF ONE CHAIN OF FLATTENINGS. The same property – the failure’s CLASS – was destroyed at three different boundaries, in series on one path: - Shebang
- Unix shebang, see Wikipedia.
- Skid
Overshoot Error - A ptrace-backed run observed late precise-timer delivery, so Hermit refuses to treat the completed execution as deterministic evidence.
- Stdio
- Describes what to do with a standard I/O stream for a child process when
passed to the
stdin,stdout, andstderrmethods ofCommand.
Enums§
- Backend
- Process instrumentation backend used to run a Hermit guest.
- Exit
Status - Describes the result of a process after it has exited.
- Failure
Kind - A serializable error. This is useful for sending an error to the parent
process. This works by converting an error into a string via its
Displayimplementation. Although we lose type information in the process of converting to a string, this preserves the error message and its error chain. - Hermit
Cleanup Stage - The phase whose original owner remains retained.
- Namespace
Use - Whether a run creates Hermit’s user and PID namespaces. A
--no-namespacerun starts the guest as a plain child and never callsunshare, so the namespace probe says nothing about whether it can run (https://github.com/rrnewton/hermit/issues/4515). - Recovery
Refusal - A refusal that leaves any existing operation owner untouched.
Constants§
- GUEST_
PROGRAM_ NOT_ EXECUTABLE_ EXIT - The guest program exists but cannot be executed as given: 126.
- GUEST_
PROGRAM_ NOT_ FOUND_ EXIT - The guest program could not be found at all: 127.
- HERMIT_
DEADLINE_ EXIT - A deadline hermit itself was asked to enforce expired: 124.
- HERMIT_
INTERNAL_ FAILURE_ EXIT - Exit status for a failure of HERMIT ITSELF, as distinct from the guest’s.
- HERMIT_
LOG_ CAP_ EXIT - Hermit’s own log output exceeded
--max-log-bytes: 123. - HERMIT_
VERIFICATION_ DIVERGENCE_ EXIT - Exit status for a completed comparison that found a divergence.
- LAST
- The file in the data directory naming the newest recording a user asked
for (
hermit record,hermit run --record-networking), which a barehermit replayreplays. - LAST_
RUN - The file in the data directory naming the newest recording of any kind,
the manifest of a plain
hermit runincluded. - LITEINST_
TOOL_ RUNTIME_ ENV - Environment variable naming the in-guest Detcore preload
(
libdetcore_liteinst.so) explicitly. - MANIFEST_
NAME - The name of the run manifest a recording directory holds (https://github.com/rrnewton/hermit/issues/4271).
- RECORDING_
METADATA_ NAME - The name of the JSON metadata file that is saved for each recording.
- RECORD_
REPLAY_ HASHES_ IO_ BUFFERS - Whether record/replay verification hashes syscall output buffers, and therefore whether its log comparison can see buffer CONTENT at all.
- RECORD_
REPLAY_ VIRTUALIZES_ TIME - Whether record/replay virtualizes time. FALSE, DELIBERATELY, AND THAT IS WHY A GREEN RECORD/REPLAY VERDICT IS NOT A DETERMINISM RESULT.
Traits§
- Context
- Provides the
contextmethod forResult. - Record
OrReplay - Helper trait.
Functions§
- backend_
delivers_ preemption_ timer - Whether
backend’s runtime arms Reverie ptrace’s PMU timer and delivers its events, so--max-timeslicereally preempts a thread that makes no syscall: the backend’s stateddelivers_preemption_timer_events. Only ptrace does (e9patch preprocessing runs on it). SaBRe and KVM accept a timer request and deliver nothing; DBT and in-guest LiteInst refuse it. - base_
mountpoint_ mount_ ids - Each of
BASE_MOUNTPOINTSthat the calling thread can open, as the kernel escapes it inmountinfo, with the mount ID of a descriptor held on it: rule 2’s proof ofContainerMountTable::select. The base mountpoints hold no byte the kernel escapes (space, tab, newline or backslash), so each is its own escaped form. - capture_
mountinfo_ identity_ order - Capture the calling thread’s mount namespace’s raw IDs in the exact row/parent order used by Detcore’s canonical mountinfo mapping.
- capture_
mountinfo_ identity_ order_ keeping capture_mountinfo_identity_order, keeping every row whose mount ID is inkeep(the rows a container mount table shows) whatever its name: a mount Hermit installed is never dropped by the seed filter, so a table showing it validates against the captured order.- current_
mount_ namespace - The calling thread’s mount namespace (
DetConfig::mountinfo_mount_namespace): the kernel’s namespace ID fromioctl(NS_GET_MNTNS_ID)on/proc/thread-self/ns/mnt, and the descriptor’s device and inode numbers. The calling thread’s, not the thread group leader’s: a library caller’s thread can have a mount namespace of its own, and every capture here (/proc/thread-self/mountinfoandfdinfo) reads that same thread’s. A kernel without the ioctl (ENOTTY or EINVAL) yields no ID; any other failure is an error. - forced_
site_ patching_ value - The value of
REVERIE_LITEINST_SITE_PATCHING(reverie_liteinst::SITE_PATCHING_ENV) that the guest sees underbackend, when Hermit decides it:0forin-guest-trap, which exists to run with no syscall site patched, andNoneforliteinst, which leaves the variable to the guest environment (unset means patching on). - launch_
on_ calling_ thread - Makes every later launch in this process run its ptrace session on the
calling thread, as launches did before the tracer thread, and leaves the
caller’s stack to the caller. For a process that must not gain threads: the
hermitCLI runs its launch inside a container child made with a rawclone, whose entry point ends with a thread exit, so a tracer thread still waiting for work would keep that child alive. The CLI launches from its main thread or from the child’s own stack, which have room. - mountinfo_
identity_ order_ keeping_ from_ raw - The identity order of the mountinfo contents
raw(seed rows dropped, asmountinfo_identity_order_from_rawdrops them), keeping a seed row whose mount ID is inkeep. - output_
backend_ stdin - Returns the stdin to hand a guest run in an output-capturing backend.
- record_
to - Records to an existing directory. The caller must keep its backing resources
alive through any
HermitCleanupUnconfirmedrecovery.HermitData::recordowns its temporary recording directory inside the retained operation. - record_
to_ with_ mountinfo - Records with producer-owned mountinfo provenance. The directory-lifetime
obligation is the same as
record_to.local_networkingandproc_levelare stored in the recording’smetadata.json, and a replay uses that network and that/proclevel. - record_
with_ output - Records and captures output. Keep the directory’s backing resources alive
through any cleanup checkpoint, as for
record_to. - record_
with_ output_ with_ mountinfo - Records with captured output and exact mountinfo provenance.
local_networkingandproc_levelare as forrecord_to_with_mountinfo. - replay_
from - Replays an existing recording. Materialization and chroot guards remain with the original runtime through any cleanup checkpoint.
- replay_
with_ gdbserver - Replays with a gdb server.
- replay_
with_ gdbserver_ and_ mounts - Replays with a gdb server and mounts inside the replay chroot.
- replay_
with_ output - Replays with captured output.
- replay_
with_ output_ and_ mounts - Replays with captured output and mounts inside the replay chroot.
- reserve_
kvm_ stdin - Saves stdin captured before Rust’s process startup can reuse a closed fd 0.
- reserve_
output_ stdin_ snapshot - Records a rewindable snapshot of the process stdin so the output-capturing
backends can replay identical input to each run of
hermit run --verify. - run
- Run the given command as deterministically as possible.
- run_
with_ backend - Run the given command using the selected instrumentation backend.
- run_
with_ backend_ timeout run_with_backendwith a hermit-enforced wall-clock bound on the guest.- run_
with_ output - Variant of
runthat also captures stdout/stderr. - run_
with_ output_ backend - Variant of
run_with_backendthat also captures stdout/stderr. - run_
with_ output_ backend_ timeout run_with_output_backendwith a hermit-enforced wall-clock bound.- thread_
mountinfo - The calling thread’s mount table (
/proc/thread-self/mountinfo): the rows of the namespacecurrent_mount_namespacenames. - unplanned_
container_ mount_ table - The container mount table of a run whose guest runs in the calling
thread’s own mount namespace, in which Hermit installed no mount
(
--no-namespace, or a library caller that built no container, on KVM), and the raw identity order Detcore validates it against, both from one read of the namespace’s rows. Shown are:
Type Aliases§
- Error
- Unplanned
Mount Table - A container mount table and the raw identity order Detcore validates it
against (
unplanned_container_mount_table).