Skip to main content

Crate hermit

Crate hermit 

Source
Expand description

§Hermit

Hermit runs Linux programs with controlled thread schedules, time and randomness for reproducible testing and record/replay. The Cargo package is hermit-run; its main executable is hermit.

Read the 2022 introduction, Hermit: Deterministic Linux for Controlled Testing and Software Bug-finding for the ideas behind the project, and the current user guide for supported commands and execution guarantees.

§Install on Ubuntu 24.04

Use x86_64 Linux and a current stable Rust toolchain from rustup. The stock Ubuntu 24.04 Linux 6.8 kernel is the oldest kernel qualified for the 0.4.1 ptrace release. Linux user, PID and mount namespaces, parent-child ptrace and seccomp filters must be available.

Hermit uses Reverie’s explicit ptracer-thread mode on older kernels. The generic Safeptrace interface for sibling-thread waits requires PIDFD_THREAD, introduced in Linux 6.9, and reports an error when it is unavailable. Linux 5.15 is a design target, with no qualification claim yet; see the compatibility follow-up.

Install the C compiler/linker and native development libraries before Cargo:

sudo apt-get update
sudo apt-get install -y build-essential pkg-config libunwind-dev liblzma-dev
cargo +stable install hermit-run --version 0.5.0 --locked
hermit --version
hermit run -- /bin/echo hello

No extra LZMA linker flags or nightly Rust are required. On other distributions, install the equivalent C toolchain, pkg-config, libunwind and LZMA development packages. The installer also provides hermit-dap and verification-report. The registry package builds the Rust library and executables; libhermit.so is a separate hermit-cdylib source build.

The default installation includes ptrace and KVM. Optional SaBRe support and e9patch preprocessing require explicit Cargo features and separately supplied runtime artifacts. LiteInst, in-guest-trap, native-exec and DBT are not available in the 0.5.0 registry release, including with --all-features; their source-development implementations remain on main. KVM requires read/write access to /dev/kvm, compatible CPU features and performance counters; ptrace is the default execution path.

§Ubuntu AppArmor and user namespaces

Ubuntu 24.04 restricts unprivileged user namespaces through AppArmor. A first run can fail with MapUid: EPERM or Setting UID map failed even when kernel.unprivileged_userns_clone=1. Ask the system administrator to allow user namespaces for the installed Hermit executable with a scoped AppArmor profile. Keep the system-wide user-namespace restriction enabled.

Find the executable’s absolute path with readlink -f "$(command -v hermit)". Have the administrator create /etc/apparmor.d/hermit-local with this content, replacing /home/YOUR_USER/.cargo/bin/hermit with that exact path:

abi <abi/4.0>,
include <tunables/global>

profile hermit /home/YOUR_USER/.cargo/bin/hermit flags=(unconfined) {
  userns,
}

Load it and retry:

sudo apparmor_parser -r /etc/apparmor.d/hermit-local
hermit run -- /bin/echo hello

This profile grants the installed Hermit executable user-namespace access while keeping kernel.apparmor_restrict_unprivileged_userns=1. Review the executable path and local security policy before applying it. Reinstalling to a different path requires updating the profile.

§Performance counters

Precise preemption uses CPU performance counters. Host security policy and virtual machines can restrict them. Hermit reports when it disables its PMU timeslice; a successful small example in that mode does not qualify CPU-bound or threaded workloads for precise preemption. Check the user guide before changing the host’s perf policy.

§Run and verify

hermit run -- /bin/echo hello
hermit run --strict -- /bin/echo hello
hermit run --strict --verify --verify-strict -- /bin/echo reproducible < /dev/null

--strict selects strict launch behavior. --verify-strict selects the full canonical observation comparison; ordinary --verify uses a weaker diagnostic comparison. Hermit does not make a changing filesystem or external network reproducible; provide stable inputs for repeatability checks.

Virtual time starts at the current host time captured once for the invocation. Hermit reports the exact epoch and reproduce with --epoch=... on stderr, or in the selected --log-file. Reuse that epoch and the same seed for time-dependent reproduction. Explicit --epoch and HERMIT_EPOCH values are preserved; --seed defaults to 0. See the virtual-time behavior notes.

§Record and replay

hermit record start -- /bin/echo recorded
hermit record list
hermit replay --autopilot

Autopilot replays the latest recording without GDB. For an interactive replay, install GDB and use hermit replay <record-id>. Run hermit --help and the subcommand’s --help for options, output locations and verification reports.

§License

BSD-3-Clause. See LICENSE.

Re-exports§

pub use canonical_verdict::Verdict;
pub use reverie::process;

Modules§

build_info
Producer values for the shared machine-readable build record.
canonical_verdict
Machine-readable verification reports shared by the Hermit producer and its consumers.
cpu_profile_admission
Whether a run may use --cpu-profile x86-64-v3, and the host facts it binds when it may (https://github.com/rrnewton/hermit/issues/4243).
e9patch
Content-addressed offline rewriting for the experimental e9patch backend.
happens_before
Debug-info resolution for happens-before anchors.
host_input_change
Name a host file that changed during one verification run.
instruction_map
Offline discovery of nondeterministic x86 instructions in ELF binaries.
liteinst_bootstrap
The shared CLI logging filter policy and the tool identity, configuration schema and effective logging directives carried by the LiteInst bootstrap. The payload is schema-validated and its config fingerprint is compared; it is not cryptographically authenticated. The host resolves the environment once; the guest consumes the same accepted directives without reading its own environment or reformatting field matchers.
liteinst_record
Bounded complete records using stock tracing field formatting.
logdiff_report
Typed machine-readable reports produced by hermit log-diff --json.
nonwaiting_write
Diagnostic writes that never wait for a reader; they live in detcore, whose own stderr writer reads the same flag. Diagnostic writes that never wait for the reader of a descriptor.
physical_exit_watch
Observes guest processes’ physical exits through pidfds, for backends whose processes leave the host some time after their scheduler-granted exit.
preemption_admission
Refuse timer requests that the selected runtime cannot implement.
proc_mount
Report the proc mount’s capabilities after container setup.
xstate_probe
The startup qualification of --cpu-profile x86-64-v3 (https://github.com/rrnewton/hermit/issues/4243).

Structs§

BackendRunFailure
The backend failed while running the guest, after Hermit launched it: the DBT evidence check, a KVM execution failure, a SaBRe guest that never reached Detcore. run --verify records it as run_failed with backend_failure instead of leaving the not_run stamp.
BackendUnavailable
A requested backend could not start because its required runtime support is absent.
Command
A builder for spawning a process.
DetConfig
Configuration options for detcore.
Detcore
The detcore tool and its per-process state.
GuestTimedOut
The guest outlived the wall-clock bound the caller asked hermit to enforce.
HermitCleanupUnconfirmed
Sendable diagnostic identifying an operation retained on its original thread.
HermitData
Holds the context necessary to run high-level hermit functions.
HermitPtraceFailure
A completed failed ptrace run, with its original nonclone primary cause. Secondary backend/cleanup errors never determine its failure classification.
Id
A recording ID.
LiteinstInGuestRefusal
In-guest LiteInst refused a run before starting its guest, because the runtime cannot honour the request or could not monitor the program.
Mount
A mount.
MountProvenance
What a recording takes from the completed recording container’s mount namespace, which a caller without a Hermit container leaves empty.
Namespace
A namespace that may be unshared with Command::unshare.
Output
The output of a finished process.
Recording
The result of recording a command.
SerializableError
⚠️ THE kind FIELD IS THE THIRD AND LAST OF ONE CHAIN OF FLATTENINGS. The same property – the failure’s CLASS – was destroyed at three different boundaries, in series on one path:
Shebang
Unix shebang, see Wikipedia.
SkidOvershootError
A ptrace-backed run observed late precise-timer delivery, so Hermit refuses to treat the completed execution as deterministic evidence.
Stdio
Describes what to do with a standard I/O stream for a child process when passed to the stdin, stdout, and stderr methods of Command.

Enums§

Backend
Process instrumentation backend used to run a Hermit guest.
ExitStatus
Describes the result of a process after it has exited.
FailureKind
A serializable error. This is useful for sending an error to the parent process. This works by converting an error into a string via its Display implementation. Although we lose type information in the process of converting to a string, this preserves the error message and its error chain.
HermitCleanupStage
The phase whose original owner remains retained.
NamespaceUse
Whether a run creates Hermit’s user and PID namespaces. A --no-namespace run starts the guest as a plain child and never calls unshare, so the namespace probe says nothing about whether it can run (https://github.com/rrnewton/hermit/issues/4515).
RecoveryRefusal
A refusal that leaves any existing operation owner untouched.

Constants§

GUEST_PROGRAM_NOT_EXECUTABLE_EXIT
The guest program exists but cannot be executed as given: 126.
GUEST_PROGRAM_NOT_FOUND_EXIT
The guest program could not be found at all: 127.
HERMIT_DEADLINE_EXIT
A deadline hermit itself was asked to enforce expired: 124.
HERMIT_INTERNAL_FAILURE_EXIT
Exit status for a failure of HERMIT ITSELF, as distinct from the guest’s.
HERMIT_LOG_CAP_EXIT
Hermit’s own log output exceeded --max-log-bytes: 123.
HERMIT_VERIFICATION_DIVERGENCE_EXIT
Exit status for a completed comparison that found a divergence.
LAST
The file in the data directory naming the newest recording a user asked for (hermit record, hermit run --record-networking), which a bare hermit replay replays.
LAST_RUN
The file in the data directory naming the newest recording of any kind, the manifest of a plain hermit run included.
LITEINST_TOOL_RUNTIME_ENV
Environment variable naming the in-guest Detcore preload (libdetcore_liteinst.so) explicitly.
MANIFEST_NAME
The name of the run manifest a recording directory holds (https://github.com/rrnewton/hermit/issues/4271).
RECORDING_METADATA_NAME
The name of the JSON metadata file that is saved for each recording.
RECORD_REPLAY_HASHES_IO_BUFFERS
Whether record/replay verification hashes syscall output buffers, and therefore whether its log comparison can see buffer CONTENT at all.
RECORD_REPLAY_VIRTUALIZES_TIME
Whether record/replay virtualizes time. FALSE, DELIBERATELY, AND THAT IS WHY A GREEN RECORD/REPLAY VERDICT IS NOT A DETERMINISM RESULT.

Traits§

Context
Provides the context method for Result.
RecordOrReplay
Helper trait.

Functions§

backend_delivers_preemption_timer
Whether backend’s runtime arms Reverie ptrace’s PMU timer and delivers its events, so --max-timeslice really preempts a thread that makes no syscall: the backend’s stated delivers_preemption_timer_events. Only ptrace does (e9patch preprocessing runs on it). SaBRe and KVM accept a timer request and deliver nothing; DBT and in-guest LiteInst refuse it.
base_mountpoint_mount_ids
Each of BASE_MOUNTPOINTS that the calling thread can open, as the kernel escapes it in mountinfo, with the mount ID of a descriptor held on it: rule 2’s proof of ContainerMountTable::select. The base mountpoints hold no byte the kernel escapes (space, tab, newline or backslash), so each is its own escaped form.
capture_mountinfo_identity_order
Capture the calling thread’s mount namespace’s raw IDs in the exact row/parent order used by Detcore’s canonical mountinfo mapping.
capture_mountinfo_identity_order_keeping
capture_mountinfo_identity_order, keeping every row whose mount ID is in keep (the rows a container mount table shows) whatever its name: a mount Hermit installed is never dropped by the seed filter, so a table showing it validates against the captured order.
current_mount_namespace
The calling thread’s mount namespace (DetConfig::mountinfo_mount_namespace): the kernel’s namespace ID from ioctl(NS_GET_MNTNS_ID) on /proc/thread-self/ns/mnt, and the descriptor’s device and inode numbers. The calling thread’s, not the thread group leader’s: a library caller’s thread can have a mount namespace of its own, and every capture here (/proc/thread-self/mountinfo and fdinfo) reads that same thread’s. A kernel without the ioctl (ENOTTY or EINVAL) yields no ID; any other failure is an error.
forced_site_patching_value
The value of REVERIE_LITEINST_SITE_PATCHING (reverie_liteinst::SITE_PATCHING_ENV) that the guest sees under backend, when Hermit decides it: 0 for in-guest-trap, which exists to run with no syscall site patched, and None for liteinst, which leaves the variable to the guest environment (unset means patching on).
launch_on_calling_thread
Makes every later launch in this process run its ptrace session on the calling thread, as launches did before the tracer thread, and leaves the caller’s stack to the caller. For a process that must not gain threads: the hermit CLI runs its launch inside a container child made with a raw clone, whose entry point ends with a thread exit, so a tracer thread still waiting for work would keep that child alive. The CLI launches from its main thread or from the child’s own stack, which have room.
mountinfo_identity_order_keeping_from_raw
The identity order of the mountinfo contents raw (seed rows dropped, as mountinfo_identity_order_from_raw drops them), keeping a seed row whose mount ID is in keep.
output_backend_stdin
Returns the stdin to hand a guest run in an output-capturing backend.
record_to
Records to an existing directory. The caller must keep its backing resources alive through any HermitCleanupUnconfirmed recovery. HermitData::record owns its temporary recording directory inside the retained operation.
record_to_with_mountinfo
Records with producer-owned mountinfo provenance. The directory-lifetime obligation is the same as record_to. local_networking and proc_level are stored in the recording’s metadata.json, and a replay uses that network and that /proc level.
record_with_output
Records and captures output. Keep the directory’s backing resources alive through any cleanup checkpoint, as for record_to.
record_with_output_with_mountinfo
Records with captured output and exact mountinfo provenance. local_networking and proc_level are as for record_to_with_mountinfo.
replay_from
Replays an existing recording. Materialization and chroot guards remain with the original runtime through any cleanup checkpoint.
replay_with_gdbserver
Replays with a gdb server.
replay_with_gdbserver_and_mounts
Replays with a gdb server and mounts inside the replay chroot.
replay_with_output
Replays with captured output.
replay_with_output_and_mounts
Replays with captured output and mounts inside the replay chroot.
reserve_kvm_stdin
Saves stdin captured before Rust’s process startup can reuse a closed fd 0.
reserve_output_stdin_snapshot
Records a rewindable snapshot of the process stdin so the output-capturing backends can replay identical input to each run of hermit run --verify.
run
Run the given command as deterministically as possible.
run_with_backend
Run the given command using the selected instrumentation backend.
run_with_backend_timeout
run_with_backend with a hermit-enforced wall-clock bound on the guest.
run_with_output
Variant of run that also captures stdout/stderr.
run_with_output_backend
Variant of run_with_backend that also captures stdout/stderr.
run_with_output_backend_timeout
run_with_output_backend with a hermit-enforced wall-clock bound.
thread_mountinfo
The calling thread’s mount table (/proc/thread-self/mountinfo): the rows of the namespace current_mount_namespace names.
unplanned_container_mount_table
The container mount table of a run whose guest runs in the calling thread’s own mount namespace, in which Hermit installed no mount (--no-namespace, or a library caller that built no container, on KVM), and the raw identity order Detcore validates it against, both from one read of the namespace’s rows. Shown are:

Type Aliases§

Error
UnplannedMountTable
A container mount table and the raw identity order Detcore validates it against (unplanned_container_mount_table).