snare
Deterministic, in-process simulation of the operating system for Rust tests — without changing the code under test.
Your code keeps calling std::net::TcpStream, std::fs::File, std::env::var, libc::sched_setscheduler,
clock_gettime, raw AF_PACKET/io_uring — exactly as it does in production. snare interposes at the
OS-call boundary underneath it and serves those calls from an in-memory model: a virtual network, a
virtual filesystem, a simulated host, a virtual clock. The test drives the other side of the wire.
use ;
use ;
use TcpStream;
let sim = new;
sim.run;
No shim types, no dependency injection, no trait objects threaded through the code under test. It is a
plain TcpStream.
The paradigm
snare 1.x required the code under test to import snare's shim types (snare::net::TcpStream
instead of std::net::TcpStream, and so on). That only tests code you could edit to use the shims, and
it never covers what a dependency does three crates down.
snare 2 removes that constraint. It patches the import tables of the loaded program (ELF GOT / Mach-O
__got / PE IAT) so that the libc and Win32 symbols a managed thread calls — socket, open, getenv,
clock_gettime, ioctl, sched_setscheduler, … — land in snare's hooks first. A hook offers the call
to the test's in-memory model; if the model declines, the call falls through to the real OS. Nothing the
code under test imports changes. This reaches straight through std and every dependency.
Every resource or object used by the code under test must be created under its active simulation.
Run initialization inside Sim::run, including constructors, dependency initialization, worker
pools and lazy initialization. Objects created before entering the simulation, or created with
interposition disabled and then handed to the code under test, violate this usage invariant.
Passthrough (snare::real) is reserved for other-side emulation and the test harness. The code under
test, including its dependencies and initialization, must remain under interposition. The harness
can configure the simulated world before the run and use independent real resources for its own
bookkeeping, fixtures and native comparisons.
Two needs, one library:
- Deterministic unit tests — write them in-crate guarded by
#[cfg(snare)]and run them withcargo snare(below), or as ordinary integration tests that build aSim. - One process, one API — the test body, its scripted peers (testers) and any higher-level harness
run in the same process as the code under test and drive the simulation through the same in-process
API (
Sim, testers,set_udp_policy, the time controls). There is no controller process and no cross-process protocol; when the harness or other-side emulator needs the real machine it says so withsnare::real.
How it works
code under test (plain std / libc / raw syscalls)
│ its OS symbols are interposed
▼
snare-interpose: import-table patch ──► per-thread hooks ──► Domain
├─ Layer stack (clock, entropy, sleep)
├─ Net backend (sockets, netlink, raw L2)
├─ Fs backend (files, /sys, /proc, /dev)
├─ Host backend (scheduling, NIC tuning)
└─ Env backend (getenv/setenv)
│ declined?
▼
the real OS
snare-interposeis the engine: the import-table patchers, the per-thread hook functions, theDomain(a thread's simulated world), theLayer/Net/Fs/Host/Envbackend traits, and thereal()escape hatch. It has no opinion about what is simulated — that is a backend.snareis the ergonomic layer built on it:Sim/SimBuilder,SimHost(one backend that models a whole host),HostProfile/EasyBuilderto configure it, theFabric(TCP/UDP/raw-L2 network),VirtualFs, and theconnect_tester/run_testers!scripted-peer API.
A thread becomes managed while it is inside Sim::run, and every thread it spawns inherits the
simulation automatically. Only managed threads are interposed; the test's own bookkeeping is not.
What is simulated
| Surface | Covered |
|---|---|
| Time | clock_gettime (CLOCK_MONOTONIC/REALTIME/TAI), gettimeofday, QueryPerformanceCounter, GetTickCount64, mach_absolute_time — a discrete-event virtual clock at a fixed epoch (the default, with or without a SimHost; wall_clock() opts out). Time jumps to the next pending sleep or timeout once every thread is blocked, and each call that returns without blocking costs a microsecond, so busy-polls still let time move. It can also run scaled to real time (time_rate), be set forward (set_time_value), and be paused, which holds every sleep and timed wait until time is moved from outside — see Controlling time. |
| Entropy | getrandom/getentropy/arc4random/ProcessPrng — seeded, so HashMap order etc. is repeatable. |
| TCP | std::net::TcpStream/TcpListener against scripted connect_tester peers, or against each other: the code under test can bind/listen/accept (accept4 flags on Linux) and connect to its own listeners, with shutdown(2) (SHUT_RD/SHUT_WR/SHUT_RDWR) and bind conflicts (EADDRINUSE, SO_REUSEADDR) as the host OS has them. |
| UDP | std::net::UdpSocket serviced from memory on every platform: bind/connect/sendto/recvfrom/sendmsg/recvmsg (and Linux sendmmsg/recvmmsg), blocking cross-thread receive, broadcast (SO_BROADCAST), multicast (IP_ADD_MEMBERSHIP, Linux IP_MULTICAST_ALL), Linux IP_RECVERR ICMP reports on MSG_ERRQUEUE, and several addresses sharing a port. Under a SimHost it adds SCM_TXTIME launch deadlines, and its sockets work with poll and epoll. TCP and UDP run together in one Sim. |
| Packet timestamps | Kernel receive and transmit stamps from the sim's clock on the plain Sim and under a SimHost: Linux SO_TIMESTAMP/SO_TIMESTAMPNS/SO_TIMESTAMPING (SCM_TIMESTAMP*, UDP and TCP) and MSG_ERRQUEUE transmit stamps (OPT_ID, OPT_TSONLY, TX_SCHED/TX_ACK); macOS SO_TIMESTAMP/SO_TIMESTAMP_MONOTONIC/SO_TIMESTAMP_CONTINUOUS; Windows SIO_TIMESTAMPING on UDP (WSARecvMsg's SO_TIMESTAMP, SIO_GET_TX_TIMESTAMP by SO_TIMESTAMP_ID). A packet is stamped when it reaches the socket, after its link delay. See Packet timestamps. |
| Raw L2 | Linux AF_PACKET/SOCK_RAW + bind(sockaddr_ll); macOS /dev/bpf* (BIOCSETIF/bpf_hdr). What EtherCAT masters (ethercrab) use. Interfaces come from the sim's topology (add_nic), and frames respect their link state and MTU. |
| Interfaces and routing | One topology per sim (NicSpec, Route, add_nic, set_link, schedule_link, route_lookup) that every backend routes real traffic through: longest-prefix routes, source selection, carrier and admin flaps that stall TCP and drop datagrams, per-interface latency/jitter/loss (NicPolicy) and counters, and bind-to-device (SO_BINDTODEVICE, IP_BOUND_IF, IP_UNICAST_IF, or set_socket_device from the test) with the host OS's errors. See Interfaces and routing. |
| Interface enumeration | if_nametoindex/if_indextoname/if_nameindex, getifaddrs, SIOCGIF*, SimHost sysfs, rtnetlink RTM_GETLINK/GETADDR/GETROUTE, macOS NET_RT_IFLIST2, and Windows IP Helper (GetIfEntry2, GetAdaptersAddresses, ConvertInterface*, GetBestRoute2) all read the topology. See Enumerating interfaces. |
| Readiness | epoll + eventfd (Linux) and kqueue/kevent (macOS, incl. EVFILT_USER wakers), so mio-based code works natively on both. Edge-triggered registrations (EPOLLET, EV_CLEAR) are reported once per event as the kernels do — data arriving, an eventfd write or a NOTE_TRIGGER is a new edge even on an fd already ready — so a fired mio::Waker or an idle writable socket does not wake every poll; EPOLLONESHOT, EV_ONESHOT, EV_DISPATCH and EV_DISABLE/EV_ENABLE are honoured. Terminal errors follow edge-triggered delivery; newly visible errors or timestamp reports produce new edges. Level-triggered waits repeat while ready. poll(2), WSAPoll and Winsock select block and honour their timeouts. |
| Socket faults and back-pressure | Raised socket errors (raise_socket_error), ICMP port unreachables (injected, from a tester, and sent back automatically after the round trip for a datagram no socket takes), one-way link stalls that hold traffic in flight (quiesce), TCP send and receive buffers (SO_SNDBUF/SO_RCVBUF) and a receive window (TcpPolicy::recv_window) that block sends with SO_SNDTIMEO and host-OS writable thresholds, and SO_LINGER aborts and waiting closes — each with the host OS's codes and call order. See Socket faults and back-pressure. |
| Connect faults | A listening address refuses (ListenerBehavior::Refusing) or drops SYNs until an instant (DelayingUntil), and an address nobody answers at makes a connect wait out the host OS's SYN retransmission plan — Linux 131 s (tcp_syn_retries, tcp_syn_linear_timeouts, TCP_SYNCNT), macOS 75 s (TCP_CONNECTIONTIMEOUT), Windows 21 s (TCP_MAXRT) — in virtual time. Nonblocking connects report EINPROGRESS/WSAEWOULDBLOCK, then poll/epoll/kqueue/select and SO_ERROR, so connect_timeout and mio behave as on the real OS. See Connect faults. |
| Protocol counters | The host's UDP and TCP counters, driven by the sim's traffic: Linux /proc/net/snmp and /proc/net/snmp6, macOS sysctl net.inet.udp.stats, Windows GetUdpStatistics(Ex/Ex2) and GetTcpStatistics(Ex/Ex2), and proto_counters() for the test. See Protocol counters. |
| Netlink | AF_NETLINK rtnetlink RTM_GETLINK/IFLA_STATS64 (a dump, or one link by index or name), genetlink CTRL_CMD_GETFAMILY, RTM_GETQDISC/ETF. |
| Files | VirtualFs: an in-memory tree behind open, sequential/positional IO, truncate, sync, stat and directory-stream calls, with glob passthrough to real paths. SimHost also renders /sys, /proc, /dev/ptp*, /dev/cpu_dma_latency (the request held open reads back through SimHost::cpu_dma_latency). Raw directory syscalls and path mutation are incomplete. |
| Host tuning | Scheduling (sched_setscheduler/affinity/priority, mlock/mlockall), resource limits (RLIMIT_RTPRIO/NICE/MEMLOCK), NIC config (ethtool rings/coalescing/channels/pause/EEE/flags/flow rules/stats, SIOC[GS]HWTSTAMP, qdiscs and ETF offload, threaded NAPI, queues/IRQs), PTP, capability gating — all modeled, never touching the real scheduler or NIC. |
| Kernel identity | Under a SimHost, uname (and Linux SYS_uname) reports the profile's sysname/nodename/kernel_release/kernel_version/machine: a fixed kernel by default, whose Linux version string carries PREEMPT_RT when the profile does, or the build machine's own with real_uname(). gethostname still reads the real name. |
| Tester stages | A tester's message chain: then_test / then_stateful_test drop or rewrite a message before later stages, then_edit_state updates state, then_action / then_stateful_action answer — run in the order added. |
| Thread classes | Every thread in a Sim is a participant unless marked (snare::sched::mark_background, mark_helper, mark_driver_thread, spawn_as): only participants hold up quiescence and the deterministic schedule. busy() / setup_scope() leases hold time still; thread names from pthread_setname_np / SetThreadDescription label each thread. SimBuilder::stuck_after fails a run a spinning participant has frozen. See Stuck runs. |
| Cooperative primitives | snare::sched::park / current_unparker, block_on / block_on_until / block_on_timeout, the Sleep future (sleep_until), WakerSet and is_driven: in a sim they wait on its clock and schedule — counted toward quiescence, their deadlines virtual timers, their wakes held at an executive's gate — and off one on real time. See Cooperative primitives. |
| Executive | An outside simulation owns a sim's clock in-process (Sim::executive, snare::sched::attach): it lets time flow in grants up to a horizon, jumps it timer group by timer group once the sim is quiescent, and acts at timestamps of its own whose effects reach the participants only when the timestamp ends — with a quiescence/participant/timer listing and an audit of what it cannot see. See Executive. |
| Socket table | One record per open socket of the code under test on every backend (socket_table(), socket_entry(id), socket_id(&sock), closed_sockets(), sockets_bound(addr)): kind, local and peer address, the listener an accepted stream came from, multicast memberships, queued/delivered/sent counts, the pending SO_ERROR and when it opened and closed. See Socket table. |
| Socket limits and privileges | SO_RCVBUF/SO_SNDBUF rounded as the host OS does, each datagram admitted against the receive buffer at its arrival (Linux truesize, macOS sb_cc/sb_mbcnt, Windows payload), FIONREAD/SIOCINQ/SIOCOUTQ, SO_NREAD, SO_MEMINFO, SO_RXQ_OVFL, SO_COOKIE; one Privileges per sim gating reserved-port binds, SO_*BUFFORCE, SO_PRIORITY/SO_MARK, interface rebinds, AF_PACKET, /dev/bpf* and a SimHost's capabilities. See Socket limits and privileges. |
| Names | getaddrinfo/freeaddrinfo, getnameinfo, gethostbyname (and Winsock's GetAddrInfoW/GetNameInfoW) answered from a per-sim host table (add_host) plus a built-in localhost — unknown names fail with the host's own code (EAI_NONAME, WSAHOST_NOT_FOUND), never real DNS; numeric hosts and hints stay the real libc's. Per-name latency and seeded failures (set_dns_policy). See Names. |
| Signals | sigaction/signal/raise/kill/pthread_kill for SIGINT/SIGTERM/SIGHUP (Windows: SetConsoleCtrlHandler, GenerateConsoleCtrlEvent, the CRT's signal/raise) against per-sim dispositions; Sim::raise_signal delivers as the host OS would, so ctrlc and signal-hook run unchanged. The semaphores and socketpairs their threads block on (sem_*, socketpair(AF_UNIX), Windows semaphores) are simulated too. See Signals. |
| Recorded events | A sim-wide log (recorded_events(), Sim::recorded_events) of what crossed each tester's boundary, what link policies did to datagrams, policy changes, injected faults and delivered signals, stamped on the sim's clock. |
| Packet capture | SimBuilder::pcapng(path) (or SNARE_PCAPNG_DIR for every sim, SNARE_PCAPNG_TESTS for chosen tests) writes what crosses the sim's network to a pcapng file Wireshark opens: fabricated Ethernet/IP headers around each TCP connection (handshake, segments, ACKs at arrival, FINs, resets, SYN retransmissions), each datagram once at its sender, ICMP port unreachables and raw L2 frames verbatim, stamped on the sim's clock per interface, optionally commented with real wall time. See Packet capture. |
| Environment | getenv/setenv/unsetenv lookups and mutations are isolated (opt-in). macOS enumeration uses the simulated snapshot; Linux vars/vars_os still read the real process environment. |
| Windows | The thread-scheduling plane (SetThreadPriority/affinity/priority-class/timeBeginPeriod, MMCSS AvSetMmThreadCharacteristicsW/AvSetMmThreadPriority/AvRevertMmThreadCharacteristics, power throttling through Set/GetProcessInformation and Set/GetThreadInformation, CPU sets) runs against a WinHost, read back with Sim::time_periods, mmcss_tasks, process_power_throttling, thread_power_throttling, process_default_cpu_sets and thread_selected_cpu_sets, and std::net::TcpStream/TcpListener/UdpSocket are serviced from memory by a Winsock fabric (socket/bind/listen/accept/connect/send/recv/sendto/recvfrom with MSG_PEEK/MSG_WAITALL and WSAEMSGSIZE truncation, the synchronous WSASend/WSARecv/WSASendTo/WSARecvFrom/WSASendMsg and the WSARecvMsg extension, blocking recv/accept, UDP broadcast/multicast/multi-IP, don't-fragment, WSAIoctl including SIO_CPU_AFFINITY, read back with Sim::socket_cpu_affinity). WSAPoll and select readiness and WSADuplicateSocket (try_clone) are modelled, IOCP is planned; a sim socket never reaches Winsock — overlapped calls and event/window-message notification on one fail with WSAEOPNOTSUPP. A pcap/npcap interposer is in place structurally. |
Registered hooks record unsupported calls in Domain::unmodelled(). An empty report is a useful
check, but does not prove full simulation: unhooked functions, direct process-global reads and CPU
instructions are outside that accounting.
Configuring the host: SimHost, HostProfile, EasyBuilder
For code that tunes the host (real-time schedulers, NIC configurators like fast-talker), attach a
SimHost describing the machine. The OS personality is fixed by cfg!(target_os) at compile time; what
you configure here are host facts.
use ;
use Ipv4Addr;
let host = new
.cpus
.isolated // /sys/devices/system/cpu/isolated
.governor
.preempt_rt // /sys/kernel/realtime
.cap // grant a capability; setters EPERM without it
.nic
.env // isolates + sets the environment
.kernel_release // uname -r; also sysname, nodename, kernel_version, machine
.build;
builder.host.build.run;
uname reports a host named snare running a fixed kernel (Linux 6.12.0, #1 SMP PREEMPT_DYNAMIC,
or #1 SMP PREEMPT_RT with preempt_rt(true); macOS Darwin 25.0.0) on the build target's machine
type. HostProfile::real_uname() copies all five fields from the machine running the test instead,
and works inside a sim.
EasyBuilder gives batteries-included presets when you just want a working machine:
use ;
let host = realtime.build; // 8-CPU PREEMPT_RT, all caps, a hw-timestamping NIC
// also: EasyBuilder::server() / laptop() / unprivileged() / minimal(), each tweakable:
let host = laptop.cpus.privileged.build;
builder.host.build.run;
Files: VirtualFs
use ;
let fs = new
.file
.dir
.passthrough // globs that reach the real filesystem
.deny // globs that always fail
.build;
builder.fs.build.run;
Separate opens of a file share live contents; duplicated descriptors also share offsets and status
flags. Writes and truncation are visible immediately. Absolute mkdir, unlink and rename
operate on the virtual tree, preserve inode identity across moves, and keep unlinked files usable
through open descriptors. Creation requires a declared directory parent, and pathname traversal
checks intermediate components before resolving . and ... Relative path mutations,
rmdir, getdents64, fdopendir, permissions and sparse storage need additional modelling.
See OPEN_BUGS.md for the file-plane boundaries.
Determinism and quiescence
Time is virtual, so a test that "waits a second" finishes instantly and every run sees the same
timestamps; randomness is seeded (SimBuilder::seed), per thread, so HashMap order, random ids and
link faults replay. When every thread is blocked with no timer left to fire, the run is deadlocked and
the blocked simulated wait gives up (EAGAIN) rather than hanging the test.
Timers due at one instant all fire there: a time skip lands 1 ns past the earliest deadline, and the
next skip waits until every timed wait it reached has returned, so a park, condvar or channel timeout —
which notices its deadline only between real-time slices of at most 2 ms — wakes at its deadline even
when a sleeper on another thread is due at the same instant and would otherwise skip straight on.
Likewise a thread woken from a native wait — a futex, condition variable or semaphore wake, or the
exit of the thread it joins — holds off every time skip until it has run.
A thread blocked on a lock that no thread of its sim holds — one held by another test running in
parallel, an unmanaged thread, or std's own process-wide locks such as the one every thread start
and exit takes — waits on the world outside, not on the sim, so it holds off time skips too until
it gets the lock. The pthread mutex hooks record which mutexes each of a sim's threads holds, so a
contended pthread mutex (std's Mutex on macOS) is judged exactly; a futex word names no holder, so
on Linux a lock wait on a word in static data (std's own statics) first waits up to 1 ms of real
time uncounted, which an outside holder in a short critical section lets go within, and a lock in
heap memory held outside the sim is still counted as a wait in it.
Thread interleaving is deterministic too with Sim::builder().deterministic(): one thread runs at a
time, and control passes only where a thread waits — a socket, sleep, mutex, condition variable,
channel, park, join or yield — to the next thread in a fixed order. Those waits are emulated rather than
handed to the kernel, so the waker always moves its waiters to the run queue itself and nothing is woken
behind the scheduler's back; locks shared with threads outside the simulation (another test running in
parallel) are waited out without letting their timing reorder this one. With the same seed, a run
replays exactly. What it cannot see it cannot order: a thread spinning on an atomic without ever
yielding keeps the baton, and a call that blocks in the OS outside snare's hooks blocks every thread.
A yield with no other thread able to run time-skips to the next pending timer, as a free-running
yield does once every other thread is parked, so a yield loop waiting on a sleeper (crossbeam's
Backoff::snooze, rayon's idle workers) gets there.
The sim's own code often runs on a participant from inside its hooked wait — the deterministic
schedule's dispatch and the time skips in it, or an idle skip as a native wait begins — where std may
already have parked the thread on its thread parker. std's parker tracks one park at a time, so a std
wait there that parks the same thread again (a channel; on macOS also a contended OnceLock, Once
or RwLock) would leave neither park able to be woken. snare's own process-wide singletons therefore
never wait to be initialised, and the parker's blocking calls (dispatch_semaphore_wait, Linux
futex, WaitOnAddress) treat a wait nested in another on the same object as a 1 ms real-time
slice that returns as a spurious wakeup, and wake the outer park to re-check, so a layer that blocks
in std there still makes progress.
Clock spins
Reading the clock never moves a discrete clock, so a thread waiting on time by polling it —
while Instant::now() < deadline {}, with or without yield_now, spin_sleep's final spin, a loop
that checks an unhooked channel between reads — would wait forever. snare catches such a clock
spin and moves time for it:
- A participant's clock reads and
sched_yield/SwitchToThreadcalls count toward a spin; any other hooked call (a socket call, a sleep, a wait, a thread spawn, a lock that contends, a signal) ends it — a wait as it begins, whether it then blocks or finds what it waits for already there, while one whose timeout is zero or already past is a poll and leaves the spin running — and so does each of the sim's own waits, such as a tester waiting for its next message or tick: the clock reads of a tester's loop and handlers never add up to a spin. After 64 in a row the spin is caught: isolated reads, a few back-to-back reads, or reads with other hooked calls between them never get there and keep holding still. A loop polling a socket is not a clock spin: each non-blocking call is charged its microsecond instead. - Each read of a caught spin moves time one step: 1/64 of the virtual time since the spin was caught, at least 1 µs. The steps grow geometrically, so an hour-long spin takes about 1,100 reads and a spin overshoots its own deadline by at most 1/64 of its length (one more read before any other hooked call moves it one more step). A step never jumps a pending timer: one that would reach the earliest timer lands 1 ns short of it, so a spinner whose deadline comes first sees it first, and the next lands just past it, as a time skip does, waking its waiter.
- Time moves only while every other participant is parked or spinning itself, no woken waiter has
yet to run and no lease is held, so other threads' work at the current instant comes first and
timers fire in order. A free-running spin waits for a waiter its step reached to take its
deadline (up to 5 ms of real time) before stepping on. Under
deterministic()each step is a scheduling point: the spinner hands the baton to any runnable thread that is not spinning, and after a step lets the threads it released run before the next. Runs replay exactly. - A yield loop that does not read the clock waits on another thread, not on a deadline of its own, and keeps the full time skip to the next timer; it gets none while a clock spinner is about, since that would jump past the spinner's deadline. A thread that read the clock since its last yield is polling, and its yields only hand others a turn.
- A paused clock, or one an executive drives, is never moved by a spin: the spinner keeps reading
the held instant, yielding the CPU, until something outside the sim moves or resumes the clock;
stuck_afterreports it. A spin that moves time is progress to the watchdog. A scaled or as-fast-as-possible clock moves on its own and awall_clock()sim reads real time, so neither is touched. Only participants' spins count; a background or driver thread spinning on the clock is left alone.
Thread classes and leases
Every thread created inside Sim::run is simulated — its sockets, clock, files and environment are
the sim's — and by default it is a participant: time skips only once every participant is blocked, a
blocked wait gives up only when they all are, and deterministic() runs participants one at a time. A
thread the run should not wait on changes its class with snare::sched:
| Class | Set with | Effect |
|---|---|---|
Participant |
the default; participate(name) for a scope |
Counts toward quiescence and runs in the deterministic schedule. |
Background / Helper |
mark_background(label) / mark_helper() |
Still simulated, but never holds up quiescence or the schedule. Its sleeps and timeouts are satisfied as the clock passes them rather than steering a time skip, and its non-blocking calls cost no time. |
Driver |
mark_driver_thread() |
As Background, and its clock reads, sleeps and timeouts (socket, poll and condition-variable timeouts alike) are real; sched::now() reads the sim's clock. |
spawn_as(class, label) makes the threads the caller creates while the guard lives start in class
(for a pool or library that spawns its own threads); later children, and a background thread's own
children, are participants again. Sim waits (a receive, a poll, an accept) still give up when every
participant is blocked with no participant timer left, however many background timers are pending.
Once no participant can give up — each is in a join, a lock, a condition variable or a channel
receive, waiting on what only a background thread can do, as when the test stops a sampler and joins
it — time skips to the background threads' timers instead, so the join returns. Under
deterministic() a thread that leaves the schedule (mark_background on a running participant, or a
participate guard dropping) hands the baton on as it goes, and a thread of another class runs
beside the schedule: whatever it wakes — a channel send, a condvar signal, a socket write — still
reaches the schedule, but when it runs relative to the participants depends on the OS, so its effects
on them only replay if they wait for it. The same holds for a background thread entering the schedule
through participate: the point where it joins depends on real time, so the run replays only if the
participants wait for it there (for a message it sends from inside the guard, say).
busy(label) (Send; also Sim::busy) and setup_scope(label) are leases: while one is held the sim
is never quiescent, so time does not skip and no blocked wait gives up, even on the discrete clock; a
deterministic schedule idles until the lease is dropped. Taking one waits out a time skip already in
flight, so from any thread, inside the sim or not, no skip lands once busy returns. A lease has a
kind: busy takes a LeaseKind::Busy one, for work the sim cannot see that lasts about as long as a
reaction, and setup_scope a LeaseKind::Setup one, which may last as long as setup takes without
anything being stuck. held_leases() lists them as LeaseInfo { label, kind, holder }, the holder
being the name of the thread that took it. Off a sim both return an inert lease (is_held() is
false) rather than panicking, so code shared with a real deployment can take them unconditionally;
sched::in_sim() says whether the calling thread runs in a sim at all, and sched::current_sim()
which one (a SimId, also Sim::id). A thread's name is whatever it set through pthread_setname_np or
SetThreadDescription (std's Builder::name), which go to the OS unchanged — a Linux name over 15
bytes still fails with ERANGE — or thread-<lineage> otherwise.
Stuck runs
A participant that runs without making a hooked call — spinning on an atomic, or computing — keeps
the sim from going quiescent, so the discrete clock cannot move and every sleeper it may be waiting
for waits for good: the test hangs. SimBuilder::stuck_after(d) (off by default) turns that into a
failure. A watchdog thread outside the sim samples it every eighth of d (between 1 ms and 250 ms)
and fires once, for d of real time, a participant has been running while none of the sim's
progress has happened: no participant blocked or woke, no lease was taken or given back, virtual
time did not move (a time skip, a clock write, a scaled or granted clock flowing, the microsecond
each non-blocking call is charged, a clock spin's step), and under deterministic() no participant began a wait in the
schedule. It never fires while every participant is blocked — on a paused clock, on an executive, on
locks only a thread outside the sim can release — since the design waits for those, nor while a
lease is held or an executive's timestamp is open, nor on a wall_clock() sim, whose time moves on
its own. A participant that legitimately runs longer than
d with no hooked call (a long computation, a blocking call snare does not hook, work inside
real) holds busy() across it. Off by default because that work looks exactly like a spin.
A stuck thread cannot be made to panic from outside, and the test would never finish, so the
watchdog attempts its report on the process's stderr — past the test harness's output capture —
with a 100 ms reporting budget, then aborts the test process. Locked or blocked stderr may leave
the report incomplete, and OS scheduling may delay detection and termination. The report names
every participant: running (and, under deterministic(), which one holds the baton), or what it
waits in and until when, what it last
waited in, and the leases it holds.
The same listing explains a deterministic schedule that has idled for a second with every participant blocked — on a held clock, on a lease, or on locks only something outside the sim can release: that is waiting rather than stuck, so it is a one-time warning on stderr and the schedule carries on.
Sim lifecycle
A Sim is running while at least one thread is inside Sim::run, and dormant between runs and
after it is dropped. Threads spawned inside a run that are still alive when it returns — a
background loop the test never stopped, a pool's idle workers — stay in the sim: their clock,
sockets and files are still its own, and the sim's state lives as long as any of them does, even
past the Sim's drop. But nothing waits on them any more, so while the sim is dormant:
- Time moves for them at real time's pace. A time skip lands no further past the reading the run ended at than real time has moved since, so a leftover thread sleeping in a loop sleeps in real time (a near-idle CPU) instead of racing the virtual clock ahead at full speed. An as-fast-as-possible clock holds still between such skips too. With no leftover thread waiting on a timer the clock does not move at all, so a sim without leftovers starts its next run where the last one ended. A paused clock, or one an executive still drives, stays held.
- A wait with nothing left to wait for blocks instead of giving up as a deadlock: a receive no
thread will ever answer stays blocked (polling a few times a second) rather than failing with
EAGAIN, so a leftover loop that retries on errors does not spin. - They no longer count anywhere else: not toward another sim's quiescence (each sim's woken
waiters are its own, however many sims run in parallel), not toward the
stuck_afterwatchdog (a leftover thread spinning on an atomic is the test's own business once the run has ended), and a deterministic schedule has already let them go to run freely. - They can still be stopped and joined from outside: set the flag they poll and join them; their sleeps end in real time.
The next Sim::run on the same sim takes its leftovers back: they are its participants again, and
time skips go back to virtual time at once (monotonic time never goes backwards across the switch).
A test that wants none of this stops its threads before the run returns.
Controlling time
The clock every Sim runs on can be driven from the test, through Sim methods or a TimeHandle
(sim.time(), or snare::time() from any thread running inside the sim):
| Call | Effect |
|---|---|
set_time_rate(r) / SimBuilder::time_rate(r) |
r in (0, 1e6]: virtual time runs at r times real time (larger values clamp). 0.0 pauses. f64::INFINITY returns to virtual time — discrete-event, or as-fast-as-possible for a SimHost or Windows sim built with wall_clock() — keeping the reading. Negative or NaN panics. |
time_rate() |
0.0 while paused, the rate while scaled, f64::INFINITY on virtual time. |
pause_time() / resume_time() |
Pausing holds the clock: reads return the same instant, and every sleep, socket timeout, poll timeout and condition-variable or futex timeout blocks until time moves, as nanosleep(2) and Sleep wait out their whole interval. Resuming restores the mode and rate in force before the pause. |
advance_time(d) |
Moves the clock forward by d, paused or not, waking what it reaches. |
set_time_value(v) / time_value() |
Sets sim time — how far the clock has moved since the sim was built — to v, CLOCK_MONOTONIC and the realtime clocks moving with it; forward only (setting it back panics and leaves the clock alone). time_value() reads without ticking. |
Monotonic time never goes backwards, whatever moves the clock: every monotonic source is
non-decreasing on each thread and across threads that synchronize, every mode switch re-anchors at
the last reading handed out, and a request to move the clock back is refused. Realtime starts at a
fixed epoch and moves with sim time; CLOCK_MONOTONIC (and Instant, mach_absolute_time,
QueryPerformanceCounter, GetTickCount64) starts at zero in every sim, so a replay reads the same
absolute values. The guarantee is per sim: an Instant taken outside the sim, or in an earlier sim
and kept in a static, is on a different timeline and may look ahead of one taken inside. A driver thread is the one
exception by design: it reads real time, and an executive's timestamp time while it acts at one.
A paused clock is moved by something outside the simulation — the test thread before or after run,
or a thread it starts under snare::real — so a sim where every thread waits on a paused clock is
waiting, not deadlocked; it says so on stderr after a second. Once resumed, a wait nothing can
satisfy gives up as before. On an as-fast-as-possible clock, a timed wait that is still held when
the clock resumes times out then, the clock jumping to its deadline as a sleep's does; one already
running when the clock is paused or scaled keeps the time it had left and holds with the clock. Under
deterministic() the clock can be paused, advanced and set; a write made while every simulated
thread is blocked on the paused clock releases the waiters in the schedule's fixed order, but one
made while a simulated thread runs lands wherever real time puts it in that thread's work, so the run
only replays if outside writes wait for the sim to block; an Executive's jump_to and
enter_timestamp_checked act only while the schedule is idle, so they always do. A real-time rate is refused, since nothing
in a deterministic run may depend on real time. A plain unix Sim built with wall_clock() runs on the
real clock and has none of these controls; time_rate(1.0) gives a controllable clock that tracks
real time. With a SimHost, rate, value and pause apply to its timestamps, PHC samples and
SO_TXTIME alike. Scaled time follows real time, so it is not reproducible run to run.
On Windows, Rust sleep durations that exceed its signed 100 ns timer range reach the
simulated clock before conversion, including std::thread::sleep(Duration::MAX). They
saturate at the clock's finite limit. Ordinary Rust sleeps retain the Windows timer grid;
native Sleep(INFINITE) and non-alertable SleepEx(INFINITE) still wait indefinitely.
This uses a typed hook on the linked Rust sleep implementation, following transparent entry
tail jumps. Entirely inlined implementations and separately linked std copies need their own
instrumentation. Hook installation errors are reported immediately.
Executive
A simulation that drives the code under test from outside — sham inside theater — owns the sim's
clock through an Executive, in the same process: sim.executive(ExecutiveConfig::default()) from any
thread, or snare::sched::attach(cfg) from a thread inside the run. Attaching fails with
AlreadyAttached, NotInSim (the free function off a sim's thread) or WallClock (a plain unix sim
on the real clock, which has no clock to own). From then until the executive drops, only it moves
time: there is no time skip on quiescence and no deadlock give-up — a wait nothing can satisfy waits
for the executive, and a deterministic schedule idles — and pause_time, resume_time,
advance_time, set_time_rate, set_time_value and their TimeHandle forms panic with "this
Sim's clock is owned by an Executive". The clock is frozen at its reading until the first grant.
| Call | Effect |
|---|---|
grant(Grant { anchor_v, anchor_wall, rate, horizon }) |
Time runs from anchor_v at the real instant anchor_wall, rate virtual seconds per real second (in [0, 1e6], NaN as 0), never past horizon. An anchor behind the clock anchors at the clock's reading, now. Timers the clock flows past fire as real time reaches them. |
freeze() |
Holds the clock where it is (horizon at the reading, rate 0). |
next_deadline() |
The earliest participant timer after the reading. Background timers are never jump targets; they fire as the clock passes them. |
jump_to(t) |
If the sim is quiescent — checked under the sim's locks in the same step as the move, so no participant starts running in between — moves the clock to t or to the earliest participant timer before it, exactly onto that deadline, and returns how many waits and events were due there; Err(NotQuiescent) changes nothing. The horizon rises to reach the landing; the rate stays the grant's. |
enter_timestamp(t) / leave_timestamp(t) |
The calling thread becomes the driver and acts at t: it reads exactly t from every clock and the code under test reads at least t, while the clock itself stands just before t, so timers at t are still pending. Participants released meanwhile — by what the driver sends, or by timers before t — wait at a gate, still counted blocked, until leave_timestamp(t) moves the clock to t, fires what is due there (returning the count) and opens the gate. It wakes only waiters something reached: a timestamp at which nothing was sent and nothing came due leaves a quiescent sim quiescent. |
enter_timestamp_checked(t) |
As enter_timestamp, only if the sim is quiescent and no participant timer falls before t, checked in the same step; on Err nothing changes but the calling thread becoming the driver. |
with_driver_time(t, f) |
On a driver thread (a pool worker doing the executive's work), runs f reading t as the sim's time; its sends are held at the gate like the executive's own. |
quiescence() |
quiescent, the epoch, runnable/blocked participants, leases, next_deadline, and the first blocker: Runnable (a running participant, by name) > Settling (a woken waiter, a participant a futex, condition-variable, semaphore or WaitOnAddress wake released, one waiting for a pthread mutex that is now free, a joiner whose thread exited, or a participant the gate let go at the end of a timestamp, yet to run) > Lease (a busy lease) > Setup (only setup leases) > Deferred (held at the gate) > TimerDue (a timer at or before the reading its waiter has yet to take). |
arm_notify(epoch, f) |
Calls f once the epoch moves past epoch — a participant parking or waking, a lease taken or given back, a class change, a deterministic schedule going idle — on the thread that moved it, with no lock of the sim held. |
participants() / timers(n) / held_leases() / drain_hints() |
Every participant (lineage, name, running or blocked, for how long in sim and real time, what it waits in, its deadline, what it last waited in, the leases it holds) then one row per lease, with its lease_kind; the earliest timers, participants' before the others', with their owners' names; the leases with their kinds; the hint_starving hints since the last drain. |
outside_wakes() |
Unmatched returns from participants' native waits since the executive attached: nothing snare observed since quiescence explains the return. Always counted, with or without an audit. Spurious OS returns, such as those allowed by WaitOnAddress, can contribute; this is not proof that an unmanaged thread released the waiter. A run requiring every wake to be accounted for asserts it stays 0. |
audit() |
With ExecutiveConfig { audit: true } (or SNARE_SCHED_AUDIT set to anything but empty or 0): background and helper threads' sim-visible effects (sends, connects, closes, futex/condvar/semaphore/address wakes, sleeps, busy), counted apart inside a setup_scope; the outside wakes; and every call the sim cannot model, as a violation. |
thread_census() / sim_id() |
Every OS thread of the process and whose it is (see Thread census); the sim's id. |
Dropping the executive (or detach) returns the clock to its base mode at its reading — never
backwards, and past an open timestamp's time — opens the gate, and time skips and deadlock give-ups
resume with the OS's own errors.
The gate holds participants leaving a sim wait (a socket, poll or sleep), a futex, a semaphore, a
WaitOnAddress, a condition variable or a join. A condition-variable wait woken inside a timestamp
lets go of its mutex while it waits at the gate and takes it back after, which the caller sees as an
ordinary wakeup. A contended mutex returns holding the code under test's lock, so it passes the gate
rather than risk blocking the executive's own threads on that lock. A participant already running when an unchecked enter_timestamp begins can see what the
driver does at t early, as can background threads, which the gate never holds. Under
deterministic() a timestamp also holds the schedule's baton: enter_timestamp waits for every
participant to block, enter_timestamp_checked takes it only if the schedule is idle, and
leave_timestamp hands it to the participants in the schedule's own order, whatever order the data
arrived in. A grant's rate runs as 0 there, with a one-time warning, so a deterministic run driven by
jumps and checked timestamps replays exactly; a flowing grant follows real time by design.
Outside deterministic(), a participant released by another thread's wake counts as Settling from
the wake until it runs: the futex, condition-variable, semaphore and WaitOnAddress hooks count,
before the OS wakes anyone, how many participants were parked on that address (up to the wake's
count), and an exit counts its joiners. A pthread mutex is judged by who holds it instead: the lock,
trylock and unlock hooks track the holder of every mutex a participant waits for, so a participant in
a contended lock is Settling only while that mutex is free or held outside the sim, and one woken
from a condition variable only while its mutex is free or its own. A holder that
re-takes the lock before the woken waiter runs, or that notifies and then sleeps holding the lock,
leaves the waiter blocked and the domain quiescent. A wake the OS hands to a thread that is not a
participant (a background thread waiting on the same address) leaves the parked participant counted
Settling until it next wakes, and a wake or unlock from a thread outside the sim is not seen at
all.
Whatever makes a participant runnable keeps the sim from being quiescent until that participant has
run: a datagram, bytes or a connection delivered to a socket it waits on (and any other readiness
change) marks every waiter woken as Settling until it has taken the lock again and re-checked — a
waiter that let go of the lock for a moment on its way back to sleep counts as asleep meanwhile, so a
wake landing then marks it too; an Unparker::unpark is such a readiness change; a futex,
condition-variable, semaphore or address wake counts as above; and a participant the gate held stays
counted until it has left its wait, so the next enter_timestamp_checked cannot slip in between the
gate opening and the participant running. A datagram still in flight is a timer at its arrival, not
a pending wake. A readiness change wakes only the waiters of the sim it belongs to (and threads
outside every sim): sims driven side by side by executives of their own never wake each other's
waiters, so none reads Settling because of a neighbour, and a jump made right after reading
quiescent goes through however busy the sims beside it are.
A driver time (enter_timestamp, with_driver_time) lapses on every thread when the executive is
dropped: the driver reads real time again.
Migrating sham and theater: sham and theater are the simulator and are never interposed. Theater
builds the Sim (seed from SimBuilder::seed) and hands it to sham's SnareDomain::new(&sim, cfg),
which owns the clock through sim.executive(cfg) in place of the old attach_driver, before the code
under test starts. The code under test (conductor) runs inside sim.run; everything theater does
with sham, and every thread sham starts, runs under snare::real, so none of it joins the sim and
nothing marks thread classes or uses driver time. Theater talks to the code under test only through
the Sim API: its network (testers, or emulator threads woken from outside), faults and time.
There is no global slot or register_test, and no environment handshake; the thread census is
sched::thread_census() (below).
Thread census
sched::thread_census() (also Sim::thread_census, Executive::thread_census) lists every OS
thread of the process — read from /proc/self/task on Linux, task_threads on macOS and a Toolhelp
thread snapshot on Windows — as a CensusThread { os_id, name, owner }, where owner is
ThisSim(class) for the calling thread's (or that) sim, OtherSim(id, class) for another sim's,
Snare for snare's own service threads (the link-delay and timer wakers, an executive's flow thread,
a stuck watchdog, the signal forwarder) and Unmanaged for everything else: the test harness, a
thread started outside every sim or under snare::real, one a runtime started for itself.
ThreadCensus::unmanaged(), this_sim() and other_sims() filter it. A thread counts as a sim's
from its own first line in the sim until it leaves, so one being created or exiting can read as
unmanaged for that moment; a run that fails on unknown threads should want one found unmanaged in
two censuses in a row, as theater's tracker did with snare v1.
Cooperative primitives
snare::sched has the blocking and future primitives a simulation's own runtime (sham's
participants) is written against. In a sim they run on its clock and its schedule; off one, on real
time.
| Call | Effect |
|---|---|
park(deadline) / current_unparker() |
Blocks until the thread's Unparker fires or the deadline passes (ParkResult::Unparked / TimedOut). An unpark that came first is consumed and returns at once. In a sim the park is a sim wait: it counts toward quiescence, its deadline (an Instant, read on the sim's clock) is a timer a time skip or an executive's jump lands on, a wake during an executive's timestamp is held at the gate, and under deterministic() it waits in the schedule. A participant parked with no deadline while nothing in the sim can move keeps waiting, as a join does. Unparker::unpark and Unparker::waker() work from any thread. |
block_on(f) / block_on_until(f, deadline) / block_on_timeout(f, d) |
Polls f, parking between polls with the thread's Unparker as the waker. Past the deadline f is polled once more, so an output ready at the deadline is not lost; a timeout past the end of the clock is no deadline. |
sleep_until(instant) → Sleep |
A future that completes when the clock reaches the instant: on its first pending poll its waker becomes a timer of the calling thread's class (a participant's steers time skips, a background thread's fires as the clock passes it or, once every participant waits on what only such a thread can do, steers them too), which the discrete time skip, Executive::jump_to, a flowing grant, a scaled clock reaching it (a real-time waker set for the real arrival) or a clock writer fires. Later polls replace the waker; dropping it, or completing, removes the timer. is_elapsed() checks the clock. On an as-fast-as-possible clock waiting on it jumps the clock to the instant, as a sleep does. Any waker works, not only block_on's: a hand-rolled executor blocking in a channel receive or a condition variable between polls is woken at the instant, under deterministic() too. The clock runs the wakers it fires on a thread of its own, outside the sim. |
WakerSet |
The wakers of every task waiting on one piece of state: register before checking the state, wake_all after changing it. |
is_driven() |
The calling thread is a participant and the sim's clock is virtual (discrete, scaled, paused or owned by an executive). |
A driver thread outside an executive's timestamp reads real Instants, so its Sleeps and park
deadlines are on real time; such threads should time their waits with real-time timers, or under
with_driver_time.
Tester stages
Each message the code under test sends a tester runs through the tester's stages in the order they were added:
| Stage | Effect |
|---|---|
then_test(|msg, from| Option<P>) / then_stateful_test |
Some passes that message — rewritten or not — to the stages after it; None drops it there. |
then_edit_state(|state, from| ..) |
Updates the tester's state, then passes the message on unchanged. |
then_action(|msg, from| TesterAction) / then_stateful_action |
Acts on the message at once; later stages still see it. |
A stage added before a dropping test has already run for that message. recording() keeps what the
code under test sent, before any stage sees it, and on_connect and cyclic actions do not go through
the chain. Stages added before with_state keep working without the state.
Framing
A tester's message type says how the byte stream is cut into messages (Packet::parse). Besides
Bytes (whatever each read returns) and Line (\n-ended text), two framings cover most device
protocols without a hand-written Packet; both keep the raw frame (frame()) and send frames as
they are:
| Type | Frame |
|---|---|
Delimited<D> |
Up to and including the first D::DELIMITER: Lf, Cr, CrLf, or any byte string on a marker type implementing Delimiter. Delimited::new(body) appends the delimiter; body() / text() drop it. |
LengthPrefixed<L> |
L::FIELD: LengthField { offset, width, endian, adjust }: the unsigned width-byte (1, 2, 4, 8) field at offset, Endian::Big or Little, plus adjust, is the whole frame's length — adjust is the header and trailer size when the field counts the payload, 0 when it counts the whole frame. |
/// [0x02][type][len: u16 LE][payload][0x03]: the length counts the payload only.
;
let tracker = ;
let robot = ;
Over TCP a frame split across reads waits for the rest and several frames in one read come out one
by one. Over UDP each datagram is parsed on its own: it may carry several frames, a frame never
spans datagrams, and an incomplete tail is dropped with its datagram (with Bytes, one datagram is
one message). A LengthPrefixed field giving a frame shorter than the field's own end panics the
tester, which fails the test.
Cyclic actions and phase
with_cyclic_action(period, ..) runs first one period into the run, then every period;
with_cyclic_action_at(period, phase, ..) runs at phase, phase + period, ... — so two emitters on
an 8 ms cycle with phases 4 ms and 8 ms tick exactly half a cycle apart. Every tester in one
run_testers! is phased from the same start, read on the sim's clock, and the next tick is always
the previous one plus a period, so ticks do not drift. with_stateful_cyclic_action[_at] adds the
tester's state.
Per-connection state
with_state(S) is one value for the whole tester. with_conn_state(|peer| C) adds one value per
peer, made as it appears — per TCP connection accepted (a reconnect starts fresh), per UDP source
address — and the conn methods see it beside the tester's state:
| Method | Sees |
|---|---|
then_conn_action(|conn, state, msg, from| ..) / then_conn_test |
The sender's C; stages in the same chain as the others. |
on_conn_connect(|conn, state, from| ..) |
The new peer's C, just made. |
with_conn_cyclic_action[_at](period, [phase,] |conn, state, peer| ..) |
Each tick, once per live peer (open connections, every UDP source) in the order they appeared, with its C; a plain Send goes to that peer only. A plain cyclic action sees no C and sends to every peer. |
until_conns(|state, conns| bool) / inspect_conns(|conns| ..) |
Every peer's (SocketAddr, C) of the run, closed connections included, kept until the next run. |
with_state and with_conn_state go in either order, and stages added before either keep working
without it.
Recorded events
Every Sim keeps one log of what happened at the edge of the code under test, readable with
snare::recorded_events() from any of its threads or sim.recorded_events() from anywhere, during
the run or after it. Each RecordedEntry has at, sim time (time_value()) without ticking the clock
(real time since the sim was built on a plain wall_clock() sim), a seq that only ever grows, and
the RecordedEvent:
| Event | Logged when |
|---|---|
Accepted |
a TCP tester accepts a connection, before on_connect runs. |
Received { transport, len } |
a tester parses a message; len is the bytes it was parsed from. |
Sent { transport, len } |
a tester's bytes go toward the code under test. |
Closed / Reset |
a tester closes or resets a connection, once, including the closes at the end of its run. |
PeerClosed / PeerReset |
the tester reads end-of-stream, or a reset it did not cause. |
Quiesced { span } / Suppressed { toward, len } |
a tester goes silent toward a peer, and each message it then drops in either direction. |
Link { from, to, len, fault } |
a UdpPolicy loses a datagram, drops it for its MTU (TooBig) or duplicates it. A perfect link logs nothing. |
UdpPolicyChanged / TcpPolicyChanged |
set_udp_policy / set_tcp_policy, with the policy now in force. |
Fault { addr, fault } |
an injected fault takes effect on the code under test: Error { errno, call } when a call reports a raised error, IcmpPortUnreachable { from } when a call reports an ICMP error, Stalled { span, direction } when quiesce holds a link, Unreachable { errno } when a send or connect finds no route or a connect finds no neighbour, LinkDown { nic } when a send leaves through an interface without carrier; Fault::Dns { name, failure } when a DnsPolicy fails a lookup; ConnectRefused (a Refusing listener), ConnectTimedOut and ConnectDelayed { until } when a connect that waited on its SYN ends. |
NicChanged { nic, admin_up, carrier } |
an interface is added, removed, or changes admin state or carrier (a scheduled change logs when it takes effect). |
Signal { signal, origin, delivery } |
a signal or console control event reaches the code under test, sent by the test (Sim), the code itself (Process) or forwarded from the real process (Real). |
UnmodelledOption { socket, local, option, refused } |
the code under test first uses, on that socket, a socket option or ioctl the sim does not model; refused under strict_sockopts. See Unmodelled options. |
clear_recorded_events() starts a fresh phase (sequence numbers carry on), and
SimBuilder::record_events(false) keeps nothing, for a long, chatty run. Recording never moves the
clock, yields or wakes a thread, so a deterministic() run logs the same entries, in the same order,
with the same stamps, every time, and logs or not without changing the run; on the free-running
virtual clock, events from different threads at the same instant may come out in either order.
Coming from the original snare: UdpSendFromTest { dropped } is a Sent followed by a
Link { fault: Lost | TooBig }.
Packet capture
Sim::builder().pcapng(path) writes every frame that crosses the sim's network to a pcapng file,
created (or truncated) at build — which panics if it cannot be — and finished when the Sim
drops. Sim::pcapng_path() says where. With no explicit path, setting SNARE_PCAPNG_DIR=<dir>
captures each sim to <dir>/<building thread's name>.pcapng (: becomes -, other characters
outside [A-Za-z0-9._-] _, and a second sim built by the same thread gets -2, then -3); if
that file cannot be written the sim warns on stderr and runs without capture.
To capture chosen tests without touching their code, set SNARE_PCAPNG_TESTS to a comma-separated
list of test thread names — cargo names each test's thread after its path, module::test — and
only sims built on a listed thread are captured, to SNARE_PCAPNG_DIR if set and otherwise to
snare-pcapng in the temporary directory, each saying where on stderr. An entry matches the whole
name or its last :: segments, so my_test lists tests::my_test; a blank list counts as unset.
An explicit pcapng(path) is captured either way.
$ SNARE_PCAPNG_TESTS=my_test,other_mod::another_test cargo test
Stamps are always the sim's clock. SimBuilder::pcapng_wall_comment(true), or
SNARE_PCAPNG_WALL_COMMENT set to anything but empty or 0, also writes the real wall time each
frame was sent at as its EPB's opt_comment (draft-ietf-opsawg-pcapng §3.5), in UTC as
wall 2026-10-02T09:15:42.123456789Z, to line a capture up with logs kept on real time; a frame
due after it was sent (an ACK after latency, a planned retransmission) carries the real time it was
written, when the capture first saw the sim's clock reach it. Wireshark lists it as the packet's
comment. The comments make the file differ from run to run.
The sim has no real frames, so capture fabricates them, once per frame at its sender:
| Traffic | Frames |
|---|---|
| TCP connect | SYN, SYN|ACK, ACK when it is accepted; SYN then RST|ACK when refused; a connect that waits shows the host OS's SYN retransmissions at their plan points up to the answer, and nothing for a timeout. A refused connect still takes an ephemeral port. |
| TCP data | Each accepted write as MSS-sized PSH|ACK segments (MSS = egress MTU − 40, − 60 for IPv6), and the receiver's ACK stamped when the bytes arrive (TcpPolicy latency, a stalled or down link). On macOS, data reaching a TCP endpoint after SHUT_RD gets a reset instead of an ACK. |
| TCP close | FIN|ACK from shutdown(SHUT_WR), the last close or a tester's close, once per end, with the peer's ACK when the last byte in flight arrives; SHUT_RD itself sends nothing. RST|ACK from a tester's Reset, SO_LINGER 0, a closing listener's unaccepted connections or incoming TCP data after macOS read shutdown. |
| UDP | Each datagram at its sender — fabric, Winsock fabric, SimHost (stamped with its SO_TIMESTAMPING TX stamp) and tester endpoints — before any link policy, so loss and duplication never change the count and a frame lost on a down link is still there. A send refused (EAGAIN, EACCES, no route) writes nothing. |
| ICMP | A port unreachable (ICMPv6 type 1 code 4) from the unreachable address, stamped when the datagram got there; an injected one at once. |
| Raw L2 | AF_PACKET writes, /dev/bpf* writes and npcap sends, verbatim on their interface. |
Frames are Ethernet II with IPv4 (DF, a per-source ID, TTL 64, or 128 on Windows) or IPv6 (v4-mapped
addresses are written as IPv4), real IP/TCP/UDP/ICMP checksums, an MSS option on SYNs and a
window of 65535. Sequence numbers start from a per-connection ISN drawn from the sim's seed and the
connection's addresses. MACs are all zero on loopback, the group address for broadcast and
multicast, the owning interface's NicSpec::mac for the host's addresses, and a locally
administered hash of the address otherwise. Each interface a frame crosses gets an interface block,
named after it, the first time it is used; stamps are nanoseconds of the sim's CLOCK_REALTIME
(real time on a plain wall_clock() sim); epb_flags marks frames the code under test sent as
outbound and its testers' as inbound.
Frames due later (a retransmission, an ACK after latency) wait until the sim's time reaches them,
and the file is complete after each write. A waiting connect's planned retransmissions, and every
frame after them, stay unwritten until the connect has played those points of its plan, so a
nonblocking connect checked late still shows only the SYNs before its answer, and its handshake at
the plan point that answered it; a connect left waiting holds the rest of the capture until it is
answered, closed or the sim drops. Frames are written in time order: one stamped before a frame
already written (a sender that read its stamp just before another thread wrote) takes that frame's
stamp. On Windows, a refused connect keeps probing for the host's refusal window; the capture shows
each SYN with the RST|ACK answering it, and the handshake if a listener appears in time.
Capture only observes: it never moves the clock, yields, wakes a thread or draws from the seed, so a
run behaves the same with or without it, and under deterministic() the same seed writes the same
bytes (wall-time comments aside). Keep the file on a local disk — a slow write holds up the thread that sent the frame.
Interfaces and routing
Every Sim has one topology of interfaces, addresses and routes, shared by the fabric, the Winsock
fabric and a SimHost's UDP sockets. It holds host facts only; what the host does with them —
error codes, strong or weak host, what happens to a link that loses carrier — is the build host's.
use ;
use Duration;
let sim = builder
.nic
.route
.build;
sim.run;
- Default topology. Loopback at index 1 (
lo/lo0/Loopback Pseudo-Interface 1, with127.0.0.0/8and::1, MTU 65536 / 16384 / 4294967295) andsim0at the next free index with no addresses and the default routes (0.0.0.0/0,::/0, metric 100). ASimHostthat declares interfaces registers them instead (Nic::network,Nic::station,Nic::policy,link_statsas the counters' start). Initial carrier followsoperstate == "up"unlessNic::carrieroverrides it;.operstate("unknown").carrier(true)represents an unknown-but-up interface. The default routes leave through its lowest one unlessHostProfile::routedeclares its own. - Open, then routed. Until an interface has an address, the sim is open: binding any address
claims it for the host (on
sim0), so tests that bind whatever they like keep working. Once one does, the code under test binds only the host's addresses (127/8 always), broadcast and multicast addresses, and the interfaces'stations; anything else isEADDRNOTAVAIL/WSAEADDRNOTAVAIL. - Who is where. An address is the host's, a station's (a tester endpoint or listener bound there,
or a
NicSpec::stationsaddress an in-process socket binds — on the segment, across the link, never reached through a host wildcard bind), or nobody's. - Routing. Every send and connect resolves a path: a bound device first (Linux
SO_BINDTODEVICE/SO_BINDTOIFINDEXuse the device even with no route; macOSIP_BOUND_IF/IPV6_BOUND_IFuse scoped routes only,ENETUNREACHon a miss,ENETDOWNwhen it is down; WindowsIP_UNICAST_IF/IPV6_UNICAST_IFfor unicast), thenIP_MULTICAST_IFor the source's interface for multicast, local delivery through loopback for the host's own addresses, then the longest prefix through a live interface (ties: the bound address's interface, the lowest metric, the earliest route). Each interface address brings its connected subnet. No route isENETUNREACH(macOSsendtofrom a specific address:EHOSTUNREACH; WindowsWSAENETUNREACH) and logsFault::Unreachable. A loopback source leaving the host isEINVALon Linux andEADDRNOTAVAILon macOS; Windows is a strong host, so a bound address only leaves through its own interface (WSAENETUNREACH, as measured) and only receives there. - Sources and delivery. A wildcard or unbound UDP socket keeps its wildcard local address and each datagram carries the route's source; a TCP client's local address is the route's source. A datagram reaches the exact bind, else a wildcard bind only for the host's own addresses; a directed broadcast reaches its subnet's stations and the host's wildcard binds; a socket bound to a device only receives what arrives on it. Recipients are served in (port, address) order.
- Multicast. A group's datagram reaches, on its port, every socket that joined the group. On
Linux a socket also receives the groups any socket of the host joined while
IP_MULTICAST_ALL(IPV6_MULTICAST_ALL) is on, the default; turning it off leaves only its own joins. macOS and Windows deliver joined groups only. On Linux and macOS the socket must be bound to the wildcard or the group itself.tests/multicast_os_truth.rscompares this with the real host. Which interface a join named is not matched against the arriving one,IP_DROP_MEMBERSHIPis not modelled, and one exact address takes one socket (SO_REUSEADDR/SO_REUSEPORTsharing is not modelled). - Bind to device from the test.
set_socket_device(id, Some(nic))(orNone; alsoSim::set_socket_device) binds an open socket as the host's option would, with that option's error for an unknown name (ENODEV,ENXIO,WSAEINVAL) and no privilege check. - Link state.
set_link,set_nicandschedule_link(a deadline on the sim's clock, so a thread blocked on the link wakes when it changes, under every clock anddeterministic()). With carrier down, Linux and macOS keep routes: sends succeed and the frame is dropped (tx_dropped,tx_carrier_errors,Fault::LinkDown), inbound is lost, and a datagram in flight across a flap is discarded. Windows media sense withdraws the interface's routes and addresses. Admin down withdraws routes (Linux keeps the addresses bindable, as measured; macOS answersENETDOWNto sockets bound there). TCP over a dead link stalls: writes succeed, bytes wait in flight and arrive in order once the link returns (or the interface is removed); a blocked read with nothing scheduled gives up like any quiescent wait. Raw L2 frames do not cross a dead link, an oversized frame isEMSGSIZE, and LinuxAF_PACKETon a down interface isENETDOWN. NicPolicy. Latency, jitter, loss and duplication on traffic crossing the interface (host to station, station to station), never host-local traffic, drawn from the sim's seed in a fixed order (loss, duplication, jitter, then the address'sUdpPolicy); delays add and losses compound. TCP takes the latency and jitter per write, in order, without loss. With no policy set, draws are exactly as before.- Counters.
nic_counters(name)counts frames with their headers (UDP +42 / +62 bytes, TCP segments of at most MTU-40 bytes +54 / +74); loopback traffic counts on loopback;set_nic_countersedits them.
add_nic, remove_nic, set_nic, set_link, schedule_link, set_nic_policy,
set_nic_counters, nic, nics, nic_counters, add_route, remove_route,
set_default_route, routes and route_lookup work from any thread of a sim, and as Sim
methods from anywhere. tests/nic_os_truth.rs compares the error codes against the real host.
Enumerating interfaces
Every way the code under test can list interfaces reads the same topology, so set_link,
set_nic and the counters show up everywhere at once:
- libc (Linux, macOS).
if_nametoindex,if_indextoname,if_nameindex/if_freenameindex, andgetifaddrs/freeifaddrs: oneAF_PACKET(Linux,ifa_dataartnl_link_stats) orAF_LINK(macOS, anif_data) entry per interface, then oneAF_INET/AF_INET6entry per host address with netmask and broadcast, in index order (glibc lists the link entries first). Stations are not listed. Flags follow the link:IFF_UPwhile admin up,IFF_RUNNING(and LinuxIFF_LOWER_UP) with carrier. Unknown names fail as the host's libc does (ENODEVfrom glibc,ENXIOon macOS; pinned bytests/nic_enumeration.rsagainst the real call). - ioctls on any socket. Linux
SIOCGIFINDEX,SIOCGIFMTU,SIOCGIFFLAGS,SIOCGIFHWADDR; macOSSIOCGIFFLAGS,SIOCGIFMTU. An interface without a configured MAC has02:00:<index>. SimHost./sys/class/net/<if>/{mtu,ifindex,operstate,carrier,flags,address,statistics/*}(readingcarrierof an admin-down interface isEINVAL, ascarrier_show), rtnetlinkRTM_GETLINK(flags,IFLA_OPERSTATE,IFLA_CARRIER,IFLA_MTU,IFLA_ADDRESS,IFLA_STATS64),RTM_GETADDRandRTM_GETROUTEdumps, and macOSsysctlNET_RT_IFLIST2all render the topology, loopback included.ethtool, hardware timestamping, queues, IRQs and PTP stay the host profile's (see NIC drivers).- Windows IP Helper.
if_nametoindex/if_indextoname(NDIS names such asethernet_32772,loopback_0), theConvertInterface*alias/name/LUID/index conversions,GetIfEntry2,GetIfTable2,GetUnicastIpAddressTable,FreeMibTable,GetAdaptersAddresses(with theERROR_BUFFER_OVERFLOWsize handshake) andGetBestRoute2, rendered through thewindows-systypes.OperStatus/MediaConnectStatefollow the link, octet and packet counters follow traffic, and addresses are listed only with carrier (media sense). Tables the sim hands out are freed byFreeMibTable; any other pointer goes to the real one.
NIC drivers: ethtool, qdiscs, threaded NAPI
On Linux a SimHost interface has a driver, declared on its Nic the way old snare's NicCaps
and DriverSeed did, and SIOCETHTOOL runs against it as net/ethtool/ioctl.c __dev_ethtool
does (Linux 6.12): a command outside the kernel's read-only list needs CAP_NET_ADMIN (EPERM,
checked first), an operation the driver lacks is EOPNOTSUPP, and the core's own checks run
before the driver's. What the code under test sets persists; SimHost::ethtool(nic) returns it
all as a NicEthtool.
| Command | Nic builder |
Rules |
|---|---|---|
GDRVINFO |
driver, bus_info, firmware, expansion_rom, register_dump_len, eeprom_len, private_flags_count |
dump lengths and private-flag count default to zero; n_stats from driver statistics |
GLINK |
— | admin up with carrier, from the topology |
[GS]RINGPARAM |
rings(Rings { .. }) |
each size within its maximum (EINVAL); the *_max given are ignored |
[GS]COALESCE |
coalesce(CoalesceParams, Coalesce), coalesce_limits |
a non-zero field outside supported_coalesce_params is EOPNOTSUPP; above the driver's limits EINVAL |
[GS]CHANNELS |
channels(Channels { .. }) |
no change succeeds at once; counts within maxima, at least one RX and TX queue, not below a queue a flow rule uses (EINVAL); the queue count (/sys/class/net/<if>/queues) follows |
[GS]PAUSEPARAM, [GS]EEE |
pause, eee |
an EEE mode outside supported is EINVAL |
[GS]FLAGS |
ntuple(slots) makes ETH_FLAG_NTUPLE settable |
unknown flags EINVAL; a change outside hw_features EOPNOTSUPP |
GRXCLSRLCNT/GRXCLSRULE/GRXCLSRLALL/SRXCLSRLINS/SRXCLSRLDEL |
ntuple(slots) |
as igb: a slot past the table, a missing queue or an empty slot EINVAL, a short rule_locs EMSGSIZE |
GSSET_INFO/GSTRINGS/GSTATS |
driver_stats([(name, value)]), SimHost::set_driver_stat |
none declared: EOPNOTSUPP, as a driver without statistics |
rtnetlink RTM_NEWQDISC/RTM_DELQDISC change the qdiscs RTM_GETQDISC dumps (every interface, in
index order). Multiqueue goes by the transmit queues the driver allocated (num_tx_queues:
Nic::tx_queues_allocated, else the channel maxima), not those in use (real_num_tx_queues, the
queues/ listing): a multi-queue interface starts with mq over one default qdisc per allocated
queue (pfifo_fast past the queues in use) and the dump lists the queues in use and any qdisc
created with a handle, as net/sched/sch_mq.c mq_attach hashes them; a single-queue one starts with
the default qdisc, and an IFF_NO_QUEUE one (Nic::no_queue, veth) with noqueue. Replacing the
root or an mq class's qdisc follows net/sched/sch_api.c tc_modify_qdisc
(NLM_F_CREATE/NLM_F_REPLACE/NLM_F_EXCL, automatic handles from 8001:), mq needs more than
one allocated queue (EOPNOTSUPP), and etf checks its parameters as etf_init does
(CLOCK_TAI only, a dynamic clock ENOTSUPP (524)); TC_ETF_OFFLOAD_ON needs Nic::etf_offload(queues)
(EOPNOTSUPP without, EINVAL on another queue, as igb on an I210). Changes need CAP_NET_ADMIN and
are acked with NLMSG_ERROR when NLM_F_ACK asks. /sys/class/net/<if>/threaded reads and, for
root, writes the threaded-NAPI switch (Nic::threaded_napi): EPERM without CAP_NET_ADMIN,
EOPNOTSUPP for a value other than 0 or 1. EasyBuilder presets ship an I210-like driver with all
of the above. Tests: tests/simhost_ethtool.rs.
Windows adapter properties and restarts
Windows tunes an adapter through its device node, not an ioctl: a program enumerates the network
class with SetupDiGetClassDevsW/SetupDiEnumDeviceInfo, opens the driver key with
SetupDiOpenDevRegKey(DIREG_DRV), matches its NetCfgInstanceId to the interface GUID IP Helper
reports, writes standardized keywords (*ReceiveBuffers, *InterruptModeration, *FlowControl,
*NumRssQueues, *RSS, *SoftwareTimestamp, EEE names) as REG_SZ values, reads their limits
from Ndi\Params\<keyword>\max, and restarts the device with CM_Disable_DevNode +
CM_Enable_DevNode (fast-talker's nic/windows.rs does exactly this). Every one of those calls is
in-process, so the Windows Sim serves them: each non-loopback interface is a device node whose
two keys (DIREG_DRV, and DIREG_DEV's Device Parameters, where the interrupt affinity policy
lives) are an in-memory registry that keeps what the code under test writes.
- Describe it.
SimBuilder::adapter(nic, Adapter::new().property("*ReceiveBuffers", "512", Some(4096)).without_property("*RSS").driver_version("..").restart_flap(d)), orSim::set_adapterlater. The defaultAdapter::new()mirrors snare 1.x'sNicCaps: rings 256 of 4096, interrupt moderation on, flow control Rx & Tx, 4 of 8 RSS queues, RSS on, software timestamping off, no EEE keyword, 2 s flap. - Read it back.
Sim::adapter_property(nic, keyword),Sim::adapter_value(nic, AdapterKey::{Driver, Device}, path, name) -> Option<RegValue>,Sim::adapter_restarts(nic). - Restart flap.
CM_Disable_DevNodedrops the interface's carrier;CM_Enable_DevNodeschedules it back uprestart_flaplater on the sim's clock (schedule_link), soGetIfEntry2'sOperStatus, media sense and the sockets all see the outage, under the virtual clock anddeterministic(). An interface that had no carrier stays down. - Elevation is
Privileges::root. A standard user may read the keys but getsERROR_ACCESS_DENIEDopening one for writing or creating a subkey, andCR_ACCESS_DENIEDfrom theCM_*calls. A write through a handle openedKEY_READisERROR_ACCESS_DENIEDfor anyone. - Codes. Missing value or key
ERROR_FILE_NOT_FOUND, short bufferERROR_MORE_DATAwith the size, a buffer without a lengthERROR_INVALID_PARAMETER, enumerating past the endERROR_NO_MORE_ITEMS, a wrongcbSizeERROR_INVALID_USER_BUFFER, a bad key typeERROR_INVALID_FLAGS;tests/nic_adapter_props_win.rs(registry_codes_os_truth,setupapi_codes_os_truth) compares them with the real registry and SetupAPI. - Passthrough. Only the sets, keys and device instances the sim minted (and the network-class
enumeration that mints them) are simulated; every other registry key, device class or
DEVINSTgoes to the real API. A sim handle passed to a SetupAPI call that is not hooked reaches the real one.CreateFileW(\\.\{GUID})for NDIS statistics is not modelled (it fails, as for a virtual adapter), so driver counters come fromGetIfEntry2alone.
The WinHost also keeps the process's working set: SetProcessWorkingSetSizeEx /
GetProcessWorkingSetSizeEx (fast-talker's reserve_working_set) never touch the real process.
Measured on Windows 11: a new process has 204 800 / 1 413 120 bytes with both limits soft; bounds
round down to 4 KiB and the minimum is raised to 20 pages; min above max, or ENABLE and
DISABLE of one limit together, is ERROR_INVALID_PARAMETER; a minimum beyond the machine is
ERROR_NO_SYSTEM_RESOURCES (the sim has 16 GiB and refuses above 7/8 of it); flags 0 keep the
limits' state; both bounds (SIZE_T)-1 trims and succeeds. Raising either bound needs
SeIncreaseWorkingSetPrivilege, whose stand-in is ipc_lock (ERROR_PRIVILEGE_NOT_HELD
without it). Sim::working_set() reads it back; working_set_os_truth compares with the real OS.
Socket faults and back-pressure
Faults on the code under test's sockets, with the host OS's codes and the order its kernel reports
them in. Every call works from any thread of a sim; the Sim methods of the same names also work
outside run.
| Call | Effect |
|---|---|
raise_socket_error(addr, io::Error) |
Makes the error pending on every stream whose local or peer address is addr and every datagram socket bound at or connected to it. A raw OS error passes as given; a kind maps to the host's code (ConnectionReset ECONNRESET/WSAECONNRESET, BrokenPipe EPIPE/WSAESHUTDOWN, ...); a kind with none panics. |
inject_icmp_port_unreachable(to, from) |
An ICMP port unreachable from from reaches the datagram sockets bound at to. |
quiesce(addr, span, Direction) |
Holds what arrives at (Receive), leaves (Send) or crosses (Both) addr for span of sim time: TCP bytes in flight and written meanwhile, and a FIN, land in order once it ends (counting against the window); bytes already received stay readable; datagrams arrive at its end. |
SO_SNDBUF / SO_RCVBUF |
Bound every stream of the code under test: what is in flight plus unread is capped at the reader's receive room plus the writer's send buffer (see Socket limits and privileges). |
TcpPolicy::recv_window |
The receive window of the endpoint at the policy's address, capping its receive room below its SO_RCVBUF (a tester's endpoint, unlimited otherwise, takes the window alone). |
SO_LINGER |
A zero timeout makes the last close abort with a reset (the peer reads ECONNRESET, a tester logs PeerReset); another makes it wait, up to that long, until what was written has crossed. macOS also takes SO_LINGER_SEC (std and socket2 use it; its SO_LINGER counts 1/100 s ticks); Windows SO_DONTLINGER, and a nonblocking close with data still crossing fails with WSAEWOULDBLOCK. |
TesterAction::RaiseSocketError(e) / IcmpPortUnreachable / QuiesceLink(span, dir) / SetRecvWindow(w) |
The same for the code under test's end the event came from (every peer from a cyclic action); QuiesceLink's direction is seen from the code under test. |
| OS | Raised error reported | ICMP port unreachable | Writable on a bounded stream |
|---|---|---|---|
| Linux | first by a send; a stream's receive returns queued bytes first, a datagram socket's the error first | ECONNREFUSED to the next send or receive of a socket connected to the sender; an unconnected one only with IP_RECVERR, which also queues a sock_extended_err (origin ICMP, type 3, code 3, the offender's address) for recvmsg(MSG_ERRQUEUE) |
free space at least half of what is queued |
| macOS | first by a send; a receive returns queued bytes first (MSG_PEEK leaves the error pending) |
ECONNREFUSED to a socket connected to the sender; an unconnected one ignores it |
2048 bytes free (SO_SNDLOWAT); EVFILT_WRITE reports the free space |
| Windows | first by any call | WSAECONNRESET in receive arrival order alongside queued datagrams, for a connected or unconnected socket unless WSAIoctl(SIO_UDP_CONNRESET, FALSE); MSG_PEEK preserves the status; SO_ERROR does not return or consume this status |
any free space |
Reporting clears the error, as getsockopt(SO_ERROR) does; while it is pending a blocked call
returns at once, epoll and poll report EPOLLERR/POLLERR (macOS poll and kqueue: readable),
WSAPoll POLLERR and select the read set. Windows ICMP receive status instead reports
WSAPoll readability and is consumed by a receive without MSG_PEEK, leaving SO_ERROR zero. A datagram sent to an address the host or a station
holds that reaches no socket or tester draws an ICMP error one round trip later (the policies'
fixed latency each way, no jitter, so seeds replay); broadcast, multicast and absent addresses draw
none. A full window makes a blocking send wait for room, up to SO_SNDTIMEO (then the bytes sent
so far, else EAGAIN/WSAETIMEDOUT), and a nonblocking one return what fit or
EAGAIN/WSAEWOULDBLOCK: on Linux what fits, on macOS nothing until 2048 bytes (SO_SNDLOWAT)
are free unless the whole write fits, on Windows the whole write while anything is free. A
tester's writes queue until the window takes them, and its close
waits for them.
Connect faults
A connect asks the destination how it answers a SYN, as the host OS would:
- a listener there (the code under test's or a tester's) accepts at once;
- an address of the host or a station with no listener resets the SYN:
ECONNREFUSEDat once on unix; Windows retries on a reset and reportsWSAECONNREFUSED2 s later; - an address nobody holds stays silent, and the connect follows the OS's SYN plan — Linux
retransmits at 1, 2, 3, 4, 5, 7, 11, … s and gives up at 131 s with the modern default
SysLimits::tcp_syn_linear_timeoutsof 4. Setting it to 0 selects the older exponential schedule and 127 s timeout.SysLimits::tcp_syn_retriesor the socket'sTCP_SYNCNTsets the retry limit; both sysctls are exposed bySimHostunder/proc/sys/net/ipv4.SysLimits::from_real_hostreads the running kernel's settings. macOS retransmits at 1, 2, 3, 4, 5, 7, 11, … s and gives up at 75 s (TCP_CONNECTIONTIMEOUT); Windows retransmits at 3 and 9 s and gives up at 21 s by default. ExtendingTCP_MAXRTcontinues with exponentially increasing intervals capped at 60 s; setting it to -1 keeps retrying until the connection completes or closes, each withETIMEDOUT/WSAETIMEDOUT. On Linux an absent address on a connected subnet fails for want of a neighbour instead, withEHOSTUNREACHafter 3 s; macOS (measured) and Windows follow their SYN plans there too; - a listener that appears meanwhile is reached at the next retransmission.
While an open sim (no interface given an address) still lets the code under test bind any address,
only addresses it has bound, loopback and stations answer; a connect to any other address times
out — at once in virtual time, but for real under wall_clock().
use ;
use ;
use ;
new.run;
| Call | Effect |
|---|---|
set_listener_behavior(addr, b) / Sim::set_listener_behavior |
How addr answers from then on, kept per address (the wildcard covers its port), before or after anything listens there and across listener replacement. DelayingUntil reads its Instant on the calling thread's clock; the Sim method, called off the sim, takes it as that far ahead of the sim's clock. |
TesterAction::SetListenerBehavior(b) |
The same for the tester's own address. |
A nonblocking connect that cannot finish at once fails with EINPROGRESS (WSAEWOULDBLOCK); a
second connect meanwhile gives EALREADY and one after it succeeded EISCONN. The socket reports
nothing to poll/epoll/kqueue/select until it is settled: writable once connected, or
POLLOUT | POLLERR | POLLHUP (EPOLLERR, EVFILT_WRITE with EV_EOF and the error in fflags,
Windows exceptfds and WSAPoll POLLERR | POLLHUP) once it failed, the error waiting in
SO_ERROR. Each point of the plan is a timer the sim can skip to, dropped when the connect settles
or the socket closes.
Socket table
Every socket the code under test opens — fabric TCP and UDP, raw L2, Winsock, a SimHost's UDP and
netlink sockets — gets one record for its whole life. Supported duplicate operations share that
record; accept makes a new one. Unix socket aliases created by dup, F_DUPFD,
F_DUPFD_CLOEXEC, dup2, Linux dup3 or try_clone share socket state; close-on-exec flags
belong to each descriptor. Replacement updates ownership across socket and virtual-file backends.
Linux eventfd/epoll aliases also share their counter and interest state; closing the last alias
retires the description. Remaining descriptor boundaries are listed in open bugs.
snare::socket_id(&sock)
names any std or mio socket of the calling thread's sim (None for a real OS socket), and
socket_entry(id), socket_table() (open, oldest first), closed_sockets() (in close order) and
sockets_bound(addr) read snapshots, as do the Sim methods of the same names from any thread.
| Field | Meaning |
|---|---|
id |
Numbered from 1 per sim in creation order and never reused. Ids are per sim: two sims both start at 1. |
kind |
Udp, TcpStream (unconnected, connected or accepted), TcpListener, Packet, Netlink, Unix (a socketpair end). |
local / peer / listener |
Where it is bound and connected, and for an accepted stream its listener's id. |
memberships |
Multicast groups joined, with the interface address or index the join named. |
queued / queued_bytes |
Datagrams (UDP) or written chunks (TCP) that have arrived and wait to be read. |
delivered / delivered_bytes / sent |
What arrived at it, and its successful sends. |
pending_error |
The pending socket error or receive status. SO_ERROR returns and clears socket errors; Windows ICMP receive status is returned only by a receive. |
created_at / closed_at |
On the sim's timeline, like recorded events. |
interface / bound_device / multicast_if |
The interface its latest traffic crossed, the one it is bound to, and the one its multicast leaves through. |
last_tx_nic / last_rx_nic |
The interface its latest datagram or connection left by and arrived on (a connection notes both when it is made). |
unmodelled_options |
Each setsockopt/getsockopt/ioctl the sim does not model that was used on it, once, in first-use order. See Unmodelled options. |
Per-socket options (SO_RCVTIMEO, SO_SNDTIMEO, SO_LINGER, SO_REUSEADDR) live in the record, so
every descriptor of a socket sees the same values. Where the hosts differ, the sim follows the one it
runs on, as measured: a blocking accept honours SO_RCVTIMEO on Linux only; after SHUT_RD Linux
still delivers what arrives while macOS reads end of stream at once and Windows fails reads with
WSAESHUTDOWN; and a bind overlapping a wildcard or equal address follows each OS's SO_REUSEADDR
rule (tests/bind_os_truth.rs compares the sim against the host).
On macOS, TCP read shutdown discards queued receive bytes. Subsequent incoming data closes
that endpoint at its arrival time and returns a reset after the reverse link delay. The local
endpoint writes with EPIPE; the peer retains previously received bytes, then reports
ECONNRESET once before EOF. MSG_PEEK preserves that error, SO_ERROR consumes it, and
kqueue EOF events carry the pending error in fflags. The native comparisons and exact
link-delay tests are in tests/macos_tcp_read_shutdown*.rs; Unix socket pairs do not send
TCP resets.
The data/reset exchange samples both link delays when it is scheduled. Directed stalls and carrier outages can delay either delivery, including when the connection is inspected later.
Packet timestamps
A socket that asks for kernel timestamps gets them from the sim's clock, on the plain Sim's
fabric and under a SimHost alike, through one implementation (tstamp.rs) that keeps each
socket's timestamping state in its socket record. A packet is stamped at the instant it reaches
the receiving socket — its send time plus the link delay it crossed — so code that reads it late
still sees when it arrived, as with a real kernel's software receive stamp. Every stamp reports
CLOCK_REALTIME except where the option says otherwise; a clock that jumps is moved up to a stamp it
hands out, so no later clock read is earlier. What each host does was measured against it
(tests/timestamps.rs, the *_os_truth tests); Windows follows
Microsoft Learn: Winsock timestamping
(tests/timestamps_win.rs).
| Host | Options | What the code under test reads |
|---|---|---|
| Linux | SO_TIMESTAMP, SO_TIMESTAMPNS, SO_TIMESTAMPING (flags or struct so_timestamping) |
recvmsg on UDP and TCP: SCM_TIMESTAMP (timeval) or SCM_TIMESTAMPNS (timespec), then SCM_TIMESTAMPING with the software stamp in ts[0] when SOF_TIMESTAMPING_SOFTWARE reports it (OPT_RX_FILTER honoured); a TCP read gets the arrival of the last byte it took. Transmit stamps on MSG_ERRQUEUE: TX_SCHED/TX_SOFTWARE at the send time, TCP TX_ACK a round trip later, each with IP_RECVERR/IPV6_RECVERR sock_extended_err (ENOMSG, SO_EE_ORIGIN_TIMESTAMPING, ee_info the stage, ee_data the OPT_ID key, counted per datagram or by byte for TCP), the looped-back packet (a fabricated Ethernet/IP/UDP or TCP header and the payload) or nothing under OPT_TSONLY, and POLLERR while one waits. SO_TIMESTAMPING validates as the kernel does (unknown flags, OPT_ID_TCP without OPT_ID, OPT_STATS without OPT_TSONLY, OPT_ID on an unconnected TCP socket: EINVAL; BIND_PHC: EOPNOTSUPP/EINVAL, the interfaces having no PHC). SIOCGHWTSTAMP/SIOCSHWTSTAMP on the plain fabric's interfaces are EOPNOTSUPP (EPERM to set without CAP_NET_ADMIN, ENODEV for an unknown name). SO_TXTIME validates as sk_setsockopt does (exactly a struct sock_txtime, known flags and clocks: EINVAL; a clock other than CLOCK_MONOTONIC without CAP_NET_ADMIN: EPERM) and reads back all 8 bytes. |
| macOS | SO_TIMESTAMP, SO_TIMESTAMP_MONOTONIC, SO_TIMESTAMP_CONTINUOUS |
recvmsg on UDP: SCM_TIMESTAMP (timeval), SCM_TIMESTAMP_MONOTONIC and SCM_TIMESTAMP_CONTINUOUS (Mach ticks, on the clock mach_absolute_time reads). Nothing on TCP, as XNU. getsockopt reports the option's bit. |
| Windows | WSAIoctl(SIO_TIMESTAMPING) with TIMESTAMPING_FLAG_RX/TX (UDP only; WSAEINVAL on TCP) |
WSARecvMsg on UDP: one SO_TIMESTAMP control message, the arrival as a QueryPerformanceCounter value on the counter the code under test reads. A WSASendMsg carrying SO_TIMESTAMP_ID buffers its send time while TxTimestampsBuffered has room (a stamp generated with the buffer full is dropped); SIO_GET_TX_TIMESTAMP takes it by id, WSAEWOULDBLOCK when none waits, and WSAEOPNOTSUPP while transmit stamps are off (measured). |
On Linux, Sim::builder().rx_timestamp_startup_delay(Duration::from_millis(5)) starts
software receive timestamp generation cold. The first successful SO_TIMESTAMP,
SO_TIMESTAMPNS or SO_TIMESTAMPING request for RX_SOFTWARE starts a single sim-wide
activation window on the sim's clock. Packets arriving before its end have no software receive
stamp, even if read after activation. Later option changes and other sockets do not restart
the window; hardware stamps and transmit reports are independent. The default delay is zero,
representing generation already active, so existing tests retain immediate timestamps.
SO_TIMESTAMP and SO_TIMESTAMPNS on a datagram supply a read-time fallback for an unstamped
packet, including the same fallback across repeated MSG_PEEK reads; TCP has no such fallback.
This configures an initial cold-to-warm profile; it does not replay Linux's host-wide static-key
reference counts or workqueue scheduling.
A Linux SimHost adds hardware stamps (tests/simhost_hwtstamp.rs). A NIC with
Nic::hardware_timestamping starts with the struct hwtstamp_config of Nic::hwtstamp_config
(off by default), and SIOCSHWTSTAMP changes the emulated configuration: EPERM without
CAP_NET_ADMIN, an unknown flag EINVAL, an unknown
transmit type or filter ERANGE, a NIC without hardware timestamping EOPNOTSUPP (and
SIOCGHWTSTAMP too); then a transmit type the driver's ETHTOOL_GET_TS_INFO lacks is ERANGE and
a filter it lacks is widened to the narrowest advertised filter that covers it, with the applied
configuration written back. This is a synthetic policy: capability bitmaps do not specify a
driver's requested-to-applied mappings. The I210 default turns PTP filters into
HWTSTAMP_FILTER_ALL. With
transmit stamping on, a send asking TX_HARDWARE gets a SCM_TSTAMP_SND report carrying only
ts[2] (the NIC's PHC at the send), and its software report is dropped unless
SOF_TIMESTAMPING_OPT_TX_SWHW asks for both (net/core/skbuff.c __skb_tstamp_tx); a datagram the
NIC's filter covers arrives with ts[2] for a socket reporting RAW_HARDWARE. UDP PTP filters
check a payload of at least 34 bytes and selected version/message fields on port 319; NTP filtering
selects destination port 123. These checks are a synthetic classifier, without complete packet or
declared-length validation. Hardware/software timestamp points have no independent NIC latency.
Hardware TX reports with legacy SO_TIMESTAMP(NS) enabled receive a software read-time fallback,
also visible in ts[0] when software reporting is enabled.
SCM_TXTIME on a SimHost socket without SO_TXTIME, or
not 8 bytes long, fails the sendmsg with EINVAL (__sock_cmsg_send). A /dev/ptp<N> answers
ptp_ioctl (drivers/ptp/ptp_chardev.c) from its PtpCaps (HostProfile::ptp_clock_caps; by
default nothing to adjust and no pins, channels or cross-timestamping): PTP_CLOCK_GETCAPS,
PTP_SYS_OFFSET (EINVAL past PTP_MAX_SAMPLES), PTP_SYS_OFFSET_EXTENDED (system clock, PHC,
system clock per sample, on CLOCK_REALTIME, CLOCK_MONOTONIC or CLOCK_MONOTONIC_RAW),
PTP_SYS_OFFSET_PRECISE only with cross_timestamping (else EOPNOTSUPP), and EACCES for the
changing requests on a read-only fd (tests/simhost_ptp.rs).
A control buffer too short for the messages is cut as the host cuts it: Linux writes the part that
fits with cmsg_len cut to it and sets MSG_CTRUNC (even with no control buffer); macOS copies the
bytes that fit, sets MSG_CTRUNC only for a message it had to cut, and drops a message that finds the
buffer full. The kernel's deferred enabling of
receive stamping (a static key that can leave the first packets after setsockopt unstamped) is not
modelled: every packet is stamped.
Unmodelled options and strict_sockopts
Every setsockopt, getsockopt and ioctl on a sim socket is modelled, ignored on purpose as
harmless, or unmodelled. An unmodelled one succeeds with no effect (a getsockopt reads an int 0),
as before, but is listed in the socket's SocketEntry::unmodelled_options and logged once per socket
as RecordedEvent::UnmodelledOption, so a test can assert the code under test used nothing the sim
ignores. With SimBuilder::strict_sockopts() it fails instead, with what the host answers for an
option it does not know — ENOPROTOOPT for an option, ENOTTY (Linux) or ENXIO (macOS) for an
ioctl request, all measured — and the first refusal is also written to stderr.
On Windows every option Winsock defines is known by level and number. An unmodelled one is kept and
read back as well (with Windows' default until set, such as IP_MULTICAST_LOOP on and IPV6_V6ONLY
on), listed, and refused under strict_sockopts() with WSAEINVAL at SOL_SOCKET and
WSAENOPROTOOPT elsewhere, the codes Winsock gives an option it does not know (measured). An option
Winsock does not define fails with those codes with or without strict mode, as does an
ioctlsocket command other than FIONBIO, FIONREAD and SIOCATMARK (WSAEOPNOTSUPP, measured).
A WSAIoctl code the sim does not carry out cannot succeed without effect, since only the OS could
fill its output: it fails with WSAEOPNOTSUPP and is listed as refused. SO_SNDLOWAT,
SO_RCVLOWAT and SO_USELOOPBACK fail WSAEINVAL, as Microsoft documents. A level the socket does
not take fails first: IPPROTO_IPV6 on an IPv4 socket (WSAEINVAL), IPPROTO_TCP on a datagram
socket and IPPROTO_UDP on a stream (WSAENOPROTOOPT); SO_KEEPALIVE is stream-only.
The harmless options are kept and read back but change nothing, because nothing a test can observe in
the sim depends on them: SO_KEEPALIVE and the TCP keepalive timers (no connection dies idle),
TCP_NODELAY and Linux TCP_QUICKACK (no Nagle or delayed ACKs), SO_OOBINLINE and SO_DEBUG, macOS
SO_NOSIGPIPE (the sim never raises SIGPIPE; std sets it on every socket), and IP_TOS, IP_TTL,
IP_MULTICAST_TTL, IPV6_TCLASS, IPV6_UNICAST_HOPS, IPV6_MULTICAST_HOPS (no QoS or hop count);
FIOCLEX/FIONCLEX for ioctls. Windows keeps the same, read back from Windows' defaults until set
(IP_TTL and IPV6_UNICAST_HOPS 128, the multicast hop limits 1, the TCP keepalive timers 2 hours,
1 s and 10 probes), plus SO_DEBUG and SO_DONTROUTE, which Microsoft's providers ignore, and the
SIO_KEEPALIVE_VALS, SIO_LOOPBACK_FAST_PATH and SIO_UDP_NETRESET ioctls. Everything else the sim does not model — SO_REUSEPORT,
IP_MULTICAST_LOOP, IP_DROP_MEMBERSHIP, IPV6_V6ONLY, SO_RCVLOWAT, SIOCGSTAMP and the
rest — is unmodelled. A SimHost socket keeps its
existing read-back of every option set, and records the same way. std's own socket calls on each host
are all modelled or harmless (tests/strict_sockopts.rs runs them under strict_sockopts).
Don't-fragment
The don't-fragment options are modelled on every IP socket of the fabric and a SimHost: Linux
IP_MTU_DISCOVER and IPV6_MTU_DISCOVER (IP_PMTUDISC_DONT, WANT — the default — DO, PROBE,
INTERFACE, OMIT; anything else EINVAL) and IPV6_DONTFRAG, macOS IP_DONTFRAG and
IPV6_DONTFRAG (flags, read back as 0 or 1). They read back as the host does, including the optlen
rules (Linux reads a short IP_MTU_DISCOVER as a byte, refuses a short IPV6_MTU_DISCOVER and
reads a short IPV6_DONTFRAG as 0; macOS refuses any short value) and the other family's errors
(Linux ENOPROTOOPT to set and EOPNOTSUPP to read an IPv6 option on an IPv4 socket, macOS
EINVAL either way). The sim never fragments: a datagram larger than its egress interface's MTU is
delivered whole, unless it may not be fragmented — Linux DO, PROBE or INTERFACE for its
family, or IPV6_DONTFRAG for IPv6; macOS IP_DONTFRAG, or IPV6_DONTFRAG on an IPv6 socket,
v4-mapped sends included — when the send fails EMSGSIZE once payload plus 8 bytes of UDP and 20
(IPv4) or 40 (IPv6) bytes of IP header pass the MTU. The MTU is the interface's configured one; no
smaller path MTU is learned. tests/dontfrag.rs dontfrag_os_truth compares all of it with the real
stack on loopback (16384 on macOS lo0, 65536 on Linux lo, which only IPv6 datagrams can exceed).
Windows has IP_DONTFRAGMENT and IPV6_DONTFRAG (flags) and IP_MTU_DISCOVER/IPV6_MTU_DISCOVER
(a PMTUD_STATE: NOT_SET, the default, DO, DONT or PROBE; anything else WSAEINVAL). A
datagram may not be fragmented under the flag or DO/PROBE for its family, and then fails with
WSAEMSGSIZE past the egress MTU, by the same header arithmetic (tests/dontfrag_win.rs).
Socket limits and privileges
A datagram socket's receive buffer is enforced as the host's kernel does: each datagram is admitted
or dropped at the moment it arrives — not when the code under test next looks — against what the
buffer holds then. SocketEntry reports rcvbuf/sndbuf (what getsockopt returns), rmem_alloc,
overflowed (dropped for a full buffer), wire_lost (lost by a link policy, never counted as a
drop) and drops (the kernel's counter: overflows plus inject_socket_drops(id, n), which
SO_RXQ_OVFL and SO_MEMINFO report). Testers' endpoints are unlimited.
A TCP stream's buffers bound it too. What the code under test has written and its peer not yet
read is capped at the writer's send buffer (its SO_SNDBUF as getsockopt reports it) plus the
reader's receive room (its SO_RCVBUF as reported; twice it on Windows), capped further by a
TcpPolicy::recv_window; a stream to a tester is bounded only by a window. An accepted stream
takes its listener's sizes. Measured against the real OS with a reader that never reads
(tests/tcp_buffers_os_truth.rs), this matches Windows exactly and Linux within 40%; macOS
rounds buffers to whole segments and grows a loopback receive buffer by an amount that varies
from run to run, and Linux autotunes a send buffer whose SO_SNDBUF was never set (measured
2.6 MB in flight); neither is modelled, so the sim usually back-pressures sooner there.
On Linux, streams with an explicitly configured receive buffer or transmit timestamps use
MSS, GSO grouping and advertised-window state to release pending data as reads create room.
Transmit reports quote the final transmitted segment and wait for that segment to cross the
link. tests/tcp_error_queue_segments.rs compares IPv4 payloads and memory charges for writes
through 128000 bytes across five receive-buffer sizes on both backends. Ordinary TCP segments share the
receive-memory budget with error reports and keep their charge until fully read. Default buffer
autotuning, kernel receive coalescing, congestion-window evolution and ACK pacing remain
approximations; this comparison does not establish arbitrary TCP traffic parity.
| OS | Buffer options | Admission | FIONREAD |
|---|---|---|---|
| Linux | SO_RCVBUF v stores max(2 * min(v, rmem_max), 2304) (-1 asks for the cap), SO_SNDBUF likewise with floor 4608; SO_*BUFFORCE skip the cap and need CAP_NET_ADMIN |
into an empty queue, or while rmem_alloc + truesize <= rcvbuf (6.18 on); while rmem_alloc <= rcvbuf, overshooting by one datagram, before 6.18 |
the next datagram's payload (a stream's unread bytes); SIOCOUTQ 0 |
| macOS | v <= 0 is EINVAL; above kern.ipc.maxsockbuf clamps, or ENOBUFS once the buffer is already at it |
while the record fits sb_hiwat - sb_cc and sb_mbcnt is under 8 * sb_hiwat |
sb_cc (16 + payload for a socket's first datagram, 32 + payload after); SO_NREAD the next payload; a datagram over SO_SNDBUF is EMSGSIZE |
| Windows | stored as given | while the queued payload is below SO_RCVBUF |
the queued payload capped at SO_RCVBUF |
SysLimits holds the sysctls (rmem_default, rmem_max, wmem_*, tcp_*_default,
unprivileged_port_start, enforce_rcvbuf) and, on Linux, the kernel facts behind admission
(udp_rcvbuf_overshoot, skb_small_truesize, skb_head_overhead): SysLimits::host() is a stock
install of the build host (Linux: a stock 6.18+ kernel, whatever the architecture),
SysLimits::from_real_host() reads this machine, measuring the running kernel's truesize
geometry over loopback with SO_MEMINFO. A datagram longer than the smallest MTU on its path
arrives as IP fragments and is charged one buffer per fragment, as reassembly chains them
(net/ipv4/inet_fragment.c inet_frag_reasm_finish; measured through veth by
tests/hw_sockbuf_truth.rs). The Linux skb sizes depend on the kernel's build, not its
architecture: Debian's 6.12 and 7.2 kernels charge the same on x86_64 and arm64 (a lone small
datagram 960 bytes), OrbStack's arm64 kernel 1152. scripts/measure-sockbuf.sh boots Debian's
x86_64 and arm64 kernels under QEMU and measures them (KERNEL_SUITE, and TESTS to run snare's
test binaries on those kernels). Set them with SimBuilder::sys_limits,
set_sys_limits or Sim::set_sys_limits; a change applies to sockets created after it.
tcp_gso_max_size selects Linux's device GSO limit for TCP grouping (65536 bytes by default).
Driver-specific values need a measured profile override; from_real_host does not discover that
device setting.
Privileges { root, net_admin, net_raw, net_bind_service, sys_nice, ipc_lock, sys_resource, rtprio_limit, nice_limit, memlock_limit } is per sim: Privileges::all() by default (every
capability, RLIMIT_RTPRIO 99, RLIMIT_NICE 40, unlimited RLIMIT_MEMLOCK, as snare 1.x), a
SimHost's profile (HostProfile::cap, HostProfile::root, with the kernel's stock limits) when it
has one, and SimBuilder::privileges over both; Privileges::none() is an ordinary user with the
stock limits (0, 0, 8 MiB on Linux) and Privileges::from_real_process() copies the machine's;
set_privileges / Sim::set_privileges change it from any thread. The host's scheduling,
mlockall and SIOCSHWTSTAMP gates read the same model, a SimHost answers geteuid/getuid (0
for root, else 1000 on Linux, 501 on macOS) and renders CapEff/CapPrm/CapBnd in
/proc/self/status.
Process limits
The three limits are each an Rlimit { cur, max } (Rlimit::INFINITY is RLIM_INFINITY) and are
what getrlimit, setrlimit and prlimit (Linux, also SYS_prlimit64 and x86_64's
SYS_[gs]etrlimit through syscall) see and change inside the sim; the real process's limits are
never touched, and other resources pass through. Raising a hard limit needs sys_resource
(CAP_SYS_RESOURCE; root on macOS), a soft limit above the hard one is EINVAL. The calls they
gate follow the build host's kernel:
| OS | Rule |
|---|---|
| Linux | sched_setscheduler/sched_setparam/pthread_setschedparam (and the raw syscalls): EINVAL for a bad policy or priority, then without CAP_SYS_NICE EPERM to enter a real-time policy with RLIMIT_RTPRIO 0 or to raise the priority above both the current one and the limit (kernel/sched/syscalls.c). setpriority: the value is clamped to −20..19 and lowering it needs CAP_SYS_NICE or 20 − nice <= RLIMIT_NICE (EACCES). mlock: EPERM with RLIMIT_MEMLOCK 0, ENOMEM past it (locks do not nest); mlockall(MCL_CURRENT): ENOMEM when the process's mapped size exceeds it; CAP_IPC_LOCK lifts both (mm/mlock.c). |
| macOS | mlock wires pages counted against RLIMIT_MEMLOCK whatever the privileges (EAGAIN past it); wiring nests per page, so a page is released after as many munlocks. mlockall is ENOSYS, as on the real system. A successful pthread_setschedparam, any policy, opts the thread out of QoS for good: later pthread_set_qos_class_self_np calls fail with EPERM (<pthread/qos.h>; tests/macos_qos_os_truth.rs checks it against the host). |
A SimHost applies these to its own simulated threads and never reaches the real scheduler or lock
memory. A plain sim gates the same calls on its privileges and then lets an allowed one through to the
real kernel. tests/rlimits.rs checks the rules, and its rlimit_sequence_os_truth (Linux) and
memlock_os_truth (macOS) replay one sequence against the real kernel in a forked child with
Privileges::from_real_process(); run them under Docker with --ulimit rtprio=…, --ulimit nice=…, --ulimit memlock=… and --cap-add SYS_NICE/IPC_LOCK/SYS_RESOURCE to cover the other
branches.
| OS | Gated |
|---|---|
| Linux | binding a port below unprivileged_port_start (EACCES without CAP_NET_BIND_SERVICE), SO_RCVBUFFORCE/SO_SNDBUFFORCE (CAP_NET_ADMIN), SO_PRIORITY outside 0–6 and SO_MARK (CAP_NET_ADMIN or CAP_NET_RAW), changing the device of a bound socket (CAP_NET_RAW), socket(AF_PACKET) (CAP_NET_RAW); EPERM otherwise. |
| macOS | binding a port below 1024 on a specific address, and opening /dev/bpf*, need root (EACCES). |
| Windows | no socket call; without sys_nice a REALTIME_PRIORITY_CLASS request runs at HIGH_PRIORITY_CLASS. |
tests/os_parity.rs compares buffer options, flood admission counts and reserved-port binds against
the real OS, and tests/socket_limits.rs truesize_os_truth the charge of every datagram size
around each step.
Protocol counters
Every sim keeps the host's UDP and TCP counters, moved by the code under test's sockets (a tester is another machine) and readable as the code under test would read them:
| OS | Where | UDP | TCP |
|---|---|---|---|
| Linux | /proc/net/snmp, /proc/net/snmp6 (with or without a SimHost or VirtualFs; a snapshot at the first read, fresh after a seek to 0) |
InDatagrams (read), NoPorts, InErrors/RcvbufErrors (overflow), OutDatagrams, IgnoredMulti |
ActiveOpens, PassiveOpens, AttemptFails, EstabResets, CurrEstab, InSegs/OutSegs (data segments), OutRsts; RtoMin 200, RtoMax 120000, MaxConn -1 |
| macOS | sysctl/sysctlbyname net.inet.udp.stats (struct udpstat, 104 bytes) |
udps_ipackets (every arrival), udps_noport, udps_noportbcast, udps_fullsock, udps_opackets |
— |
| Windows | GetUdpStatistics(Ex/Ex2), GetTcpStatistics(Ex/Ex2) per family |
dwInDatagrams (arrivals), dwNoPorts, dwInErrors (overflow), dwOutDatagrams, dwNumAddrs |
as Linux, RtoAlgorithm 4, dwRtoMin 5 |
proto_counters() (and Sim::proto_counters) returns them per family for the test. A unicast
datagram to a port of the host nobody holds is a no-port; a broadcast, or a multicast one for a
group a socket of the host joined, that no socket took is ignored-multi; a group nobody joined
counts nowhere. The IP and ICMP lines count those datagrams and TCP data segments and the port
unreachables sent back; checksum, memory, send-buffer and retransmission counters stay 0.
tests/proto_counters_os_truth.rs compares the layouts and which counter a closed port and an
unread datagram move with the real host.
Names
Inside a Sim, name resolution never leaves the process. getaddrinfo, getnameinfo and
gethostbyname — and on Windows GetAddrInfoW, GetNameInfoW — answer from the sim's host table
and a built-in localhost (::1, 127.0.0.1; on Windows "" too). Any other name fails as the
host OS fails an unknown name: EAI_NONAME (std's "failed to lookup address information"),
HOST_NOT_FOUND, WSAHOST_NOT_FOUND (11001), at once and without a packet.
let sim = builder.add_host.build;
sim.run;
| Call | Effect |
|---|---|
add_host(name, addrs) / remove_host(name) |
Resolves name (case-insensitive, one trailing dot ignored) to addrs in table order. Also on SimBuilder and Sim. A name with no address of the asked family fails with EAI_NONAME (glibc, macOS) or WSANO_DATA (Windows). |
set_dns_policy(name, ..) / set_default_dns_policy(..) |
DnsPolicy { latency, failure, failure_rate }: the lookup waits latency on the sim's clock (virtual, so it time-skips and orders under deterministic()), then fails with failure — or, with a failure_rate, fails that often (TryAgain unless failure says otherwise), drawn from the sim's seed. Failures are logged as Fault::Dns. |
SimBuilder::resolve_real(name) |
Looks name up on the real resolver when the sim is built and keeps the answer; panics if it fails. |
SimBuilder::real_dns() |
Sends unknown names to the real resolver, counted as unmodelled calls. Panics in build() with deterministic(), since a real lookup blocks outside the schedule. |
Numeric hosts, a null node and service names go to the real libc / ws2_32, so hints semantics and
error precedence are the host's own; Windows reads a shorthand such as 127.1 as numeric only with
AI_NUMERICHOST, and otherwise as a name. A known name is answered by asking the OS for each of its
addresses as a numeric host with the caller's hints and joining the lists it returns; freeaddrinfo
takes such a list apart again on any thread, even after the sim is gone. AI_ADDRCONFIG is
ignored. getnameinfo reverses an address to the first table name holding it (and loopback to
localhost on Linux and macOS), otherwise numerically, or EAI_NONAME / WSAHOST_NOT_FOUND with
NI_NAMEREQD. The asynchronous resolvers (getaddrinfo_a, GetAddrInfoEx, DNSServiceGetAddrInfo,
DnsQuery_W, res_query) are not modelled and are reported as unmodelled calls.
Signals
Each Sim keeps its own signal dispositions. The code under test installs its handlers with the
OS's own calls, so ctrlc and signal-hook run unchanged, and the real process's
handlers are never touched; two sims in one test binary each see only their own.
let sim = new;
let signals = sim.signals; // Clone + Send, usable from any thread
sim.run;
| Platform | What is simulated |
|---|---|
| unix | sigaction, signal (BSD semantics), raise, kill of the own process (getpid(), 0, -getpgrp(); kill(-1) skips the caller, as on the host, so it goes to the OS) and pthread_kill for SIGINT, SIGTERM, SIGHUP. Every other signal, and other pids, go to the OS. A sim's table starts from the process's dispositions, so an inherited SIG_IGN reads back. Delivery is synchronous on the sending thread, as POSIX requires of a signal a process sends itself: SA_SIGINFO handlers get a siginfo_t with si_code SI_USER (Linux: SI_TKILL for raise/pthread_kill) and si_pid the parent for the test's signals; SA_RESETHAND and SA_NODEFER behave as on the host. pthread_kill to another thread of the sim runs the handler there at its next hooked call. |
| Windows | SetConsoleCtrlHandler (a list per sim, plus the ignore-CTRL+C flag), GenerateConsoleCtrlEvent for this process (group 0 or its own id; asynchronous, as on Windows; CTRL+C to a nonzero group succeeds and reaches no one, as Windows documents) and the CRT's signal/raise for SIGINT, SIGBREAK, SIGTERM. Each event is handled on a new thread, last registered handler first, until one returns TRUE. A close event's handlers get 5 s and a logoff or shutdown's 20 s of sim time before the delivery reports HandledThenExit. |
| both | Sim::raise_signal / SignalHandle::raise returns Handled, HandledThenExit, DefaultAction (never carried out), Ignored, or Unavailable for a signal the host OS lacks (Break/Close/Logoff/Shutdown on unix, Terminate/Hangup on Windows). From a thread outside the sim it is delivered on a new thread of the sim, joined before it returns; inside an executive's timestamp its effects wait for the timestamp to end like any other wake. |
The waits those libraries' threads block on are simulated: POSIX semaphores (sem_wait,
sem_timedwait, sem_clockwait, sem_trywait, sem_post; named semaphores on macOS), Windows
semaphore handles in WaitForSingleObject / ReleaseSemaphore, and socketpair(AF_UNIX, SOCK_STREAM | SOCK_DGRAM), served by the fabric like any socket (poll, epoll, kqueue, EOF,
EPIPE; an unnamed address as the host reports it). pipe/pipe2 stay real. Under
deterministic(), once every root thread of a run has left and nothing can run, the schedule lets
leftover daemon threads (the ctrlc thread) go; they wait outside it until the sim runs again.
SimBuilder::forward_real_signals() opts a sim in to the real SIGINT/SIGTERM/SIGHUP (a console
control handler on Windows): while any sim forwards, a process-wide forwarder delivers each real
signal into every forwarding sim, and one that no sim handled or ignored is raised for real against
the process's own disposition. The last forwarding sim to drop restores the originals. Signal masks,
sigwait/signalfd, EINTR on blocked calls and SIGALRM are not modelled.
Reaching the real machine: snare::real
The harness and other-side emulation sometimes need the real OS — to read a real fixture file,
or a real environment value — while the code under test stays simulated. Wrap those harness
operations in real:
let key = real; // real environment
let fixture = real.unwrap; // real file
real(f) runs f with interposition switched off on the calling thread only (it nests, and other
managed threads stay simulated). Threads spawned inside real also run outside the simulation.
Keep resource construction and execution of the code under test inside Sim::run, outside any
real closure. A native object created by the harness must not become a resource of the code
under test.
The cargo snare harness
Write deterministic tests in the crate, guarded by #[cfg(snare)], and run them under interposition:
$ cargo snare test # builds with --cfg snare, patches in the shims, runs the tests
cargo snare sets --cfg snare (and --cfg rustix_use_libc) and injects the drop-in shims via
cargo --config patch.crates-io... — no edits to your Cargo.toml. Plain cargo test leaves the
#[cfg(snare)] tests out.
Shims (shims/)
Some crates bypass libc in ways an import-table hook cannot see — raw io_uring rings in mmap'd memory,
AF_XDP UMEM. For those, snare provides drop-in replacement crates injected via [patch.crates-io]:
io-uring— emulates the ring in-process, executing each SQE through libcread/writeso the interposer sees ordinary I/O.xsk-rs(AF_XDP) — routes frame TX/RX through libcsend/recv.sc,syscalls— route raw syscalls through libc so thesyscallhook catches them.
The shims honor snare::real: each resource is tagged with the passthrough mode it was created in, and
using it in the other mode panics rather than silently mixing an emulated ring with real-OS I/O.
Platform support
| Linux (x86_64, aarch64) | macOS (aarch64, x86_64) | Windows (x86_64, aarch64) | |
|---|---|---|---|
| Interposition | ELF GOT | Mach-O __got (+ variadic trampoline on arm64) |
PE IAT |
| Time / entropy / sleep | ✅ | ✅ | ✅ |
| TCP | ✅ | ✅ | ✅ (Winsock) |
| UDP | ✅ | ✅ | ✅ (Winsock) |
| Readiness | ✅ (epoll, poll, eventfd) |
✅ (kqueue/kevent, poll) |
✅ (WSAPoll, select) |
| Raw L2 | AF_PACKET |
/dev/bpf |
pcap/npcap (structural) |
| Host scheduling | ✅ | ✅ (pthread/Mach) | ✅ (WinHost) |
| Interfaces and routing | ✅ (getifaddrs, SIOCGIF*, netlink) |
✅ (getifaddrs, SIOCGIF*, sysctl) |
✅ (IP Helper) |
| NIC tuning / PTP | ✅ (ethtool, qdiscs, sysfs) |
partial (sysctl) |
adapter properties + restart (SetupAPI/registry/CM_*) |
| Process limits | ✅ (RLIMIT_RTPRIO/NICE/MEMLOCK, mlock) |
✅ (RLIMIT_MEMLOCK, mlock) |
priority class, working set (WinHost) |
| Names / signals | ✅ | ✅ | ✅ (GetAddrInfoW, console control events) |
Linux means glibc (target_env = "gnu"). Every other target — musl, 32-bit x86 and ARM, Android,
the BSDs, illumos, Redox — stops at a single compile_error! naming the supported targets. A crate
that also builds for those targets gates the dev-dependency:
[]
= "2"
and puts its sim tests behind the same cfg.
The minimum supported Rust version is 1.88.
Repository layout
crates/snare-interpose/ the engine: patchers, hooks, Domain, backend traits, real()
crates/snare/ Sim, SimHost, HostProfile, EasyBuilder, Fabric, VirtualFs, testers
crates/cargo-snare/ the `cargo snare` subcommand
shims/ io-uring, xsk-rs, sc, syscalls drop-ins ([patch.crates-io])
Running the tests
cargo test # macOS / Linux
cargo clippy --all-targets
cargo xwin build --workspace --target x86_64-pc-windows-msvc # Windows cross-build (cargo-xwin)
Linux-only surfaces are exercised in a container (docker run --rm -v "$PWD":/work -w /work rust:latest cargo test). scripts/test-windows.sh cross-builds the workspace and runs its Windows tests in a
Parallels VM; scripts/test-windows-host.sh runs only the scheduling-plane target.
The Windows module-loading test needs the interpose-probe DLL in the target's profile directory,
beside deps. The full Windows runner builds the workspace first to produce it; cargo test --no-run alone does not. On a native Windows checkout, run cargo build -p interpose-probe before
cargo test --workspace --all-targets, using the same target and profile for both commands.
A note on the source
Behaviour that mirrors a kernel interface is cited in the code to its authority — a man-page section
(man 2 sched_setscheduler), a kernel header (<linux/if_link.h> struct rtnl_link_stats64), or a
Documentation/ path (Documentation/networking/timestamping.rst) — so the modeled struct layouts,
magic numbers, and errno conventions can be checked against the real thing.