Module death_note 

Durable record of what killed the server process on paths that never reach the graceful-shutdown log. The server’s death note: a durable record of what is killing the process on paths that never reach the graceful-shutdown log.

Two production servers died on 2026-08-16 (pids 52844 and 2681) with identical faces: the main log ends mid-flight on routine lines — no drain, no panic output, no OS crash report. This module exists so the third such death is a diagnosis instead of a mystery. It arms three recorders around the run loop, writing to <AION_HOME>/logs/aion-server.death.log:

• an armed/disarmed bracket: one line when the server boots, one line when the run scope exits by ordinary control flow (clean shutdown or an error return, written by [DeathNote]’s Drop). A note that is armed but never disarmed, with no other entry, means the process was destroyed without the run loop seeing it — the SIGKILL / raw-_exit class no in-process handler can observe.
• a panic hook, chained in front of the previously installed hook, that records every panic’s thread, location, payload, and backtrace. A panic in a spawned task can be survivable; the entry says so. Death by panic reads as a PANIC entry followed by the drop-written DISARMED line as the unwind leaves the run scope.
• a termination-signal watcher for the catchable signals whose default action kills the process silently (SIGHUP, SIGQUIT, SIGUSR1, SIGUSR2): the note records the signal, then re-applies the signal’s default action so process behaviour is unchanged — the death is noted, not prevented. SIGTERM and SIGINT are recorded as observations only: the graceful drain in [crate::run] owns those two, and both watchers coexist because tokio and this module register through the same signal-hook-registry chain.
• breadcrumbs: work sites call [breadcrumb] as they begin a unit of in-flight work (today: every declared action body the server executes), so a corpse’s LAST breadcrumb names what was running when the process died. The third 2026-08-16 death (pid 25680, ~16:27Z) happened mid-declared-action with only an ordinary INFO log line to say so; this makes that fact a note entry the corpse reader sees first.

Every entry carries its pid

One home’s note is shared by every server that ever ran against it, and those lives INTERLEAVE: an unclaimed multi-server boot, a restart’s succession handover, and — routinely, since the birth claim landed — a second aion server that arms, is refused the home, and abandons. So the frame of every line is <rfc3339> pid=<pid> <ENTRY>, and the reader (crate::control::outcome::read_fate) SELECTS by pid rather than scanning a bracket until the next ARMED.

🔴 That bracket scan was a real defect, measured 2026-08-26: a refused boot’s ARMED line landed between a live server’s SIGNAL and its OUTCOME, the scan stopped there, and aion server stop reported “the bracket never closed … the kill -9 shape” about a server that had just drained cleanly with its outcome on disk two lines below. Interleaving is ORDINARY now, so the framing has to be right rather than the interleaving rare.

There is deliberately no reading of the un-framed format that preceded this one. A line with no pid= tag is reported as UNATTRIBUTABLE — the note is on disk and readable by eye — never guessed onto a pid.

NOT covered, stated plainly: SIGKILL/SIGSTOP (uncatchable by kernel contract); the fatal-fault signals (SIGSEGV/SIGBUS/SIGILL/SIGFPE) and raw abort(), whose handlers must run in async-signal context and therefore require unsafe this workspace denies (the OS crash reporter remains the observer for that class — both 2026-08-16 corpses left no crash report, which is itself evidence against that class); and a direct exit() that bypasses the run scope. With the note armed, each of those reads as ARMED-without-DISARMED plus the absence of every entry above — a narrow, named remainder instead of an anonymous death.

Structs

DeathNote

The armed death note. Constructed by DeathNote::arm early in the server run loop; its Drop writes the DISARMED line, so any ordinary exit from the run scope — clean shutdown, error return, or a panic unwinding through it — closes the bracket on the record.

Functions

breadcrumb

Append a BREADCRUMB entry to the armed death note, naming work now in flight so a corpse’s last breadcrumb identifies the site that never finished. A silent no-op when no note is armed (tests, tools, the CLI).

note_path

The death note’s path under home — the ONE derivation, shared with the reader (crate::control::outcome::read_fate) so the writer and the reader cannot drift apart: a drifted reader would report “the exit left no recorded account” over a perfectly good note, silently.