1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
//! `prctl(PR_SET_PDEATHSIG, SIGTERM)`, armed by a restart-handoff
//! replacement on its own, from inside its already-`exec`'d
//! `Application::serve()` - so the kernel delivers `SIGTERM` to it the
//! moment its real parent (whatever spawned it) dies, for any reason, a
//! crash or a `kill -9`/closed terminal included, not just the graceful
//! paths this codebase already handles elsewhere. `SIGTERM`, not
//! `SIGKILL`, gives an orphaned replacement a chance to run its own
//! existing graceful-shutdown path first; if it has none, `SIGTERM`'s
//! default disposition still terminates it, so this is never weaker than
//! `SIGKILL` in practice - just occasionally slower.
//!
//! **Why this runs post-`exec`, in the child, and not via a
//! `Command::pre_exec` hook in the parent (this function's own original
//! design):** attaching a `pre_exec` closure forces `std::process::
//! Command::spawn()` to fall back to a raw `fork()`+`exec()` sequence
//! instead of the safer, more efficient `posix_spawn()` it otherwise
//! uses - `posix_spawn()` has no way to run arbitrary user code between
//! fork and exec, so any `pre_exec` hook at all disables it. Forking a
//! *multi-threaded* process (which every `#[tokio::main]` app - including
//! every restart-handoff replacement this spawns - is, by default) is a
//! well-known, genuinely hazardous operation: the child inherits the
//! parent's entire memory image but only the one thread that called
//! `fork()`, including whatever locks other, now-vanished threads
//! happened to be holding at that exact instant (glibc's own malloc
//! arena lock being the classic case).
//!
//! This was confirmed as the *actual* root cause of a real, severe bug,
//! not a theoretical concern: a live restart handoff under sustained
//! traffic produced a consistent, sustained rate of genuine
//! `ConnectionRefused` failures (tens per second, for as long as traffic
//! kept flowing) - present identically regardless of which listener-
//! sharing mechanism was used (this crate's original fd-duplication
//! design and a since-tried `SO_REUSEPORT`-based replacement both showed
//! the exact same failure rate), which ruled out the listener code
//! entirely and pointed at something else common to every handoff.
//! Disabling *only* this `pre_exec` hook, with everything else unchanged,
//! eliminated the failures completely (a real end-to-end test went from a
//! sustained ~45-47 failures/second to a single transient blip in over
//! 11,000 requests). Calling `prctl` here instead - after `exec()` has
//! already replaced the process image - sidesteps the fork hazard
//! entirely: `Command::spawn()` can use `posix_spawn()` again (no
//! `pre_exec` hook attached anywhere), and by the time this function
//! runs, the process is a completely fresh image with no inherited
//! multi-threaded state at all. See `docs/GOTCHAS.md`.
//!
//! `libc` 0.2 (as pinned in this workspace) declares neither
//! `PR_SET_PDEATHSIG` nor `prctl` itself for real Linux targets (only for
//! Android and an obscure L4Re variant - confirmed by inspecting the
//! vendored source) - both are declared locally below rather than adding
//! a new dependency for two constants.
const PR_SET_PDEATHSIG: c_int = 1; // <linux/prctl.h>, stable since Linux 2.1.57
extern "C"
/// Arms this process's own death-signal delivery - see this module's own
/// doc comment for why this is called post-`exec`, by the replacement
/// itself, rather than via a pre-fork `Command::pre_exec` hook.
///
/// Deliberately temporary - see `disarm_pdeathsig`'s own doc comment for
/// why this must be cleared again once this process actually announces
/// itself ready, not left armed for its entire working life.
pub
/// Clears the death-signal delivery `arm_pdeathsig` just set - called from
/// `Application::serve()` right after this process announces itself ready
/// (`readiness::announce_ready`), before it starts actually accepting
/// connections.
///
/// This is the fix for a real, confirmed bug distinct from (and found
/// only after fixing) the `pre_exec`-vs-fork-hazard one this module's own
/// doc comment describes: `PR_SET_PDEATHSIG` fires whenever the parent
/// process exits *for any reason at all*, and this replacement's real
/// parent - the predecessor it's taking over from - is *expected* to
/// exit normally, deliberately, the moment its own drain completes, as
/// the successful conclusion of every ordinary handoff. Left armed, that
/// completely normal exit delivers `SIGTERM` to this brand-new
/// replacement, which its own `lifecycle::wait_for_termination()` (the
/// same graceful-shutdown signal handling every app already has) then
/// correctly, faithfully - but *wrongly*, given the actual situation -
/// treats as a real shutdown request: it stops accepting new connections
/// and starts draining, even though it's supposed to be the new
/// long-lived primary server, not something else being asked to retire.
/// Confirmed as the actual mechanism (not just plausible) by direct
/// experiment: a real end-to-end test showed a sustained, otherwise-
/// inexplicable ~45-47 failures/second for as long as traffic kept
/// flowing after a handoff, present with `arm_pdeathsig` called *either*
/// via the old `pre_exec` hook *or* this module's own post-exec
/// replacement for it - ruling out the fork-hazard theory entirely, since
/// post-exec calls it too - and vanishing completely (a single transient
/// blip in over 11,000 requests) only once pdeathsig was disabled
/// altogether. The fix here keeps the *original* safety guarantee
/// (detecting this process's own real parent already having crashed
/// *before* the handoff actually completes) while eliminating the false
/// positive: from the moment this process is about to start serving,
/// its predecessor's own exit is no longer something to react to at all.
pub