1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
//! Is trusty-memory available right now? (#7685)
//!
//! Why: the owner ruling of 2026-09-12 makes Claude Code's own auto memory a
//! FALLBACK — trusty-memory is the memory, and auto memory stays on only when
//! trusty-memory is not available. Every write site that turns auto memory off
//! (the project-tier `autoMemoryEnabled: false` key and the
//! `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1` spawn assignment) therefore needs one
//! yes/no answer, resolved the same way in every caller. A session that loses
//! both memories is the failure this module exists to prevent.
//! What: [`probe_memory_reachability`] issues ONE `memory.health` call against
//! the derived socket and classifies the answer as a [`MemoryReachability`].
//! Only a daemon that answers AND reports `status: "ok"` is reachable — a wedged
//! or degraded daemon, an unreadable body and a refused call all keep the
//! fallback on. [`probe_memory_reachable_blocking`] is the same probe as a
//! boolean for the synchronous launch path, run on its own short-lived thread so
//! it is safe to call from inside an async runtime. [`resolve_memory_reachable`]
//! is what launch consumers actually call: it takes whatever the caller already
//! resolved and probes only when nothing did, so one launch asks the daemon once
//! rather than once per write site.
//!
//! This is deliberately NOT [`crate::daemon::doctor`]'s `probe_health`: that one
//! retries three times and classifies four outcomes because it renders a doctor
//! row. A launch needs a fast, single, fail-safe gate — see [`PROBE_TIMEOUT`].
//! Test: `core::memory_reachable::tests`.
use std::time::Duration;
use trusty_common::memory_rpc::MemoryHealthStatus;
/// How long one launch-time reachability probe may take.
///
/// Why: this runs on the launch path, twice per managed session (preparation
/// and spawn). The doctor's 10-second budget is for a report an operator is
/// waiting on; a launch cannot pay that, and the fail-safe answer (`false` →
/// auto memory stays on) is cheap to be wrong about.
/// What: 1.5 s, the whole budget for the single `memory.health` round trip.
/// Test: `unreachable_socket_answers_false`.
pub const PROBE_TIMEOUT: Duration = Duration::from_millis(1500);
/// The trusty-memory health method both probes call.
const HEALTH_METHOD: &str = "memory.health";
/// What one reachability probe observed about trusty-memory (#7685).
///
/// Why: "is trusty-memory the memory" has one yes and several different noes,
/// and an operator reading `tm doctor`'s `auto_memory` row needs to tell them
/// apart — a daemon that is down is restarted, a wedged one is sampled first
/// (#4001). The launch path only needs the yes/no, via [`Self::is_reachable`].
/// What: `Reachable` only when the daemon answered with a healthy status;
/// `Unhealthy` when it answered with any other status (or none it could read);
/// `Refused` when it answered with a JSON-RPC error; `Unreachable` when nothing
/// answered within [`PROBE_TIMEOUT`].
/// Test: `reachable_when_the_daemon_reports_ok`,
/// `wedged_daemon_is_not_reachable`, `degraded_daemon_is_not_reachable`,
/// `malformed_health_body_is_not_reachable`,
/// `health_body_without_a_status_is_not_reachable`,
/// `refused_health_call_is_not_reachable`,
/// `unreachable_socket_answers_false`.
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum MemoryReachability {
/// Answered, and reported `status: "ok"`.
Reachable,
/// Answered, but with a status that is not `"ok"`.
Unhealthy(MemoryHealthStatus),
/// Answered the health call with a JSON-RPC error, carrying its code.
Refused(i64),
/// Nothing answered: a dial failure, a transport error, or a timeout.
Unreachable,
}
impl MemoryReachability {
/// Whether trusty-memory can be relied on as the memory.
pub fn is_reachable(&self) -> bool {
matches!(self, Self::Reachable)
}
/// One clause naming what the probe saw, for an operator-facing row.
pub fn describe(&self) -> String {
match self {
Self::Reachable => "trusty-memory is healthy".to_string(),
Self::Unhealthy(MemoryHealthStatus::Wedged) => {
"trusty-memory is answering but reports its worker pool WEDGED (its oldest \
in-flight operation has outlived its bound)"
.to_string()
}
Self::Unhealthy(MemoryHealthStatus::Degraded) => {
"trusty-memory is answering but reports itself DEGRADED".to_string()
}
Self::Unhealthy(other) => {
format!("trusty-memory is answering but reports no healthy status ({other:?})")
}
Self::Refused(code) => {
format!("trusty-memory refused its health call (code {code})")
}
Self::Unreachable => "trusty-memory is not answering (unreachable)".to_string(),
}
}
}
/// Probe the host's trusty-memory and classify what answered.
///
/// Why: the one implementation of "is trusty-memory available" that the launch
/// path, `tm doctor`'s `auto_memory` row and `tm doctor --fix` all consult, so a
/// session and the check that grades it can never disagree about which state
/// they are in.
/// What: resolves the daemon socket via
/// `trusty_common::memory_rpc::resolve_memory_socket_or_unreachable` — which
/// always yields a path, so an absent daemon simply fails to dial — and hands it
/// to [`probe_socket_reachability`].
/// Test: through [`probe_socket_reachability`].
pub async fn probe_memory_reachability() -> MemoryReachability {
let socket = trusty_common::memory_rpc::resolve_memory_socket_or_unreachable();
probe_socket_reachability(&socket).await
}
/// [`probe_memory_reachability`] against an explicit socket.
///
/// Why: the resolver reads the host's own daemon layout, which a test cannot
/// point anywhere; this seam is what lets the stub daemon answer instead.
/// What: one `memory.health` call bounded by [`PROBE_TIMEOUT`]. The answer's
/// `status` is read through the shared [`MemoryHealthStatus`]; only `Ok` is
/// [`MemoryReachability::Reachable`] (#7685: `"wedged"`, `"degraded"`, an unknown
/// value, a missing field and an unreadable body all fail closed, toward keeping
/// auto memory on). A JSON-RPC error is `Refused`; a transport failure or a
/// timeout is `Unreachable`.
/// Test: `reachable_when_the_daemon_reports_ok`,
/// `wedged_daemon_is_not_reachable`, `degraded_daemon_is_not_reachable`,
/// `malformed_health_body_is_not_reachable`,
/// `health_body_without_a_status_is_not_reachable`,
/// `refused_health_call_is_not_reachable`,
/// `unreachable_socket_answers_false`.
pub async fn probe_socket_reachability(socket: &std::path::Path) -> MemoryReachability {
match trusty_common::memory_rpc::call_memory_tool_at_with_timeout(
socket,
HEALTH_METHOD,
serde_json::json!({}),
PROBE_TIMEOUT,
)
.await
{
// #7685: an answer is not health — a wedged pool answers too.
Ok(body) => match MemoryHealthStatus::from_health_body(&body) {
MemoryHealthStatus::Ok => MemoryReachability::Reachable,
other => MemoryReachability::Unhealthy(other),
},
Err(e) => match e.downcast_ref::<trusty_common::memory_rpc::MemoryRpcError>() {
Some(rpc) => MemoryReachability::Refused(rpc.code),
None => MemoryReachability::Unreachable,
},
}
}
/// [`probe_memory_reachability`] as a boolean, for a synchronous caller.
///
/// Why: `prepare_session*` and `RuntimeAdapter::spawn` are synchronous and are
/// reached from BOTH async handlers (the daemon's spawn routes, the HTTP client)
/// and plain sync callers (`standalone::load`, `deploy_validate`). Threading an
/// awaited bool to all of them would turn a two-line behaviour change into a
/// cascade through seven unrelated call chains; the launch path already makes
/// blocking daemon-touching calls in exactly this position (`session_mcp_env`
/// touches trusty-search, `claude_supports_native_output_style` spawns
/// `claude`). See #7685.
/// What: runs the probe on its own scoped thread with a private current-thread
/// runtime, so it is correct whether or not the caller already sits inside a
/// tokio runtime (`Handle::block_on` would panic there). Any failure to build
/// the runtime, and a panicking probe, both answer `false` — the fail-safe
/// direction, since `false` leaves auto memory ON.
/// Test: `blocking_probe_answers_inside_a_runtime`.
pub fn probe_memory_reachable_blocking() -> bool {
block_on_probe(async { probe_memory_reachability().await.is_reachable() })
}
#[cfg(test)]
thread_local! {
/// How many adapter spawns on this thread had no resolved reachability (#7685).
///
/// Why: a launch path that throws away the reachability its preparation
/// resolved makes the runtime adapter probe a second time, and that is
/// invisible in the session it produces. Counting the adapter's own
/// unresolved reads lets a test drive a real launch path and assert the
/// adapter reused the prepared answer. Thread local, so parallel tests cannot
/// disturb the count.
/// What: incremented by [`resolve_spawn_memory_reachable`] when the adapter
/// was built with `None` and is about to probe.
/// Test: `spawn_managed_on_main_hands_the_adapter_the_prepared_reachability`.
pub(crate) static ADAPTER_REPROBES_ON_THIS_THREAD: std::cell::Cell<usize> = const { std::cell::Cell::new(0) };
}
/// The reachability a launch should act on: a value already resolved, or a probe.
///
/// Why (#7685): one launch used to answer this question twice — once in
/// `prepare_session_inner` and once again in the runtime adapter's spawn — so a
/// hung trusty-memory cost the launch two [`PROBE_TIMEOUT`] budgets instead of
/// one. Every consumer now asks through here with whatever the caller already
/// resolved, so threading a value UP the call chain is the only thing needed to
/// collapse the second probe.
/// What: `Some(value)` is taken verbatim and no socket is dialled; `None` runs
/// [`probe_memory_reachable_blocking`].
/// Test: `resolved_reachability_skips_the_probe`,
/// `unresolved_reachability_runs_the_probe`.
pub fn resolve_memory_reachable(resolved: Option<bool>) -> bool {
resolve_memory_reachable_with(resolved, probe_memory_reachable_blocking)
}
/// [`resolve_memory_reachable`] for the runtime adapter's spawn (#7685).
///
/// Why: the adapter is the one consumer whose probe is always a SECOND probe —
/// preparation already asked — so a launch path that hands it `None` pays twice.
/// A separate entry point makes that re-probe countable by a test without
/// counting preparation's own probes.
/// What: [`resolve_memory_reachable`]; under `cfg(test)` a `None` also bumps
/// `ADAPTER_REPROBES_ON_THIS_THREAD`.
/// Test: `spawn_managed_on_main_hands_the_adapter_the_prepared_reachability`.
pub fn resolve_spawn_memory_reachable(resolved: Option<bool>) -> bool {
#[cfg(test)]
if resolved.is_none() {
ADAPTER_REPROBES_ON_THIS_THREAD.with(|n| n.set(n.get() + 1));
}
resolve_memory_reachable(resolved)
}
/// [`resolve_memory_reachable`] with the probe supplied.
///
/// Why: the seam a test counts. Asserting "the adapter did not probe again" needs
/// the probe itself to be observable, and the real one dials a host socket no
/// test can point anywhere.
/// What: `resolved.unwrap_or_else(probe)` — nothing else, so the production
/// entry point above cannot drift from what the test drives.
/// Test: `resolved_reachability_skips_the_probe`,
/// `unresolved_reachability_runs_the_probe`.
pub(crate) fn resolve_memory_reachable_with(
resolved: Option<bool>,
probe: impl FnOnce() -> bool,
) -> bool {
resolved.unwrap_or_else(probe)
}
/// Drive one reachability future to completion from synchronous code.
///
/// Why: see [`probe_memory_reachable_blocking`]. Isolating the bridge keeps the
/// unsafe-to-get-wrong part (not touching the caller's runtime) in one place,
/// and lets a test drive a socket-pinned probe through the identical mechanism.
/// What: a scoped thread with its own current-thread runtime. A runtime that
/// cannot be built, and a probe that panics, both answer `false` — the fail-safe
/// direction, since `false` leaves auto memory ON. The panic is contained by the
/// explicit `join`: `std::thread::scope` re-raises only the panics of threads it
/// had to join itself.
/// Test: `blocking_probe_answers_inside_a_runtime`,
/// `a_panicking_probe_answers_false_without_propagating`.
fn block_on_probe<F>(probe: F) -> bool
where
F: std::future::Future<Output = bool> + Send,
{
std::thread::scope(|scope| {
scope
.spawn(|| {
let runtime = match tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
{
Ok(runtime) => runtime,
Err(err) => {
tracing::warn!(
"could not build a probe runtime for the trusty-memory \
reachability check; assuming unreachable: {err}"
);
return false;
}
};
runtime.block_on(probe)
})
.join()
.unwrap_or(false)
})
}
#[cfg(test)]
#[path = "memory_reachable_tests.rs"]
mod tests;