Module progress 

FORWARD-PROGRESS ODOMETER, the engine’s own answer to “is this worker busy, or hung?” (lane/health-busy-vs-hung, memra#50, 2026-09-03).

THE DEFECT THIS EXISTS FOR. /health used to read ONE signal for a BUSY worker: the age of the scheduler-loop heartbeat (WorkerHealth::beat, stamped once per iteration in worker.rs). That is a proxy for progress, not progress itself, and the proxy broke the moment ONE iteration legitimately ran longer than the stall threshold. Measured, on the glm5 ship-gate stress arm (darklanes research/glm5-serving-launch-20260901/soak-20260901/ RESULT.md, RED finding 1): waves of 8 to 22 admitted sessions carrying 20k-88k-token prompts primed inside one scheduler iteration, the beat did not land for >120 s while the worker was PROGRESSING NORMALLY, /health answered 503 unhealthy for three guard ticks, and the supervisor SIGTERMed a server with 22 requests in flight. A false restart costs every in-flight request plus a full model load.

WHAT THIS PUBLISHES, and why it is honest. Every completed PRIME CHUNK stamps this odometer: a token count, an event count, and the monotonic time of the last advance. The stamp sits where the chunk’s host-side result already exists, the chunk’s logits are a Vec<f32>, i.e. a device-to-host copy has already drained that stream (see prime_chunk_ppn’s exit-publication note). So an advance is not “the host queued some launches”; it is “the device finished that chunk’s work and the host read the answer back”. That is the strongest liveness attestation available without a second thread.

WHAT IT CANNOT DETECT, stated so nobody reads more into it:

• A worker looping FOREVER INSIDE one chunk (a wedged kernel, a hung driver call, a deadlock inside a single prime call) advances nothing, so it is caught, but only after the stall threshold, exactly as before. This buys correctness under load, not faster hang detection.
• A worker making progress on the WRONG work (a livelock that re-primes the same chunk forever, a scheduler that starves one session while another runs) reads healthy. This is a liveness signal, not a fairness or a correctness one.
• Chunk granularity is the resolution: with MEMRA_PRIME_CHUNK=0 a prompt primes in one call up to PRIME_CHUNK_LAUNCH_CAP (65,520 tokens), so the odometer’s own gap can be a whole 65k-token prime. A deployment that pins the monolithic rollback seam is back to sizing MEMRA_HEALTH_STALL_S from its prefill rate by hand.
• It is PROCESS-GLOBAL, not per-session. One live session priming keeps the process healthy while another session’s work is stuck behind it. That is correct for the question /health asks (“should this process be RESTARTED?”) and wrong for any per-request SLO, which admission and the first-token deadline own instead.

Structs

Progress

The odometer’s observable state.

Functions

events

Completed prime chunks so far. Used by prime_cache_overlaid to tell a CHUNKED walk (which already stamped per chunk) from a MONOLITHIC one (which stamped nothing, and whose only honest progress point is the call’s own completion).

note_prime_rows

One prime chunk completed on this process’s worker thread, carrying rows token rows.

snapshot

What the odometer has seen. None until the first advance, a process that has never primed anything reports nothing rather than reporting an age measured from boot.