Isolate-wide GC and heap, polled from the engine rather than parsed from a
log the graal fleet does not write. Cumulative, so
rate(engine_gc_pause_millis_total) / 1000 is the fraction of wall clock
stopped, which is the figure that identified #684. Every room on the node
shares one isolate, so a collection here stops all of them at once.
The collection counters carry a collector label and are emitted by
record_engine_gc_collector; emitting an unlabelled copy of the same name
here would double every sum.
Split by collector. Incremental collections are a few milliseconds and
harmless; a complete collection traces the whole live set, and this live set
is a permanently reachable card database, so it costs about 1.3ms per MB
whether it reclaims anything or not. Only the complete series is worth
alerting on.
Wall clock the engine was stopped, from a probe inside the isolate rather
than from the collector. record_engine_gc_collector is the better signal
where it exists, but Substrate registers collector beans for its Serial GC
only, so on the G1 image it reports nothing at all and this is the only
series that shows a stall. Cumulative, so
rate(engine_stall_millis_total) / 1000 is the fraction of wall clock lost.
The rules work between a seat answering and the next prompt appearing, split
by seat count. Measured over two days of captures, four seats run this at a
p99 of about 2.9s against 0.35s for two, and put 2-5% of decisions over two
seconds where two seats put none. A fleet-wide quantile averages the many
clean rooms against the few bad ones and shows neither.