Isolate-wide GC and heap, polled from the engine rather than parsed from a
log the graal fleet does not write. Cumulative, so
rate(engine_gc_pause_millis_total) / 1000 is the fraction of wall clock
stopped, which is the figure that identified #684. Every room on the node
shares one isolate, so a collection here stops all of them at once.
The collection counters carry a collector label and are emitted by
record_engine_gc_collector; emitting an unlabelled copy of the same name
here would double every sum.
Split by collector. Incremental collections are a few milliseconds and
harmless; a complete collection traces the whole live set, and this live set
is a permanently reachable card database, so it costs about 1.3ms per MB
whether it reclaims anything or not. Only the complete series is worth
alerting on.
Wall clock the engine was stopped, from a probe inside the isolate rather
than from the collector. record_engine_gc_collector is the better signal
where it exists, but Substrate registers collector beans for its Serial GC
only, so on the G1 image it reports nothing at all and this is the only
series that shows a stall. Cumulative, so
rate(engine_stall_millis_total) / 1000 is the fraction of wall clock lost.
The rules work between a seat answering and the next prompt appearing, split
by room shape. Measured over two days of captures, a room with four seats and
three bots runs this at a p99 of about 2.9s against 0.35s for every other
shape, and puts 2-5% of decisions over two seconds where the others put none.
A fleet-wide quantile averages the many clean rooms against the few bad ones
and shows neither. Bot decision time is not the problem and is not recorded
here: it sits at a p99 of 0.75s in exactly those rooms.