Module sysmon 

Session-level system-performance sampler.

Enabled per session with sysmon=<categories> — sysmon=all, sysmon=any, or a comma list of cpu,io,ram,rambw,storage. all and any differ in exactly one way: all ABORTS when a subsystem is unavailable, any enables what the host supports and NAMES what it skipped — an explicit opt-in to best-effort, so the skip is announced rather than silent. Reads /proc (and statvfs) on a fixed interval (default 5 s, sysmon-interval=<seconds>) and publishes host-side utilization as session-scoped gauges plus a live observer callback for display surfaces: measurement of the machine under the workload, not of the workload.

The categories:

• cpu — /proc/stat. Three separate measures: the MEAN utilization from the aggregate cpu line, the MAXIMUM single-core saturation from the cpuN lines (with the core named), and the QUARTILES of the per-core distribution (p25/p50/p75). A pinned compaction thread can saturate one core while the mean reads 3% — both facts matter and neither substitutes for the other.

  The quartiles exist because neither the mean nor the max answers “is there headroom”. The max says 100% for a single hot thread on an idle box; the mean says 3% for a box where every core is at 3% and for a box with one pegged core, without distinguishing them. p50 low with max_core pegged is one hot thread with headroom to spare; p50 high is genuine saturation. Any adaptive guard that throttles on CPU wants the quartile, not the max. The per-core vector is already built to find the max, so the shape costs one sort per window.

• io — /proc/diskstats, every line. Utilization per device is Δio_ticks / Δwall (io_ticks is the 10th stat field: milliseconds the device had I/O in flight — the same quantity iostat -x %util reports). Only the HIGHEST device utilization is recorded, with the device named. On NVMe this saturates as an idle-detector rather than a capacity meter (dozens of requests run in parallel), which is exactly what a “is the disk the bottleneck” glance wants.

• ram — /proc/meminfo. Two separate measures: (MemTotal − MemAvailable) / MemTotal (committed: what is actually claimed and not readily reclaimable — the kernel’s own estimate) and (MemTotal − MemFree) / MemTotal (everything, page cache included). On a database host the second sits near 100% by design; the pair reads as “how much is spoken for” vs “how much is touched”.

• rambw — memory bandwidth via resctrl MBM (/sys/fs/resctrl/mon_data). REQUIRED-EXPLICIT when enabled: if the resctrl interface is not mounted (or sysmon-membw-gbps= is not set to provide the peak reference), the session ABORTS with instructions — never a silent skip. sysmon=all includes it, so all on a host without resctrl aborts; a host without it runs sysmon=cpu,io,ram,storage.

• storage — filesystem SPACE utilization: statvfs over every /dev/-backed mount in /proc/mounts (deduplicated by source device), 1 − available/total per mount, only the highest recorded, mount point named. Space is the disk measure io cannot see: a device can be I/O-idle and one write from full.

Counters are cumulative, so every rate-like utilization here is a pairwise delta over the sample window; the published gauge is the latest window’s value and any windowing beyond that belongs to MetricsQL at query time.

Structs

Categories

Which categories a sysmon= setting enabled.

CpuReading

Per-category CPU readings: the mean, the hottest core, and the SHAPE of the per-core distribution.

CpuTicks

Cumulative jiffy counts for one cpu line: (busy, total).

MemInfo

The three /proc/meminfo fields the two utilization measures need, in kB as the kernel reports them.

SysmonConfig

Sampler configuration, resolved by the runner from session params.

SysmonSample

One completed sample window. Each field is Some exactly when its category was enabled — a disabled category is absent, not zero.

Enums

Selection

What a sysmon= value asked for: a fixed category set, or “whatever this host supports”.

Functions

check_rambw_requirements

The rambw prerequisites, checked BEFORE the session starts so an unsupported host aborts with instructions instead of silently monitoring less than was asked for.

core_util_quartiles

Per-core utilization quartiles between two snapshots: (p25, p50, p75, cores). Same shared-prefix rule as max_core_util.

cpu_util

Utilization between two tick snapshots: Δbusy / Δtotal.

max_core_util

The most saturated core between two snapshots. Core lists of different lengths (offline/online between samples) compare over the shared prefix.

max_disk_util

The highest per-device utilization between two diskstats snapshots taken dt_ms apart. Devices present in only one snapshot are skipped (hotplug between samples); an empty intersection yields None.

mem_utils

The two memory measures: (committed, everything-including-page-cache).

parse_categories

Parse a fixed category list (all or comma names). See parse_selection for the any form.

parse_dev_mounts

WRITABLE /dev/-backed mount points from /proc/mounts, deduplicated by source device (bind mounts and btrfs subvolumes re-list one device many times; space is a per-DEVICE fact).

parse_diskstats

Per-device cumulative io_ticks (ms with I/O in flight) from /proc/diskstats. Field layout per the kernel’s Documentation/iostats: major minor name <17 stat fields>; io_ticks is stat field 10, i.e. whitespace token 12.

parse_meminfo

parse_proc_stat

Aggregate + per-core cumulative ticks from /proc/stat.

parse_selection

Parse a sysmon= value: all, any, or a comma list of category names. Unknown names are errors that NAME the valid set — a typo silently monitoring nothing would be the failure mode this surface exists to avoid.

resolve_any

Resolve sysmon=any against THIS host: every category the host supports, plus one human-readable line per category that had to be skipped and why. cpu/io/ram/storage are /proc-backed and always available on Linux; rambw carries real prerequisites.

spawn

Spawn the sampler. check_rambw_requirements must have passed first — the runner aborts the session on its Err rather than calling this. Runs until session shutdown; publishes each window to the session gauges and to observer.sysmon_update.