Skip to main content

Module observe

Module observe 

Source
Expand description

What runnerd learns about running jobs without GitHub’s help: CPU, memory and network of each runner container from its cgroup v2 files and /proc (metrics), the job a runner took and its log from inside the container (capture), and which runner pulled what through the Docker Hub mirror (mirror). Reads files and parses output; it starts no container, writes nothing to the host, and decides nothing about scheduling.

Modules§

capture
The job a runner took and the job’s log, read from inside its container while it runs. A small script, run with docker exec as root, waits for the runner’s _diag/Worker_*.log, prints the job’s ID (check_run_id) and run ID from the job message there, then follows the job’s log pages in _diag/pages, which the runner writes in 4 KiB blocks and deletes once uploaded: the script hard-links each page so none is lost, and prints what it has not printed before. These pages are what GitHub shows, with secrets already masked by the runner.
metrics
CPU, memory and network of runner containers and of their slice, read from cgroup v2 files and /proc/<pid>/net/dev every SAMPLE_INTERVAL. Memory leaves out the page cache the way docker stats does (memory.current less inactive_file), so the numbers match what the bash tool reported. A container whose files vanish ends its series; that is not an error.
mirror
Which runner the Docker Hub mirror served, and how many bytes: the mirror’s access log (Docker log lines with timestamps, in the combined log format) attributed by source address. A runner container’s Docker Engine pulls for every container of its job, so the mirror sees the runner container’s address. An address belongs to a runner only between its container’s start and its removal, so a later container given the same address never gets an earlier one’s bytes. Docker’s --since takes whole seconds, so each read starts with lines already counted; a MirrorPosition skips them by their exact time, and among lines of the same time, by how many.