Module observe 

What runnerd learns about running jobs without GitHub’s help: CPU, memory and network of each runner container from its cgroup v2 files and /proc (metrics), the job a runner took and its log from inside the container (capture), and which runner pulled what through the Docker Hub mirror (mirror). Reads files and parses output; it starts no container, writes nothing to the host, and decides nothing about scheduling.

Modules

capture

The job a runner took and the job’s log, read from inside its container while it runs. A small script, run with docker exec as root, waits for the runner’s _diag/Worker_*.log, prints the job’s ID (check_run_id) and run ID from the job message there, then follows the job’s log pages in _diag/pages, which the runner writes in 4 KiB blocks and deletes once uploaded: the script hard-links each page so none is lost, and prints what it has not printed before. These pages are what GitHub shows, with secrets already masked by the runner.

metrics

CPU, memory and network of runner containers and of their slice, read from cgroup v2 files and /proc/<pid>/net/dev every SAMPLE_INTERVAL. A container’s memory is an estimate of what its job holds without page cache: memory.current less the file LRU lists (active_file, inactive_file), so anonymous memory, tmpfs and shared memory (shmem, on the anon lists) and kernel memory, reclaimable slab included, since slab in use (an open file’s dentry and inode) can’t be freed. Page cache on the file lists is usually the kernel’s to drop before memory.max OOM-kills (“reaches this limit and can’t be reduced”, Documentation/admin-guide/ cgroup-v2.rst), though dirty, writeback or pinned pages may not be dropped at once. docker stats subtracts only inactive_file, so it counts page cache read twice. The slice’s memory for the high-water mark is that conservative number, memory.current less inactive_file, as it always was: it stops starts before the slice is short of memory, not after. Whether a cgroup met its limit comes from memory.events, never from how close a sample came to it. A container whose files vanish ends its series; that is not an error. Beside the slice, it reads the host’s memory: MemTotal and MemAvailable from /proc/meminfo and the stall shares of /proc/pressure/memory; and, every HOLDERS_INTERVAL, which programs outside the slice hold the most of it, by the name and memory lines of each /proc/<pid>/status whose /proc/<pid>/cgroup is not in the slice. Only a program’s name leaves this module, never its command line, arguments, environment or paths.

mirror

Which runner the Docker Hub mirror served, and how many bytes: the mirror’s access log (Docker log lines with timestamps, in the combined log format) attributed by source address. A runner container’s Docker Engine pulls for every container of its job, so the mirror sees the runner container’s address. An address belongs to a runner only between its container’s start and its removal, so a later container given the same address never gets an earlier one’s bytes. Docker’s --since takes whole seconds, so each read starts with lines already counted; a MirrorPosition skips them by their exact time, and among lines of the same time, by how many.