runnerd

runnerd runs ephemeral GitHub Actions runners on a Linux host: one fresh container per job, from one pool shared by the repositories you configure, kept inside a memory and CPU budget.

It is in early development: the daemon schedules runners, records every job, serves a JSON API and a web dashboard, and runnerctl reads and controls it from a terminal.

Install

Releases are early 0.0.x builds for Linux on x86_64 and aarch64. The installer downloads runnerd and runnerctl from the latest release, checks their SHA-256 against the release's checksum files, copies them to ~/.local/bin and runs runnerd doctor:

curl -fsSL https://github.com/ayagmar/runnerd/releases/latest/download/install.sh | sh

Pass --prefix DIR to install into DIR/bin instead, or --version v0.0.4 for a specific release (for v0.0.3 and earlier, run that release's own install.sh):

curl -fsSL https://github.com/ayagmar/runnerd/releases/latest/download/install.sh | sh -s -- --prefix /opt/runnerd

It never uses sudo and installs nothing else. Every release archive, install.sh and sha256.sum has a GitHub build provenance attestation. To check install.sh before running it, or an archive you downloaded yourself:

curl -fsSLO https://github.com/ayagmar/runnerd/releases/latest/download/install.sh
gh attestation verify install.sh -R ayagmar/runnerd
sh install.sh
gh attestation verify runnerd-x86_64-unknown-linux-musl.tar.xz -R ayagmar/runnerd

From crates.io, the daemon is the crate runnerd and runnerctl is in runnerd-cli, so install both. cargo install builds them with Rust 1.99 or later; cargo binstall downloads the same release archives as install.sh instead:

cargo install --locked runnerd runnerd-cli
cargo binstall runnerd runnerd-cli

Requirements

• Linux with cgroup v2, and Docker using the systemd cgroup driver.
• Sysbox, so each runner container can run its own Docker Engine without being privileged. Sysbox publishes .deb packages; on Arch Linux it is the AUR package sysbox-ce-bin; elsewhere, build it from source.
• A systemd slice whose MemoryMax is the budget for all runners.
• The GitHub CLI, logged in with the repo scope.

runnerd doctor checks each of these and prints the fix for what is missing. Given a config, it also checks the slice the config names and that the config fits in it:

runnerd doctor runnerd.toml

Runner image

image/ builds the container every job runs in: the GitHub Actions runner, its own Docker Engine, and one Dockerfile stage per profile on a common base. A profile's image_stage picks the stage:

docker build --target jvm -t runnerd-runner:jvm image

runnerd runs the image runnerd-runner:<image_stage>; build the stages your profiles use before starting it. Rebuilds install the same versions: Ubuntu packages come from one snapshot date and every download is checked against its SHA-256 (ADR 0023).

Documentation: https://ayagmar.github.io/runnerd/

Running

runnerd serve --config runnerd.toml

Without --config, it reads $XDG_CONFIG_HOME/runnerd/runnerd.toml. It reads the gh CLI's token at start (and again when GitHub rejects it), then every poll_interval ([host], default 15s) asks GitHub for each repository's queued jobs. A job is served when every label in its runs-on is one of its repository's labels. For each such job it registers a just-in-time runner on the repository and starts it in a free slot of the repository's profile, if the slice's MemoryMax still has room for the profile's memory. The runner takes one job, its container exits, and runnerd removes it; a runner that got no job within 5 minutes is removed. Jobs that cannot be served yet wait in GitHub's queue.

A slot is one concurrent job of a profile. Containers are new for every job, but each slot keeps a volume with its Docker Engine's images and its own changes to ~/.cache, the tool cache and the Android SDK, over the profile's read-only snapshot. When a profile has no job running and its slot 1 changed those by reseed_at ([caches], default 2G), slot 1's changes become the new snapshot. The pnpm store and the Maven repository are shared by all slots and emptied above their max_size when no job runs. The Docker Hub mirror runs while any runner does.

A restarted runnerd adopts the runner containers still running and deletes this host's runners that have no container. SIGTERM or SIGINT, or runnerd stop, stops polling and waits for running jobs; a second signal, or runnerd stop --force, kills them.

History

runnerd records every job its runners may run in a SQLite database, $XDG_STATE_HOME/runnerd/runnerd.db by default ([history] in the config changes the path and what is kept how long). For each job it keeps:

• its timeline: queued, its runner's container started, the job started, finished, and the container and runner registration removed;
• its steps with their start and end, as GitHub shows them;
• CPU, memory and network of its container, sampled every 2 s from the container's cgroup files (memory without page cache, as docker stats counts it) and /proc/<pid>/net/dev; the samples for 7 days, the job's totals and peaks for good;
• the bytes the Docker Hub mirror served it, from the mirror's access log;
• its log, up to 50M, as GitHub shows it, with the secrets the runner masked still masked.

It also keeps the sizes of the cache volumes, measured at start and whenever no runner runs after jobs ended, and the slice's CPU and memory use.

Where the data comes from:

• The job a runner took comes from inside its container: runnerd runs a small script there (docker exec, as root) that reads the job's ID from the runner's _diag/Worker_*.log within a second of the job starting. That is sooner than GitHub's API, whose list of queued jobs still shows a job for several seconds after a runner took it; runnerd does not start a second runner for a job one of its runners holds.
• Step timing comes from GitHub's jobs API (GET /repos/{owner}/{repo}/actions/jobs/{id}), read every poll while a job runs, not from the runner's _diag logs. At runner 2.338.1 the Worker log has every step's start and end, but under the runner's internal names (Set up runner, Pre Job Hook), not the ones GitHub shows (Set up job, Complete job), and in a free-form trace that changes between runner versions. The API's steps have second precision, which is all GitHub keeps.
• The log comes from the runner's _diag/pages, which the same script follows. The runner writes the job's log there in 4 KiB blocks and deletes each page once uploaded, so lines arrive in blocks of 4 KiB, and a quiet step's lines arrive when it ends.

API

runnerd serves JSON on $XDG_RUNTIME_DIR/runnerd.sock (mode 0600; only the user it runs as may connect). The types are in crates/core/src/api.rs and its modules. Times are seconds since the Unix epoch, sizes bytes, CPU millicores.

A forced pause cancels the run of each of the repository's running jobs (GitHub cancels runs, not single jobs, so the run's other jobs are cancelled too) and lets the runner report it cancelled, which takes under a minute; a runner still running 90 s later is removed. Its runner registration is deleted once GitHub lets go of it.

A new config takes effect without a restart, and running jobs keep running. A removed profile, or slots past a lowered slots, stay until their runners finish, and no runner starts in a profile while it runs as many as its slots. A runner still waiting for a job of a removed repository, or of one moved to another profile, is removed. Runners started afterwards get the new limits, image and labels. history.database changes only at the next start.

Every event but the live ones has an ID; send the last one seen as Last-Event-ID and the stream resumes after it, across daemon restarts. Live events are not stored: a client that missed some reads the job's samples or log. slot.changed, budget.changed and repository.changed carry the state as it is after a change, not each transition: a pause and a resume that land before the daemon publishes its status send nothing, since the state did not change. Applying repository.changed and repository.removed to the repositories of a status keeps them equal to what /v1/status returns.

curl --unix-socket "$XDG_RUNTIME_DIR/runnerd.sock" http://runnerd/v1/status
curl -N --unix-socket "$XDG_RUNTIME_DIR/runnerd.sock" http://runnerd/v1/events

Dashboard

runnerd serve also serves a web dashboard on 127.0.0.1:7477 (--web <address> picks another loopback address, --web off serves none). It shows:

• the memory budget as blocks, one per slot sized by its memory cap and filled to the job's last sampled memory;
• every slot with its job, current step, and memory and CPU over the last minutes;
• the slice's memory and CPU over the last half hour, against MemoryMax and the CPUs of all slots;
• each job's phases, its steps on one time axis, its CPU and memory against the caps its container ran with (the profile's limits as configured now for jobs recorded before runnerd kept them), and its log;
• the history, filtered by repository and outcome, with 30 days of jobs per day, run and wait times and memory peaks;
• the cache volumes with their bounds and their sizes over 30 days, with a button to reseed a snapshot or empty a cache;
• runnerd.toml in an editor that checks it against the slice's MemoryMax as you type and saves and reloads it only when it fits, next to the doctor's checks;
• pause and resume for each repository; a pause while a job runs asks first, and confirming cancels the job.

Open pages update as things happen, over one server-sent event stream per page. Memory and CPU are sampled every 2 s, so a peak shorter than that does not show; the dashboard calls them sampled peaks.

Any local user can reach the port, so every page needs a session cookie, which a browser gets from a one-time login URL. The daemon keeps the current URL in runnerd.web next to its socket ($XDG_RUNTIME_DIR/runnerd.web, mode 0600) and writes a new one each time one is used. runnerctl web opens it; without the CLI:

xdg-open "$(cat "$XDG_RUNTIME_DIR/runnerd.web")"

Writes such as a pause also need an Origin of the dashboard itself, so another site the browser has open cannot send them. The pages load nothing from elsewhere: the Datastar script and the font are in the binary.

runnerctl

runnerctl is the daemon's command-line client. It talks to the socket only (--socket to pick another), prints aligned text, coloured on a terminal unless NO_COLOR is set, and with --json prints the API's response type unchanged.

$ runnerctl status
runnerd 0.0.4  4.5G of 33G reserved in runnerd.slice

PROFILE  SLOT  STATE  REPOSITORY      RUNNER                       UP
jvm         1  busy   halyard/ledger  runnerd-5f2a91c0-jvm-1-k3j9  21s
jvm         2  free

REPOSITORY          PROFILE  STATE   QUEUED  LABELS
halyard/ledger      jvm      active       0  self-hosted,linux,x64,jvm
halyard/storefront  node     paused       2  self-hosted,linux,x64,node

Memory and CPU come from samples every 2 s, so a peak is the highest sample: a spike shorter than that does not show.

runnerctl tui

A full-screen dashboard for hosts reached over SSH, where the web dashboard is not open. It shows what the web dashboard does:

• the budget as one bar, a block per slot as wide as its memory and filled as far as its job uses it;
• a card per slot with the job, its step, run time, and memory and CPU sparklines;
• the slice's memory and CPU, the repositories, and the latest jobs;
• the history, filtered by outcome and repository;
• a job: its timeline, its steps as bars on one time axis, memory and CPU against its container's caps, and its log, followed while the job runs.

Profiles take the dashboard's colours in config order, mint first and pink second. NO_COLOR leaves bold, dim and block glyphs. At 80×24 the slots become one line each and the charts drop out first.

SIGINT, SIGTERM and SIGHUP quit and SIGTSTP suspends, each after giving the terminal back. When the daemon goes away the header says reconnecting. Once it is back, the event stream resumes after the last event read, or the state is read again if none was; an open job's log carries on from its next line. Build without it with cargo build -p runnerd-cli --no-default-features.

From the bash ci tool:

Config

One file, runnerd.toml, lists the repositories to serve, the runner profile each one uses, the shared caches and the systemd slice whose limits form the budget. examples/runnerd.toml is a complete example.

runnerd check-config runnerd.toml

This reads the slice's MemoryMax from systemd and fails when slots × memory over all profiles, plus the Docker Hub mirror, does not fit in it. A slice without a unit file has no MemoryMax; the check passes with a warning, since nothing would hold the runners to their caps. To check against a size without systemd, pass it: --memory-max 33G.

Sizes take a binary unit (K, M, G, T), as Docker and systemd use them.

isolation = "unsafe-dind" runs privileged containers, so a job can take over the host. It is refused unless the [host] table also sets allow_unsafe = true.

Building

cargo build --release

The toolchain is pinned in rust-toolchain.toml. Before sending a change, run the checks listed under "The gate" in AGENTS.md; CI runs the same. CONTRIBUTING.md explains which CI jobs a change triggers, and ARCHITECTURE.md where the code lives.

The tests that start real containers are ignored by default. They need Docker with sysbox and the images:

docker build --target jvm -t runnerd-runner:jvm image
docker build --target base -t runnerd-runner:base image
cargo test -p runnerd-core --test docker_smoke --test cache_reseed -- --ignored

examples/sandbox.toml is the setup for the end-to-end test against the throwaway repository ayagmar/runnerd-sandbox.

License

MIT