runnerctl 0.1.1

Manage independent ephemeral GitHub Actions runner pools on one Docker host
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170

# runnerctl

[![crates.io](https://img.shields.io/crates/v/runnerctl.svg)](https://crates.io/crates/runnerctl)
[![CI](https://github.com/clroot/runnerctl/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/clroot/runnerctl/actions/workflows/ci.yml)
[![Rust](https://img.shields.io/badge/Rust-1.85%2B-orange.svg)](https://www.rust-lang.org)
[![License](https://img.shields.io/crates/l/runnerctl.svg)](https://opensource.org/licenses/MIT)

A Rust CLI for managing multiple GitHub Actions runner pools on a single Linux Docker host. Configure each pool's organization, authentication profile, labels, image, and capacity. The manager runs the official GitHub runner in ephemeral containers, removes containers after their jobs finish, and creates replacements.

## Requirements and installation

Install and run `runnerctl` on the Linux host that will run your GitHub Actions runners. You need:

- Rust and Cargo (Rust 1.85 or later; the current stable toolchain is recommended). Install them with [rustup](https://rustup.rs) if needed.
- A C compiler and linker for building native dependencies during installation.
- Docker Engine and CLI, with Docker running and your user able to access its socket.

A systemd user service is optional.

### Install from crates.io

Install the latest published release from [crates.io](https://crates.io/crates/runnerctl):

```bash
cargo install runnerctl --locked
runnerctl --version
```

Cargo installs the binary into `~/.cargo/bin` by default. Make sure that directory (or `$CARGO_HOME/bin` if customized) is on your `PATH`. The `--locked` flag uses the release's pinned dependency versions. Run the same install command again to update to a newer release.

### Build the runner image

The CLI installation does not build the Docker image used by runner containers. Before continuing to the quick start, build the default image on the Docker host used by `runnerctl`. Docker can fetch the build files directly from the `v0.1.1` release:

```bash
docker build -t local/runnerctl-runner:2.337.0 \
  'https://github.com/clroot/runnerctl.git#v0.1.1:runner'
```

This tag matches the CLI's default pool image. The image build verifies the runner archive's SHA-256 checksum. To build another runner version, supply both the `RUNNER_VERSION` and architecture-specific `RUNNER_SHA256` build arguments, and use a matching image tag. Custom images must follow the image contract below.

## Quick start

```bash
runnerctl init --org '<YOUR_ORG>'
runnerctl auth login --profile '<YOUR_ORG>'

runnerctl pool add validate --org '<YOUR_ORG>' --auth '<YOUR_ORG>' \
  --labels validate --replicas 4
runnerctl pool add build --org '<YOUR_ORG>' --auth '<YOUR_ORG>' \
  --labels build --replicas 2 --docker-socket

runnerctl doctor
runnerctl service install
runnerctl start --all
runnerctl status
```

PAT input is hidden at the terminal. For automation, use `auth login --profile NAME --token-stdin` or `--env VARIABLE`. Do not pass tokens as command arguments. Configuration stores only credential file or environment variable references. Credential files use mode `0600`, management directories use `0700`, and runner containers receive only short-lived registration tokens.

A fine-grained PAT needs the target organization's **Self-hosted runners: Read and write** permission and any approval required by organization policy. `doctor` checks runner read access, local image availability, and the observed health of running pools (errors, retry delays, unknown runners, and unavailable capacity). Intentionally stopped or zero-replica pools are not marked degraded; registration write access is checked when creating a runner. GitHub App authentication is planned for a later release.

To run without a service, use `runnerctl manager` in the foreground. The CLI and manager must use the same `--home` directory. The default is `$XDG_CONFIG_HOME/runnerctl` or `~/.config/runnerctl`. When the manager is stopped, CLI configuration and intent changes are still saved under an exclusive lock, but container provisioning requires the manager to be running.

File credentials are recommended for systemd because user services do not automatically inherit shell environment variables. To keep the service running without an active login, configure `sudo loginctl enable-linger "$USER"`. `service install` creates a unit pointing to the current binary and enables and starts it. Reinstall the service if you move the binary.

## Managing multiple pools

```bash
runnerctl pool list
runnerctl pool show build
runnerctl status --pool build
runnerctl --json status
runnerctl scale build 3
runnerctl stop build
runnerctl start build
runnerctl logs <runner-id> --pool build --follow
runnerctl upgrade --pool build --image local/runnerctl-runner:2.337.0
runnerctl pool remove build
```

`replicas` is the pool's concurrent capacity, including busy runners. New pools are **stopped by default**. `scale` updates the desired capacity without starting a stopped pool. Starting or stopping every pool requires `--all`.

Stopping, scaling down, deleting, and updating pools limit new provisioning and wait for existing jobs to finish naturally. **Idle runners may remain until they receive and complete their next job.** Checking GitHub's busy flag cannot eliminate the race between job assignment and container termination, so an idle observation alone does not trigger forced termination. These runners are marked `draining`; elapsed time does not automatically cancel work. Use `stop build --force` when immediate termination is necessary; this can interrupt running jobs. After a forced stop, remote registration cleanup may remain pending until GitHub connectivity is restored.

Stopping the manager process does not stop running containers. On restart, the manager recovers from SQLite and container ownership labels. Recreating a pool with the same name assigns a new UUID, distinguishing it from the previous pool. Containers belonging to another manager or an existing Compose deployment are left untouched.

To route jobs to a specific pool within an organization, include its automatically registered pool label:

```yaml
jobs:
  validate:
    runs-on: [self-hosted, 'pool:validate', validate]
    steps:
      - uses: actions/checkout@v4
      - run: echo 'validation job'
```

Pool names are not access-control boundaries. Configure runner group repository access policies in GitHub. Sharing the host Docker socket is intended for trusted workflows and does not provide security isolation between pools. The default image includes the Docker CLI for Docker builds in shell steps. Workflows using `jobs.<job>.container`, service containers, or host bind mounts need separate validation of their workspace path mappings.

## Configuration

See [config.example.toml](config.example.toml) for three pools across two organizations.

```bash
runnerctl config check
runnerctl config apply
# Use optimistic concurrency control in external automation.
runnerctl config apply --revision 3
```

The sum of all configured replicas, including stopped pools, must not exceed `manager.max_runners`. Creation reservations and containers awaiting cleanup also count toward actual capacity. The current implementation rotates through pools and creates **one runner at a time**, staying within `max_parallel_creates`. Increasing that setting does not yet enable parallel creation.

Manual file edits take effect only after `config apply`. CLI configuration changes are rejected while unapplied edits exist, preventing accidental overwrites. Delete pools with `pool remove`. Changing a pool's organization or authentication profile requires the pool to be stopped and its previous registrations fully cleaned up. Image, label, and runner group changes apply to a new generation of runners.

SQLite is the source of truth for applied configuration. If the process exits after saving the database but before writing TOML, the file may differ from the applied state. Compare the applied configuration and revision from `--json status` with the file, then recover by applying the desired file contents with `config apply`. Saved credential files are not automatically deleted. After rotating a token, check that an old file is no longer referenced before removing it.

## Failures and logs

- GitHub and Docker are polled periodically. Docker event streaming is not yet implemented.
- API observation failures are reported as `unknown` and do not cause running containers to be deleted.
- Authentication errors and rate limits are shared within an authentication profile. Failed pools retry with bounded exponential backoff and jitter (at most 600 seconds). After runner exits, provisioning pauses during that delay while observation and cleanup continue; dependency failures pause reconciliation to avoid repeatedly hitting a failing Docker daemon or API. Retry scope and deadlines survive manager restarts.
- A runner that exits without completing a job is recorded as a failure. All exited runners found in a reconciliation pass are cleaned before replacement is attempted. Runners started before the current host boot were stopped by the reboot, so they are cleaned and replaced immediately without being recorded as failures. Normal completion requires both the completion hook and container exit.
- Replacement containers receive new names and fresh filesystems, avoiding registration state left by container restarts.
- A live but unresponsive process may remain `unknown`. Inspect its state and use an explicit `stop --force` followed by `start` if necessary.
- Before deleting a container, the manager saves recent logs, up to 1 MiB, and a `_diag` archive, up to 32 MiB, in `<home>/logs`. Archive failure delays deletion and reports an error.
- Docker logs are limited to three 10 MiB files. Archived files expire after seven days by default; retention cleanup runs during successful reconciliation of an active pool.
- `logs --follow` polls the selected container and ends when that container is replaced. Previous logs remain in the archive directory.
- Rolling updates replace runners after their jobs finish, without adding extra capacity. Startup failures trigger backoff for the affected pool while existing jobs continue.

Automatic runner updates are disabled; updates are delivered through images. Automatic tracking of runner version support deadlines is not yet implemented. Monitor GitHub's update requirements and rebuild images accordingly.

## Image contract

The default Dockerfile carries forward the Ubuntu, Git, and Docker CLI setup from the existing deployment. `/entrypoint.sh` starts as root to configure Docker socket group access, consumes `/run/runnerctl/token`, and then runs `config.sh --ephemeral --disableupdate` and `run.sh` as a regular user. The PAT is not injected into this container.

Custom images must handle `GITHUB_ORG`, `RUNNER_NAME`, `RUNNER_LABELS`, and `RUNNER_GROUP`, consume and remove `/run/runnerctl/token` during registration, and exit after one job. The completion hook must create `/run/runnerctl/completed`, and diagnostic files must be written to `/home/runner/actions-runner/_diag`. Extend `runner/Dockerfile` with additional build tools when possible.

## Testing and migration

For development, clone the repository and install the local checkout instead of the published crate:

```bash
git clone https://github.com/clroot/runnerctl.git
cd runnerctl
cargo install --path . --locked
docker build -t local/runnerctl-runner:2.337.0 runner
```

Run the checks from the repository root:

```bash
cargo fmt --check
cargo clippy --all-targets --locked -- -D warnings
cargo test --locked
```

See [tests/README.md](tests/README.md) for isolated tests using real Docker containers. These tests create and clean up only dedicated containers labeled with a randomly generated manager ID. They do not register runners with a real organization or execute GitHub workflows.

Start migration with one test runner using a new label. Verify your actual workflows and Docker usage, then stop existing Compose slots individually and increase the new pool's capacity. Adjust the existing recovery timer's management scope so it cannot recreate migrated runners, and disable it once migration is complete.

## Publishing a release

Maintainers commit the desired version in `Cargo.toml` and `Cargo.lock`, then create and publish a GitHub release for the matching `v<version>` tag. Publishing the release triggers the repository workflow, which checks the tag against `Cargo.toml`, runs formatting, clippy, tests and a publication dry run, then uploads the crate to crates.io. The repository's `CARGO_REGISTRY_TOKEN` secret is already configured; prerelease GitHub releases are rejected.

## References

- [GitHub runner API and permissions](https://docs.github.com/en/rest/actions/self-hosted-runners)
- [Ephemeral runners and update policies](https://docs.github.com/en/actions/reference/runners/self-hosted-runners)
- [Job completion hooks](https://docs.github.com/en/actions/how-tos/manage-runners/self-hosted-runners/run-scripts)