1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
name: mutants
# Mutation testing in two modes, because the single nightly this replaces
# produced nothing at all.
#
# Its one run in history was CANCELLED at the GitHub Actions six-hour
# ceiling: no `mutants.toml`, no `[package.metadata.mutants]`, so it
# enumerated mutants across the whole tree with a rebuild each and could
# never finish. `continue-on-error: true` meant it never went red either,
# and the artifact step uploaded `mutants.out/` from a killed run. So the
# signal this project explicitly chose OVER coverage-as-a-gate — on the
# reasoning that coverage answers "what is untested" and this codebase is
# past that, into "are the tests load-bearing" — was emitting nothing.
#
# Worth automating at all because the hand process fails in ways that look
# like success: a `sed` written against the pre-`fmt` shape silently
# no-ops and the test "passes" a mutation never applied; a mutation that
# doesn't compile prints no `test result:` line and reads as a pass; and a
# test filter matching zero tests reports `ok. 0 passed`.
# `scripts/mutate.sh` guards a single experiment against all three.
on:
pull_request:
schedule:
- cron: "0 3 * * *"
workflow_dispatch:
permissions:
contents: read
jobs:
# The one that matters. `--in-diff` mutates only the lines this PR
# touched, which is minutes rather than hours, and asks exactly the
# question worth asking at review time: is the test that came with this
# change load-bearing? Blocking, because on a diff-sized surface a
# surviving mutant is nearly always a real gap rather than an
# equivalent-mutant argument.
in-diff:
name: cargo-mutants (changed lines)
if: github.event_name == 'pull_request'
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- uses: actions/checkout@v7
with:
fetch-depth: 0
- uses: dtolnay/rust-toolchain@stable
- uses: Swatinem/rust-cache@v2
- run: cargo install cargo-mutants --locked
- name: Diff against the merge base
run: |
set -euo pipefail
base="${{ github.event.pull_request.base.sha }}"
git diff "$base" -- 'src/**/*.rs' > pr.diff
wc -l < pr.diff | xargs echo "diff lines:"
- name: Mutate the changed lines
run: |
set -euo pipefail
# No Rust changes in this PR — nothing to mutate, and an empty
# diff must not read as a pass.
if [ ! -s pr.diff ]; then
echo "no src/**/*.rs changes; nothing to mutate"
exit 0
fi
cargo mutants --in-diff pr.diff --no-shuffle --timeout 120 -- --lib
- uses: actions/upload-artifact@v7
if: always()
with:
name: mutants-in-diff
path: mutants.out/
retention-days: 14
# Whole-tree sweep, sharded TWENTY-FOUR ways. Third number, and the
# first one derived from a completed run rather than an estimate.
#
# The tree holds 6097 mutants (about 6040 after `--exclude src/main.rs`). Guessing the runner's speed got this
# wrong twice: 8 shards did not fit, and 16 only fit for 5 of them —
# the other 11 were cancelled at the cap having got 81-99% of the way
# through, which is the most annoying possible outcome.
#
# Measured from that run instead of estimated. The SLOWEST shard
# managed 309 mutants in 320 minutes: 0.97 mutants/min. At that rate a
# 16th of the tree needs 395 minutes, so no plausible cap rescues 16
# shards. 24 shards is 254 mutants each, ~263 minutes — real headroom
# under a 350 cap rather than the 20 minutes that turned out to be
# noise.
#
# Prefer more shards over a longer cap: GitHub kills a job at 6h no
# matter what this says, so headroom has to come from doing less work
# per shard.
#
# Re-measure before changing this. `cargo mutants --list | wc -l` gives
# the count; the per-mutant cost is dominated by the rebuild, so it
# scales with compile time rather than test count. Non-blocking:
# across the full surface, surviving mutants are a reading list, and
# turning them into a build failure means either chasing every
# equivalent mutant or switching the job off — and the second is what
# actually happens.
full:
name: cargo-mutants (shard ${{ matrix.shard }}/24)
if: github.event_name != 'pull_request'
runs-on: ubuntu-latest
# 350, not 320. The first real run's shards finished at 298-300
# minutes — 20 minutes of headroom on a 320 cap, which is close
# enough that a marginally slower runner tips a shard into a
# timeout, and a timed-out shard reports nothing useful. Measured,
# not guessed: see the shard-count note above.
timeout-minutes: 350
continue-on-error: true
strategy:
fail-fast: false
matrix:
shard:
steps:
- uses: actions/checkout@v7
- uses: dtolnay/rust-toolchain@stable
- uses: Swatinem/rust-cache@v2
- run: cargo install cargo-mutants --locked
- name: Mutate this shard
# `--exclude src/main.rs`: the test command below is `-- --lib`,
# which does not compile the binary target at all — so every
# main.rs mutant is reported MISSED regardless of coverage, and
# 57 of them were, in the first real run. They are noise, and
# they cost compute to produce.
#
# main.rs is not untested: `tests/cli.rs` drives the real binary
# and its `every_advertised_subcommand_routes_somewhere` catches
# exactly the "delete match arm" mutants this was reporting —
# verified by hand. Covering it here instead would mean
# `--all-targets`, which recompiles and re-runs the process-level
# suite for every mutant in the tree.
run: >
cargo mutants --shard ${{ matrix.shard }}/24
--exclude src/main.rs
--no-shuffle --timeout 120 -- --lib
- uses: actions/upload-artifact@v7
if: always()
with:
name: mutants-shard-${{ matrix.shard }}
path: mutants.out/
retention-days: 14