1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
# Continuous performance tracking via CodSpeed (https://codspeed.io).
#
# Instrument: WALLTIME (real elapsed time) on CodSpeed *macro runners*
# (`runs-on: codspeed-macro`) — bare-metal, low-noise, isolated. This is the
# accurate signal that CAN gate perf decisions: it reflects real time, so it
# captures the memory-bandwidth and multi-thread effects the CPU-simulation
# instrument misses (simulation would have shown the null Phase-7 SIMD change as
# a "win" — see docs/perf-experiments.md Experiment 10).
#
# Macro runners require a GitHub *organization* (not a personal account) and draw
# from the org's monthly macro-runner minutes. To fall back to the free,
# always-on CPU-simulation instrument, set `runs-on: ubuntu-latest` and
# `mode: simulation` (a deterministic *complexity* tripwire, not a wall-clock
# verdict). Even with walltime, the local best-of-N protocol in CLAUDE.md remains
# the release-gating authority; CI walltime is the continuous regression signal.
name: CodSpeed
on:
push:
branches:
pull_request:
workflow_dispatch:
jobs:
benchmarks:
name: Run benchmarks
runs-on: codspeed-macro
permissions:
contents: read
id-token: write # OIDC auth to CodSpeed — no CODSPEED_TOKEN secret needed
steps:
- uses: actions/checkout@v4
- name: Set up Rust toolchain, cache, and cargo-codspeed
uses: moonrepo/setup-rust@v0
with:
channel: stable
cache-target: release
bins: cargo-codspeed
# `--all-features` satisfies the benches' required-features
# (dequant: gptq/awq/bnb/gguf; parsing: npz/pth/gguf).
- name: Build the benchmark targets
run: cargo codspeed build --all-features
- name: Run the benchmarks
uses: CodSpeedHQ/action@v4
with:
mode: walltime
run: cargo codspeed run