# CI Debugging
A short guide for reading a failed GitHub Actions run, mapping the
failure back to a local repro, and triaging the most common
breakage classes.
## Where to Find the CI Config
The CI workflow is `.github/workflows/ci.yml`. There is exactly
one job: `fmt-and-build` (named "fmt, build, test,
observability"). It runs on `ubuntu-latest` with a 30-minute
timeout, and uses `concurrency: ci-${{ github.ref }}` so that a new
push to the same branch cancels the in-flight run.
The job runs, in order:
1. `cargo fmt --check`
2. `cargo build --offline` (with `|| cargo build` as a fallback
for the first run when the offline cache is empty)
3. `cargo test --offline` (with the same `|| cargo test` fallback)
4. `cargo run --offline --example claim_status_report` — and
asserts the output contains exactly 7 `claim_allowed=false`
and 1 `claim_allowed=true` lines.
5. `cargo run --offline --example submission_readiness` — and
asserts the verdict is `submission_readiness: NO` with 0
`[FAIL]` lines.
6. `bash -n scripts/regenerate_paper_artifacts.sh` — syntax
check on the artifact-regen script (does not run it).
## Reading a Failed Run
1. Open the failed workflow run in the GitHub Actions tab.
2. Click the `fmt-and-build` job.
3. The left sidebar lists each step; the failing step is
highlighted red. Click it to jump to the logs.
4. Read the last 30-50 lines of that step's log. The error
itself (the `error:` block from rustc, or the bash error line)
is almost always at the bottom.
## Mapping a CI Failure to a Local Repro
Each step in `.github/workflows/ci.yml` has a 1:1 local command:
| `cargo fmt --check` | `cargo fmt --check` |
| `cargo build --offline` | `cargo build --offline` |
| `cargo test --offline` | `cargo test --offline` |
| claim-status verification | `cargo run --offline --example claim_status_report` |
| submission-readiness verification | `cargo run --offline --example submission_readiness` |
| `bash -n scripts/regenerate_paper_artifacts.sh` | `bash -n scripts/regenerate_paper_artifacts.sh` |
If the local repro passes and CI still fails, check:
- the cached `target/` directory on the runner is from the
previous successful run on that branch — try
`cargo clean && cargo test --offline`.
- the toolchain pin — CI uses `dtolnay/rust-toolchain@stable`,
which is the same as the local default, but a `rust-toolchain`
override on your machine can desync.
## Most Common Breakage Classes
- **`cargo fmt --check` fails** — run `cargo fmt` locally, commit
the result, push. There is no project `rustfmt.toml`; the
default toolchain fmt is the gate.
- **`cargo test --offline` fails on `paper_artifacts.rs`** — the
4 pre-existing content checks in that file assert
`target/paper-results/*` is up to date. Run
`cargo run --offline --example paper_benchmarks` to regenerate,
then re-run tests.
- **claim-status check fails (not 7/1)** — a `CLAIMS.md` flag was
flipped. This is intentional only if the change is the P-number
that unblocks the flag. Otherwise revert the `=true` -> `=false`
and re-run.
- **submission-readiness check fails (`submission_readiness: NO`
is missing)** — the example printed a different verdict,
usually because one of the 9 artifact checks failed. Scroll
back in the log to find the `[FAIL]` line.
- **bash syntax check fails** — usually a stray quote or
unmatched `()` in `scripts/regenerate_paper_artifacts.sh`. Run
`bash -n scripts/regenerate_paper_artifacts.sh` locally to see
the line number.
## When the CI Cache Lies
The CI uses `actions/cache@v4` keyed on
`${{ runner.os }}-cargo-${{ hashFiles('Cargo.lock') }}`. If you
just merged a `Cargo.toml` change that altered the dependency
graph, the cache may briefly hold a stale `target/`. The
`cancel-in-progress: true` on the concurrency group handles most
of this, but if a flaky failure looks unrelated to your change,
`cargo clean && cargo test --offline` locally is the fastest
disambiguation.
## When All Else Fails
1. Reproduce locally with the exact commands above.
2. If the local repro is green, push a no-op commit (or empty
rebase) to retrigger CI and confirm it's not a cache flake.
3. If the local repro is red, bisect with `git bisect` against a
known-good commit on `main`. Each test file is independent
under `cargo test --test <name>`, so you can also narrow by
running the failing test file alone.