[Versão em Português Brasileiro](TESTING.pt-BR.md)
# Testing Guide
> Most of this suite runs on a machine with no GPU, which is exactly why it is worth running.
## Why Categorized Tests
- The contract has two halves, and only one of them needs a card.
- The argv gates, the exit codes, the JSON envelope, the payload reduction, the configuration cycle, the model catalog, and the process lifecycle are all provable without CUDA.
- Only the half whose success criterion is a real transcript needs an NVIDIA GPU, and that half is marked `#[ignore]` instead of silently skipped.
- Categorising by requirement rather than by module means a requirement without a test is a build failure, not an oversight someone notices later.
- The suite is organised so a contributor with no card can still prove that a change did not break the contract.
## Test Counts
- The suite carries 205 test functions in total, counted at the time this guide was written.
- 136 of them are integration tests that drive the shipped binary, spread across ten files under `tests/`.
- 69 of them are unit tests living beside the code in twelve modules under `src/`.
- 10 of them are marked `#[ignore]` because their success criterion needs a real NVIDIA GPU.
- Count them yourself rather than trusting this paragraph after a change.
```bash
rg -c '#\[test\]' tests/*.rs src/*.rs
rg -c '^\s*#\[ignore' tests/*.rs
```
## Test Categories
### Unit Tests in src
- `src/audio.rs` covers decoding, resampling to 16 kHz mono, silence chunking, hallucination filtering, and repeat collapsing.
- `src/catalog.rs` covers the five seeded model rows and the official digest of `small`.
- `src/commands.rs` covers which failures end the whole invocation and which stay per-target.
- `src/config.rs` covers the seeds, the accepted ranges, the unset behaviour, and the writable key inventory.
- `src/engine.rs` covers the language allowlist and the refusal when no engine feature is compiled.
- `src/error.rs` covers the exit codes, the categories, and the retryability rule.
- `src/gpu.rs` covers the VRAM gate, the model suggestion, and the device index check.
- `src/i18n.rs` covers locale resolution and the guarantee that both locale files expose identical keys.
- `src/model.rs` covers digest computation, presence, and verification.
- `src/paths.rs` covers path resolution from flags and from the platform directories.
- `src/signal.rs` covers shutdown classification.
- `src/surface.rs` covers every payload reduction knob and the rule that an error envelope survives them.
### Integration Tests in tests
- `tests/cli_surface.rs` drives the shipped binary across the whole agent contract.
- `tests/etd.rs` proves that a mutating verb never inherits ambient state and always names its target.
- `tests/fr011_gates.rs` proves that input validation runs before the resource gate, so exit codes 2, 64, 65, and 66 stay reachable without a card.
- `tests/fr015_models.rs` proves the model catalog contract inside a throwaway cache, and never opens a socket.
- `tests/fr025_config.rs` proves the persisted knob cycle inside a throwaway configuration directory.
- `tests/fr038_lifecycle.rs` proves the born-execute-die shape, the signal handling, the broken pipe, the absence of residue, and that stdout carries pure JSON.
- `tests/fr043_argv_surface.rs` proves the shape an agent sees, meaning help, discovery, schema, payload reduction, and locale.
- `tests/fr090_remaining.rs` carries the requirements with no home in the other suites, and holds most of the GPU-gated tests.
- `tests/fr_coverage.rs` reads the product requirements document and fails when a requirement has no test naming it.
- `tests/hardening.rs` carries regressions for defects that review found and execution confirmed.
### The Coverage Gate
- `tests/fr_coverage.rs` is the test that guards the other tests.
- It parses the requirements document, collects every requirement identifier mentioned anywhere under `tests/`, and fails when one is unnamed.
- A mention in a test name or in a comment counts, which is why every test file opens by citing the requirements it defends.
- A requirement that can only run on real hardware is expected to appear on an `#[ignore]` test, because the test must exist even when it cannot run here.
- Adding a requirement without covering it therefore breaks the suite immediately.
### GPU-Gated Tests
- Ten tests are marked `#[ignore]` because their criterion is a real transcript, real VRAM accounting, or real inference timing.
- They are written and compiled, not omitted, so the intent is auditable even where the hardware is absent.
- Run them deliberately on a machine with an NVIDIA GPU and a build that carries the engine.
## Verb Coverage
- Every one of the twenty-one leaf verbs is invoked by at least one integration test, so no verb ships unexercised.
- `transcribe` is covered across the transcript, batch, VAD and output-file files, and is the only verb that loads a model.
- `inspect` is covered by the media-header files, which assert container facts without any model present.
- `models list`, `models download`, `models path`, `models remove` and `models verify` are covered by the model catalog and cache files.
- `doctor` is covered by the host-readiness files, including the case where it reports and then fails.
- `config path`, `config list`, `config get`, `config set` and `config unset` are covered by the configuration files, which assert the full write-read-unset cycle.
- `schema` and `commands` are covered by `tests/documented_surface.rs`, which reads both out of the running binary rather than from a list kept beside the test.
- `version` is covered wherever a test needs the compiled engine, which is most of them.
- `completions` is covered by its own file, and is the ONE verb asserted to write something other than JSON.
- `licenses` is covered by the attribution file.
- `locale show`, `locale set` and `locale clear` are covered by the locale files, including the two-path contract where `--lang es` degrades and `config set locale es` refuses.
- Add a verb without adding a test and `documented_surface` fails, because it derives the list from the parser instead of repeating it.
## How to Run
### Run Everything That Works Without a GPU
- This is the default, and it is NOT the whole suite.
- `default = []` in `Cargo.toml`, so nothing behind a Cargo feature compiles here.
- Twenty-nine integration files carry at least one test behind `stub-engine`, and one carries a test behind `cpu`.
- No file is gated as a whole: the attribute sits on individual tests, so every file still compiles here and the gated tests simply do not exist in the binary.
- Run this to check that the default build stays green, then run the command below.
```bash
cargo test
```
### Run the Full Suite — the one to run on every change
- `stub-engine` replaces the inference call with a deterministic generator.
- It is what lets batch accounting, deadlines, segments and cold load execute on a host with no usable CUDA build.
- Without it, `cargo test` reports success while sixty-eight tests were never compiled into any binary, so there is nothing to report as skipped.
- Measured on 2026-08-28: the default build runs 347 tests and this one runs 415, across the same 47 test targets.
```bash
cargo test --features stub-engine -- --test-threads=1
```
- `--test-threads=1` matters: several tests fabricate the environment through `PATH`, and parallel threads share it.
- The gate `tests/fr_coverage.rs` records which requirements are proved ONLY behind a feature, and fails when that set changes.
### Run Only the Unit Tests
- Use this for a fast loop while editing a single module.
```bash
cargo test --lib
```
### Run One Integration File
- Use this when you are working on one area of the contract.
```bash
cargo test --test fr025_config
cargo test --test fr043_argv_surface
cargo test --test fr038_lifecycle
```
### Run One Test by Name
- Pass a substring of the test name.
```bash
cargo test unset_restores_seed
```
### Run the GPU-Gated Tests
- Do this only on a host with an NVIDIA GPU, a working driver, and the CUDA toolkit.
```bash
cargo test --features cuda -- --ignored
```
### Build the Engine Before Testing It
- The default build carries no inference engine, so a transcript test cannot pass without a feature.
```bash
cargo test --features cuda
cargo test --features cpu
```
### See the Test Names Without Running Them
- Use this to confirm what the suite actually contains.
```bash
cargo test -- --list
```
### Compile the Tests Without Executing Them
- Use this to catch a compilation break quickly.
```bash
cargo test --no-run
```
## Profiles
### Headless Profile
- Run `cargo test` with no features on a machine with no GPU.
- Expect every non-ignored test to pass, because none of them needs a card.
- This is the profile a contributor without NVIDIA hardware uses.
### GPU Profile
- Run `cargo test --features cuda` on a machine with a card, a driver, and the toolkit.
- Then run `cargo test --features cuda -- --ignored` to exercise the ten deferred tests.
- Expect the first transcript to be slower, because it pays the model load.
### Feature Matrix Profile
- Build and test with no feature to confirm the engine refusal path stays reachable.
- Build and test with `cpu` to confirm the engine compiles without CUDA.
- Build and test with `cuda` to confirm the native path.
```bash
cargo test
cargo test --features cpu
cargo test --features cuda
```
## Environment Variables
- The product reads no environment variable of its own, so no variable configures the behaviour under test.
- Isolation is achieved through `--config-dir` and `--cache-dir`, which every integration test passes so it can never touch the operator's real configuration or model cache.
- The variables that appear in the test sources belong to the harness or to the operating system, never to the product.
- `CARGO_MANIFEST_DIR` is used by the coverage gate to locate the requirements document.
- `CARGO_BIN_EXE` is what `assert_cmd` resolves internally to find the freshly built binary.
- `PATH` is manipulated in `tests/hardening.rs` to control which external tools the child process can discover.
- `CUDA_VISIBLE_DEVICES` is set on the child process in `tests/fr090_remaining.rs` for the sole purpose of proving that it changes nothing on stdout.
- Never introduce a product environment variable to make a test pass, because that would contradict the contract the suite exists to defend.
## Troubleshooting
### A transcript test fails with exit code 70
- The build carries no engine, and the message is `native engine not compiled`.
- Rebuild with `--features cuda` or `--features cpu` before expecting a transcript.
### A transcript test fails with exit code 69
- The host has no CUDA toolkit, meaning `nvcc` is not on `PATH`.
- Install the toolkit, or accept that this test belongs to the GPU profile.
### doctor exits 70 during a test run
- This is the expected outcome on a host with no NVIDIA GPU, and the test asserts it.
- Confirm the host with `whisper-nvidia-cli doctor` before assuming a regression.
### The coverage gate fails after adding a feature
- A requirement identifier now exists with no test naming it.
- Read the failure message, which lists the missing identifiers, and add a test whose name or comment cites each one.
- Use `#[ignore]` when the criterion truly needs hardware, but never omit the test.
### A test writes into your real configuration
- That is a bug in the test, not in the product.
- Every integration test must pass `--config-dir` and `--cache-dir` pointing at a temporary directory.
### A residue assertion fails
- The lifecycle suite asserts that nothing survives the process.
- Look for a temporary path created without being registered for cleanup, since a forced exit skips destructors.
### The suite is slow on the first run
- The first `cargo test` compiles the binary and every integration target.
- Later runs reuse the build, and `cargo test --lib` skips the integration targets entirely.
## See Also
- [AGENTS.md](AGENTS.md) for the contract the suite defends.
- [HOW_TO_USE.md](HOW_TO_USE.md) for the guided walkthrough.
- [CROSS_PLATFORM.md](CROSS_PLATFORM.md) for what each platform requires.