# Testing Guide — youtube-legend-cli
- A categorized test suite that mirrors the provider pipeline
- Languages: [English](TESTING.md) | [Português Brasileiro](TESTING.pt-BR.md)
## Why Categorized Tests
- The crate ships one binary, one library, three examples and one benchmark
- The examples are `single_url`, `batch` and `json_output`, under `examples/`
- The benchmark is `cache_bench`, under `benches/`
- The tests are split into four categories that match the build and runtime surface
- UNIT TESTS live in `#[cfg(test)]` modules inside `src/`, are fast and deterministic, touch no network, and run under `cargo test --lib`
- DOC TESTS live in the rustdoc comments and run under `cargo test --doc`, which compiles and executes the blocks in `///` and `//!`
- INTEGRATION TESTS live under `tests/integration/`, cover the cross-crate surface, may need `wiremock`, and run under `cargo test --test <name>`
- BENCHMARKS live under `benches/`, are Criterion micro-benchmarks for the hot path, and run under `cargo bench --bench cache_bench`
- The split exists because the failure modes differ
- A unit test that hits the network is a flake waiting to happen
- An integration test that runs on every `cargo test` slows every local run
- Categorizing makes the trade-off explicit instead of accidental
## Test Categories
- Nine integration test targets are declared in `Cargo.toml` and live under `tests/integration/`
| `corpus` | PRD corpus against a `wiremock` upstream, plus one real-network OBSERVATION | No, except the observation | Yes, except the observation |
| `rss` | Enforces the NFR-002 RSS budget of 100 MiB | No | Yes, on Linux only |
| `offline_cache` | Cache hit round-trip with no network, for NFR-005 | No | Yes |
| `signal_handler_stress` | `SIGINT` and `SIGTERM` against a real spawned process | No | No, every case is `#[ignore]` |
| `cli_probing` | CLI flags and exit codes | No | Yes |
| `parse_properties` | `proptest` property tests over the `video_id` and `srv3` parsers | No | Yes |
| `envelope_snapshots` | `insta` snapshots of the JSON envelopes and of the config surface | No | Yes |
| `repo_invariants` | Repository invariants, including tracked `include_str!` paths and version agreement | No | Yes |
| `delivery_e2e` | The binary itself, argv to stdout, delivering subtitle bytes against a local upstream | No, mock | Yes |
- Nothing runs the gated cases for you, because this project has no CI
- You exercise them by hand with `cargo test --all-targets -- --ignored`
- In `corpus`, only `observes_the_real_upstream` stays behind `#[ignore]`
- It asserts OUR contract rather than the third party's mood: any exit code from this project's sysexits table passes, and a code outside it fails
- MEASURED on 2026-09-01: a 429 from the daily quota produced exit `69` with the cause named, which is the contract working and not a defect
- The other corpus cases run in the ordinary gate against a mock
- CORRECTED on 2026-09-04: `signal_handler_stress` is NOT gated to Linux
- Every case in it carries `#[ignore]`, and the two signal cases carry `#[cfg(unix)]`
- The practical reading is that they run on Linux and macOS when you ask for them, and compile away on Windows
- CORRECTED on 2026-09-04: `rss` IS gated to Linux, by a `#![cfg(target_os = "linux")]` at the top of the file
- The measurement reads `/proc/self/status`, which macOS and Windows do not have
- MEASURED on 2026-09-04 with `cargo test --all-features --locked` on macOS: the whole suite exited `0` and the `rss` target reported `0 passed; 0 failed; 0 ignored`
- Zero cases there is the `cfg` doing its job and not a defect, so read that line as NOT MEASURABLE on this host
- Run that command yourself before citing any per-target count here, because the counts move with every case somebody adds
## How to Run
### Unit tests
```bash
cargo test --lib
```
- This runs every `#[cfg(test)]` module in the library
### Doc tests
```bash
cargo test --doc
```
- This compiles and runs every code block in the rustdoc
### Integration tests, one by one
```bash
cargo test --test corpus
cargo test --test rss
cargo test --test offline_cache
cargo test --test signal_handler_stress
cargo test --test cli_probing
cargo test --test parse_properties
cargo test --test envelope_snapshots
cargo test --test repo_invariants
cargo test --test delivery_e2e
```
- Those nine names are the complete set, and they are the same nine the manifest declares
### Integration tests, all at once
```bash
cargo test --tests
```
- This skips doctests and lib tests, which is what you want while iterating on one suite
### Integration tests, including the gated ones
```bash
cargo test --test corpus -- --include-ignored
cargo test --test signal_handler_stress -- --include-ignored
```
- `--include-ignored` is the standard `cargo test` way to run `#[ignore]`-gated cases
- Nothing runs them for you, so invoke them yourself or sweep the whole set with `cargo test --all-targets -- --ignored`
### Examples
```bash
cargo run --example single_url
cargo run --example batch
cargo run --example json_output
```
- The examples are compiled by `cargo test --all-targets`, so a break in one of them fails the sweep
### Benchmarks
```bash
cargo bench --bench cache_bench
```
- `cache_bench` holds three Criterion micro-benchmarks, named `cache_path`, `srt_to_text` and `noteey_to_text`
- CORRECTED on 2026-09-04: this line used to describe a URL length check and a BCP 47 locale parser, and `benches/cache_bench.rs` benchmarks neither
- What it really measures is the cache path composer, the SRT-to-text parser and the noteey-shaped parser
- Confirm the target still compiles with `cargo bench --no-run`, because the full run happens only on demand
## Local Quality Gates
- CORRECTED on 2026-08-31: this section used to describe twelve jobs in a workflow file that does not exist
- The harm was concrete, not cosmetic: it announced an `audit` job, so nobody ran the tool locally, and the first run ever performed found two live advisories in the dependency tree
- This project ships no CI, and CI is forbidden in it
- Every gate below runs on your machine, and all of them must exit `0` before a change counts as done
- `cargo fmt --all --check` checks formatting
- `cargo clippy --all-targets -- -D warnings` turns lints into errors
- `cargo clippy --all-targets --features i18n-full -- -D warnings` is REGISTERED since 2026-08-31, because the default build is `default = []` and the optional locale catalogues compile in no other gate
- Without that line the catalogues are invisible to `check`, `clippy`, `test` and `doc`
- `cargo test --all-targets --no-fail-fast` runs the whole suite, and `--no-fail-fast` is not optional
- MEASURED on 2026-09-01: a run without it reported ONE failure, and the same tree with it reported that failure plus three further targets cargo had never reached
- A red target hides every target after it, and an audit that reads only the first one measures less than it claims
- `cargo doc --no-deps` must emit zero warnings, and not merely exit `0`
- `cargo doc --no-deps --document-private-items` is REGISTERED since 2026-09-01, because the gate above documents the public surface and a broken intra-doc link on a private item is invisible to it
- Run on this tree for the first time, that gate exited `101` on four separate links, one pointing at a symbol the crate no longer has and one at a method behind `#[cfg(test)]`
- `cargo audit` reads `Cargo.lock` and reports known advisories
- REGISTERED on 2026-09-04: this line used to be the whole control, and a control that runs only when somebody remembers is not a control
- The gate `repo_invariants::the_dependency_tree_carries_no_known_advisory` now runs it, behind `#[ignore]` because it fetches the advisory database over the network
- `cargo deny check` covers licences, bans and duplicate crates
- `cargo semver-checks check-release --baseline-rev <last tag>` covers public API compatibility, and it is a RELEASE-time measurement rather than a per-change gate
- MEASURED on 2026-09-04: it builds two whole copies of the crate and took 104.7 s on this tree, against 0.01 s for the invariant described next
- Read its verdict rather than assuming one, because on 2026-09-04 it answered `no semver update required` for `0.3.4 -> 0.4.0`, since under Cargo rules a `0.x` crate already breaks by growing its MINOR
- The per-change half of that control is `repo_invariants::public_surface_matches_the_record_and_removals_forced_a_breaking_bump`, which is offline and compares the tree against `docs/public-surface.txt`
- It exists because on 2026-09-04 four `pub mod` items were deleted and one of them, `provider::provider_noteey`, had been published at `v0.3.4`
- `cargo audit` reports crates that are locked without being compiled, and that is a feature, because enabling a feature can pull one of them into the real build
- Run the dependency gates on every dependency change
## Why the Ignored Sweep Is Not a Gate
- `cargo test --all-targets -- --ignored` is NOT a mandatory gate, and the reason is measured
- MEASURED on 2026-08-31: it exited `101` with three of four failing, and the SET that fails CHANGES between runs
- Those cases depended on a provider capped at five requests per day
- A gate whose green depends on someone else's quota can never be reliably green
- A flaky test teaches the reader to re-run until it passes, which is the opposite of a control
- Run the sweep as an OBSERVATION, and read a failure as information about the third-party service rather than as a regression here
## Configuration in the Tests
- The binary reads NO environment variable to govern its behaviour, and the test suite does not invent one either
- MEASURED on 2026-08-31: `TEST_INTEGRATION`, `WIREMOCK_PRINT_RESPONSES` and `YT_LEGEND_CACHE_DIR` have zero occurrences across `src/`, `tests/` and `benches/`
- `YT_LEGEND_NO_NETWORK` was REPLACED by the `--offline` flag, per the comment at `tests/integration/offline_cache.rs`
- `RUST_LOG` was removed on 2026-08-31 and is no longer read, so setting it changes nothing about CLI output
- Steer a test run the way you steer a real run: pass `--log-level`, `--log-format` and `--offline`, or point the binary at a scratch `config.toml` with `--config`
- That is exactly how the `corpus` suite aims the binary at its local `wiremock` upstream
- MEASURED on 2026-09-04: one environment variable IS set by the suite, and it is `XDG_CACHE_HOME`
- It is an OS path convention that `directories` reads to place the cache root, and it steers nothing about what the binary prints
- The gate `repo_invariants::no_integration_test_invokes_the_binary_without_isolating_xdg_cache` DEMANDS that override in every test that spawns the binary
- Without it the run writes a browser-profile directory into the real cache root of whoever ran the suite, and that directory outlives the process
## Troubleshooting
### Flaky `corpus` test, FIXED on 2026-08-31
- The corpus was intermittent because four parallel cases competed for the five-a-day quota of `provider-noiz`, and the set that failed CHANGED between runs
- All four now mount `wiremock` and point the binary at it with `--config`, so they run offline
- A failure there is now OURS and not the third party's, because none of the four touches the network
- The one exception is `observes_the_real_upstream`, which stays behind `#[ignore]` and exists to be READ rather than to be green
- If you do see a failure, run `cargo test --test corpus -- --ignored` with `--log-level trace` to see the full HTTP exchange
- The expected outcomes are a non-empty subtitle body for every URL, or `EX_NOINPUT` (`66`) for a URL whose captions YouTube removed
- Add that second kind to the skip list in `corpus.rs`
### `signal_handler_stress` looks like a no-op
- Every case in that target carries `#[ignore]`, so a plain `cargo test` reports zero tests run and that is correct
- Ask for them with `cargo test --test signal_handler_stress -- --include-ignored`
- The two signal cases carry `#[cfg(unix)]`, so they compile away on Windows and Windows signal behaviour is covered by nothing measured
### `cargo doc` reports broken intra-doc links
- Run `cargo doc --no-deps` and treat any warning as a failure, because zero warnings is the gate and exit `0` is not
- Run `cargo doc --no-deps --document-private-items` as well, because the public-surface pass never sees a private item
- The message names the offending `///` block, so fix the path or use a relative URL
### `cargo bench` aborts on missing Criterion
- The benchmark target needs `criterion` in `[dev-dependencies]`
- If a compile error mentions `criterion`, run `cargo build --benches` to force the dev dependency download
## Provider Tests
- The CLI has TWO providers, and `--provider` accepts both plus `auto`
- MEASURED on 2026-09-04: the `auto` chain walks both, in the order `provider-decopy` then `provider-noiz`
- Both speak plain HTTP, and neither launches a browser
- `provider-noiz` is capped at five requests per day, so no mandatory test may depend on it
- The `provider_getsubs_wiremock` and `provider_noteey_wiremock` tests were removed on 2026-09-04 along with the providers they exercised
- The `ProviderYouTubeDirect` tests from v0.3.0 went with that provider as well
- Unit tests for the parser, covering NFC normalisation and speaker marker cleanup, live in `src/parse/mod.rs` under `#[cfg(test)]`
## See Also
- [README](../README.md) — install and run
- [docs/ARCHITECTURE.md](ARCHITECTURE.md) — module map and provider pipeline
- [docs/CROSS_PLATFORM.md](CROSS_PLATFORM.md) — six declared cross-compile targets
- [docs/MIGRATION.md](MIGRATION.md) — what changed on the way to v0.4.0