# ADR-0034: Help-driven testing — anti-pattern and v0.1.20 corrective process
- **Status**: Accepted
- **Date**: 2026-06-15
- **Context**: The v0.1.19 audit on 2026-06-15 surfaced 11 GAP-2026-NNN items. Five of them (GAP-2026-001, 002, 005b, 006 plus the doc-drift variants 003 and 004) share a single root cause: flags were declared in the `clap` derive struct (and therefore rendered into `--help`) BEFORE the corresponding implementation landed. Once the help text advertised a feature, downstream documentation (`SKILL.md` EN+PT, `CLAUDE.md`, CHANGELOG entries, recipes) was synchronized with the design intent rather than the binary behavior. The flag then sat unimplemented for one or more releases while every help-consuming agent believed it worked. The chain `clap field → --help → SKILL doc → user expectation → silent failure` is now reproducible for any future flag. Without a process gate, the same pattern recurs on every new subcommand. This ADR formalizes the anti-pattern and codifies a help-driven testing rule that ties every visible flag to a regression test.
- **Decision**: Mandate help-vs-code co-design as a release gate. Specifically:
1. **No flag is added to `clap` derive struct without a paired integration test** asserting the binary behaves as the help text claims. The test name MUST include the flag (e.g. `count_by_size_top_n_returns_sorted`, `write_preserve_timestamps_keeps_mtime`, `edit_partial_multi_applies_matched_pairs`). A simple unit test is not enough — the test must invoke the binary as a subprocess, parse the NDJSON envelope, and assert on both the exit code and the JSON shape.
2. **Help text is auto-generated from clap and is therefore the API contract**. If the help says "Sort by file size (top N)", the envelope MUST contain `mode: "by_size"` when `--by-size` is passed. Discrepancies are release blockers and trigger an immediate revert of the help line until the test exists.
3. **Documentation (`SKILL.md`, `CLAUDE.md`, CHANGELOG`) is updated ONLY AFTER the test passes green in local scripts**. The v0.1.19 reverse pattern (write doc from design, defer impl) is forbidden. The `docs-rules/` and `skill/atomwrite-{en,pt}/SKILL.md` linters should emit a warning when a flag mentioned in docs does not appear in any `tests/cli_*.rs` file.
4. **Quarterly help-vs-reality audit**: a release-cadence battery walks every subcommand's `--help` output, dispatches each advertised flag against a stub stdin, and verifies the envelope matches. The 2026-06-15 audit found 11 drifts in a single pass; without this gate the next pass would find a similar number.
5. **Conformance guard for known good subcommands**: in v0.1.20, the resolver-first convention (ADR-0027) gained a textual assertion that `&args.target` appears exactly once in `write.rs`. A similar per-subcommand conformance test is now in scope for the "every flag has a test" rule.
- **Consequences**:
- **+** The 11 gaps closed in v0.1.20 each ship with at least one regression test that names the flag in the test ID, so a future regression of the same flag is caught at `cargo test` time, not by an end user.
- **+** Documentation drift is mechanically detectable: a `rg` scan for the flag name in `tests/cli_*.rs` produces a coverage table that the release checklist can require.
- **+** New contributors get a single, checkable rule ("if the help says X, the test must assert X"), eliminating the design-vs-impl race that produced GAP-2026-001 (the `count --by-size` shell game where the help advertised a `Vec<(PathBuf, u64)>` sort path that `cmd_count` never built).
- **+** The "documentation tells a lie" failure mode (gaps.md:610) is now a release-blocker, not a tolerated post-release finding.
- **-** Each new flag costs one test upfront, raising the line count of `tests/cli_*.rs` by ~15-40 lines per flag. Mitigated by the fact that a flagged behavior with no test was 100% of the v0.1.19 gap inventory — the test cost is the price of admission.
- **-** The rule applies retroactively only to flags added from v0.1.20 forward; existing flags (like the global `--json` which is accepted but ignored, ADR-style documented as a compat shim) remain grandfathered.
- **-** Auto-generated clap help text cannot distinguish "implemented" from "declared"; a separate linter would be needed to enforce. The linter is itself test-shaped and lives in `tests/cli_help_coverage.rs` rather than a separate binary.
- **Rationale**: The five help-driven drift cases share a fingerprint beyond the technical symptom. GAP-2026-001 had `pub by_size: bool` declared in `CountArgs` (`src/cli_args.rs:88`) but the `cmd_count` branch was never written (`src/commands/count.rs:108-127` shows the `if args.by_extension { ... } else { ... }` with no `by_size` arm). GAP-2026-002 had the atomic layer carry `preserve_timestamps: bool` (`src/atomic.rs` AtomicWriteOptions) but `WriteArgs` in `src/cli_args.rs` lacked the field — clap never rendered `--preserve-timestamps` for `write` because the field was not in the struct, but `edit` and `replace` did expose it. GAP-2026-005b had `--partial` mentioned in `SKILL.md` from the G117 design notes (ADR-0026) but the `EditArgs` struct never gained the field. GAP-2026-006 had `diff --algorithm` listed in the SKILL `diff` section but the flag was never added to `DiffArgs`. GAP-2026-003 had the `alias = "lang"` text rendered into help by clap's display logic but the field's real `long = "language"` collided with `GlobalArgs.lang` (ADR-0037 documents the fix). In all five cases the documentation described the design intent and the implementation followed the help, not the other way around. The audit's 5-Whys analysis (gaps.md:596-606) names the absence of help-driven testing as the systemic root cause. Codifying the rule turns a recurring ad-hoc finding into a single, auditable invariant.
- **Examples of failures this rule prevents**:
- **Count by-size**: a test named `count_by_size_top_n_returns_sorted` would have asserted `mode: "by_size"` and three entries; the absence of the test meant the flag could ship with no implementation and no detection until a user tried it on a real corpus.
- **Write preserve-timestamps**: a test named `write_preserve_timestamps_keeps_mtime` would have asserted the file's mtime was unchanged across the write; in v0.1.19 the field did not exist, so clap rejected the flag at parse time and the failure was visible only to the user.
- **Edit partial single-pair**: a test named `edit_partial_single_pair_returns_no_matches_exit_1` would have pinned the v0.1.20 decision (ADR-0036) that single-pair `--partial` is still `NO_MATCHES` rather than silent application of zero pairs. Without the test, a future refactor could "fix" the asymmetry and break the explicit semantic.
- **Diff algorithm**: a test named `diff_algorithm_patience_matches_default_output` would have asserted that `--algorithm patience` produces the same envelope as the default; its absence meant the flag was promised in docs but missing from clap.
- **Scope lang alias**: a test named `scope_lang_alias_matches_long_form` would have asserted that `atomwrite scope --lang rust` parses the same as `--language rust`; pre-ADR-0037 the test would have failed loudly, surfacing the `GlobalArgs.lang` namespace collision at code-review time rather than as a UX regression in the field.
- **Alternatives considered**:
1. Add a `clap` custom derive macro that requires a `#[help_test = "..."]` attribute pointing at a test file. Rejected: requires a new crate, complicates the `Cargo.toml`, and the macro can only verify the test exists, not that it asserts the right behavior.
2. Run a post-build binary that diffs help output against a "known good" snapshot. Rejected: brittle (help text changes are legitimate when the API does), high false-positive rate, and the snapshot itself becomes a maintenance burden.
3. Move the responsibility to a CODEOWNERS rule requiring a docs reviewer to sign off on every flag. Rejected: human review does not scale to 30 subcommands and produces inconsistent enforcement; the rule needs to be machine-checkable.
4. Keep the existing ad-hoc review process and rely on the quarterly audit. Rejected: the 2026-06-15 audit is exactly the quarterly audit running, and it still found 11 gaps. The audit detects the bug after release; the rule must prevent the bug at PR time.
- **Trigger to revisit**: If the help-coverage test (`tests/cli_help_coverage.rs`) flags false positives (a flag in help that is intentionally not actionable, like the `hide = true` `--json` compat shim), introduce a `#[arg(defer_test_to = "...")]` attribute or an allow-list file. If the rule slows down subcommand prototyping to the point of being skipped, consider a `#[arg(experimental)]` marker that auto-issues a warning at runtime and is exempt from the gate until the experimental phase ends.