# How cargo-gamma works
This document describes the architecture of cargo-gamma: the model that makes it fast, the stages
of a mutation campaign, the boundaries that preserve correctness, and the trade-offs the design
makes. It intentionally does not describe source modules, internal helper types, test fixtures, or
how to extend the implementation.
Those mechanics are recorded in the [implementation guide](IMPLEMENTATION.md).
For commands, configuration, mutator names, and operational advice, see
[the README](../README.md).
## Contents
- [The architectural idea](#the-architectural-idea)
- [System boundaries](#system-boundaries)
- [A campaign from start to finish](#a-campaign-from-start-to-finish)
- [Discovering the campaign](#discovering-the-campaign)
- [Representing every mutant in one program](#representing-every-mutant-in-one-program)
- [Building a viable schema](#building-a-viable-schema)
- [The scratch workspace](#the-scratch-workspace)
- [Establishing the oracle](#establishing-the-oracle)
- [Selecting only tests that can matter](#selecting-only-tests-that-can-matter)
- [Executing mutants safely](#executing-mutants-safely)
- [Identity and knowledge across campaigns](#identity-and-knowledge-across-campaigns)
- [Verdicts and scoring](#verdicts-and-scoring)
- [Reports and integrations](#reports-and-integrations)
- [Correctness principles](#correctness-principles)
- [Costs and limitations](#costs-and-limitations)
## The architectural idea
Mutation testing asks whether a test suite would detect small defects. A conventional mutation
tester changes one location, recompiles, runs tests, restores the source, and repeats:
```text
total cost ≈ mutants × (build + tests)
```
That model becomes impractical when a workspace has thousands of mutants and compilation takes
minutes.
cargo-gamma instead compiles all selected mutants into one set of test binaries. Each mutation is
placed behind a runtime guard, and a process-local selector activates exactly one guard:
```text
total cost ≈ one instrumented build + mutants × selected tests
```
This representation is a **mutant schema**. It removes compilation from the per-mutant loop. The
remaining problem is to minimize the tests and process time needed to decide each verdict without
changing the verdict itself.
```mermaid
flowchart LR
subgraph Conventional["Conventional mutation testing"]
C1[Edit one mutant] --> C2[Build]
C2 --> C3[Run tests]
C3 --> C4[Restore]
C4 --> C1
end
subgraph Gamma["cargo-gamma"]
G1[Discover all mutants] --> G2[Build one mutant schema]
G2 --> G3[Activate next mutant]
G3 --> G4[Run relevant tests]
G4 --> G3
end
```
The design therefore optimizes each component of campaign cost:
1. **Fixed campaign cost:** discovery, copying, instrumentation, and one build.
2. **Per-mutant cost:** process launch plus the tests capable of observing that mutant.
3. **Failure cost:** extra time needed to classify compilation failures, hangs, enumeration failures,
and excessive memory use without turning them into false detections.
## System boundaries
The architecture separates responsibilities into the following domains.
The CLI builds on macOS with the system allocator. Linux and Windows use the
`rallocator` integration; that dependency is available only on those hosts.
```mermaid
flowchart TB
User[CLI and configuration] --> Coordinator[Campaign coordinator]
Coordinator --> Engine[Source and mutation engine]
Coordinator --> Workspace[Scratch workspace and Cargo]
Coordinator --> Supervisor[Process supervisor<br/>and safe platform adapters]
Engine --> Schema[Instrumented mutant schema]
Schema --> Workspace
Workspace --> Binaries[Test binaries]
Binaries --> Supervisor
Supervisor --> Verdicts[Verdicts and measurements]
Verdicts --> Coordinator
Coordinator --> Reports[Console, JSON, HTML, SARIF, CI]
Runtime[Dependency-free guard runtime] -. vendored into .-> Workspace
```
### Campaign coordinator
The coordinator owns policy: command and configuration precedence, package and test selection,
incremental reuse, scheduling, scoring, and reporting. It combines evidence from the other domains
but does not parse Rust or implement operating-system containment itself.
The command-line surface follows the same boundary: each subcommand exposes only settings consumed
by that operation. Run-only controls such as progress are not global. `list` separates population
and registry modes so `list mutants`, `list files`, `list mutators`, and `list presets` each
document only their applicable selection and output controls; bare `list` remains shorthand for
`list mutants`.
### Source and mutation engine
The engine discovers mutation sites, evaluates source-level suppression, gives sites stable
identities, and emits instrumented source. It is deterministic for a given source and mutation
selection. Each candidate records whether source-visible evidence proves the replacement,
whether it is an unresolved optimistic guess, or whether the user explicitly supplied it. The
classification is diagnostic evidence rather than an automatic filter. The ordinary default
withholds only measured noisy optimistic site classes; a non-default selector that includes the
applicable mutator admits them. `explain` and `list mutators` expose which mutators apply this
explicit-selection rule. The engine does not know about Cargo processes, test verdicts, timeouts,
or reports.
### Scratch workspace and Cargo
The workspace boundary keeps all rewriting away from the checkout. It prepares a buildable copy,
injects the guard runtime, preserves Cargo's view of path dependencies and configuration, and owns
the build artifacts reused during the campaign.
### Process supervisor
The supervisor launches one contained process tree, observes output and resource use, and guarantees
that termination reaches descendants. Campaign policy decides whether the observed process means
pass, kill, timeout, stall, or memory exhaustion; the supervisor provides the race-sensitive
mechanism.
### Guard runtime
The injected runtime is intentionally tiny, dependency-free, and independent of cargo-gamma's own
dependency graph. Adding dependencies, features, or a build script to it could perturb feature
unification in the workspace under test and change what the tests prove.
Every instrumented package is linked to the same runtime vendored for the campaign. An existing
dependency on cargo-gamma's implementation crate is redirected to that copy; an unrelated
dependency occupying the `gamma_rt` crate name is refused. This keeps every guard in one test
process on the same active-mutant and reach-observation state. Redirecting an existing dependency preserves
its `features` and `default-features` settings — including workspace-level declarations, member
overrides of `workspace = true`, and target-specific declarations — so a package that already opted
into a runtime feature keeps that selection after the redirect.
## A campaign from start to finish
A campaign is an ordered evidence pipeline. Later stages depend on facts established by earlier
ones; they are not interchangeable background jobs.
```mermaid
sequenceDiagram
actor User
participant Gamma as cargo-gamma
participant Cargo
participant Schema as Scratch schema
participant Tests as Test processes
User->>Gamma: Run campaign
Gamma->>Cargo: Read workspace metadata
Gamma->>Gamma: Discover scope, cfg, mutants, and suppressions
Gamma->>Gamma: Validate reusable campaign knowledge
Gamma->>Schema: Synchronize scratch workspace
Gamma->>Schema: Instrument selected mutants
loop Until the schema compiles
Gamma->>Cargo: Build all instrumented test targets
Cargo-->>Gamma: Structured diagnostics
Gamma->>Schema: Withdraw blamed mutants
end
Gamma->>Tests: Run unmutated baseline
Tests-->>Gamma: Timing, output, and memory evidence
Note over Gamma,Tests: Reach observations accompany ordinary mutant launches
loop One active mutant per process
Gamma->>Tests: Run relevant tests with mutant selected
Tests-->>Gamma: Verdict and measurements
end
Gamma->>Gamma: Score and persist safe knowledge
Gamma-->>User: Reports and exit status
```
The visible phases correspond to these architectural stages:
- **Analyzing:** configuration, Cargo metadata, and discovery of the workspace source scope.
- **Copying:** synchronizing the checkout into the scratch workspace.
- **Mutating:** collecting candidates and rewriting every selected target package.
- **Building:** checking all instrumented test targets together, withdrawing compiler-rejected
mutants until the complete schema checks, then building the converged binaries once.
- **Baselining:** proving the unmutated test oracle is green and measuring its behavior.
- **Optimizing:** learning which tests reach which mutation sites.
- **Testing:** activating and judging the remaining mutants.
- **Reporting:** projecting one verdict set onto its output surfaces.
## Discovering the campaign
Discovery answers four questions before source is changed:
1. Which Cargo packages and targets are in scope?
2. Which source files are compiled under the selected features, target, profile, and Rust flags?
3. Which mutation sites and suppressions exist in those files?
4. Which test targets can link each mutated package?
Cargo metadata supplies package ownership, target roots, features, and dependency relationships.
The selected Cargo build settings also determine the active `cfg` predicates. Treating source that
the build never compiles as ordinary live code would create mutants no test could execute and
misreport them as test-suite failures.
The source engine parses Rust for structure and spans but rewrites the original bytes rather than
pretty-printing an AST. Textual rewriting preserves comments, formatting, macros, and literal
spelling. Byte-accurate spans are therefore part of the correctness model, not merely an
implementation choice.
Selection narrows the population before expensive work:
- package, file, mutator, and diff selection decide what may become a mutant;
- `--re` selects mutants by a regular expression over the full name shown by `list mutants`:
workspace-relative location, qualified function path, change description, and any suppression
marker. Repeated patterns
match any one of them; control characters are encoded as in the plain listing before matching.
The filter is applied before sharding and compilation. With `list files`,
it leaves only files containing a selected mutant. Run and listing JSON reports mark this
population as name-filtered, so `merge` retains historical verdicts absent only because of
the regex. See [Sharding and merging](#sharding-and-merging) for source presentations
and staged reports;
- conditional-compilation evidence excludes code absent from this build;
- explicitly tagged project policy can mark every implementation whose final written trait-path
segment is a named unqualified Rust identifier as ignored, without coupling selection to path
qualification or human-readable report text;
- suppressions withdraw explicitly accepted sites;
- sharding assigns stable portions of the population to separate campaigns.
`unsuppress` audits whether each directive still governs any mutant in its source file, regardless
of a run's name filter. It rejects `--re` instead of treating a partial match as evidence that a
directive has become idle.
Trait-name policy is deliberately lexical. `impl Debug`, `impl fmt::Debug`, and
`impl core::fmt::Debug` share the final written identifier `Debug`, while an imported alias retains
the identifier written in the implementation. Without rustc name resolution, discovery cannot
semantically distinguish identically named imported traits. Matching mutants remain in reports
with an `ignored` verdict and configuration suppression reason. Each rule records whether it
matched during discovery, and an unmatched rule is a usage error rather than a silent no-op; this
makes a typo in selection policy visible before it can quietly change the mutation population.
In-source suppression has two equivalent channels. Whole items can use the inert
`#[gamma::skip]` attribute supplied by the `cargo-gamma-attrs` package. Statements and expressions,
where custom attributes remain unstable, can use the comment spelling
`// #[gamma::skip(...)]`. The source engine interprets the comment form;
the attribute crate validates the compiled form while expanding to the annotated item unchanged.
Both channels permit one timeout multiplier per directive across positional and named spellings;
stating another is a usage error rather than an ordered override. A genuinely bare directive
selects every mutator in scope, while every comma-delimited argument in a non-bare directive must
be non-empty; malformed leading, repeated, comma-only, and trailing commas are usage errors rather
than alternate spellings of the all-mutator form.
For `cfg_attr`, a definitely false predicate leaves the nested directive inactive, a true
predicate applies it, and an unknown predicate applies it conservatively rather than manufacturing
a survivor the author believed suppressed.
The same selector language also supports `#[gamma::expect_survived(...)]` and
`#[gamma::expect_killed(...)]`, with equivalent comment spellings for statements and expressions.
These are assertions about the test oracle, not suppressions: governed mutants still run and still
count normally. Once a governed mutant has a score-bearing outcome, a disagreement fails the
campaign's correctness gate. `expect_killed` requires `killed`;
`expect_survived` accepts every undetected outcome — `survived`, `uncovered`, `timeout`, or
`outofmem` — because its assertion is that the suite did not detect the mutant, not that execution
completed normally. Unviable, ignored, not-built, flaky, and pending mutants prove neither
expectation. An expectation that currently governs no mutant is not reported as an idle
suppression.
`#[gamma::test_timeout_multiplier(...)]` and its `timeout_multiplier` compatibility spelling apply
a positive per-site multiplier to the normal baseline-derived timeout. They change how long the
mutant is allowed to run, not how it is scored, and may be combined with a skip or expectation
directive only when the site states one unambiguous multiplier.
Every uncertainty fails toward **keeping** work. Running an unnecessary mutant costs time; silently
dropping a valid mutant improves the score without evidence.
A file cargo-gamma cannot analyze but `rustc` can build — one nested past the parser's recursion
limit, for instance — is reported as an unanalyzable file rather than failing the campaign, and this
holds whether the file was selected for mutation or read only for the module declarations it
contributes. The reported set is keyed and ordered by absolute path, so narrowing a selection moves
such a file between the two paths without moving it in the report, and two runs over one workspace
name the same files in the same order regardless of which worker claimed which file. When a
declaration-only file is skipped, the modules only it declares are treated as absent, exactly as if
the selection had never mentioned it.
## Representing every mutant in one program
An expression mutation is encoded as a branch:
```rust
// original
a < b
// instrumented
(if ::gamma_rt::a(7u32) { (a) <= (b) } else { a < b })
```
The environment selects ordinal `7` for the lifetime of one test process. All other guards return
false and execute the original program.
Blocks and statements use equivalent forms suited to their syntax:
| Expression | Produce the replacement value |
| Block | Replace the block body |
| Iterator-returning block | Wrap original and replacement in a shared `Either` type so opaque `impl Iterator` arms agree |
| Iterator-valued expression | Wrap original and replacement in a shared `Either` type so pipeline transformations retain one concrete type |
| Iterator payload in supported `Option`/`Result` containers | Map the iterator payload into `Either`, including early-return paths |
| Loop control | Conditionally execute the replacement while retaining the original diverging tail |
| Statement | Delete or replace the statement |
| Match-arm pattern | Add a false guard so matching falls through to a later wildcard arm |
Statement deletion is split into concrete mutators such as `stmt.delete_call` and
`stmt.delete_assign`, so selection and suppression name the operation precisely.
### Exactly one active mutant
Only one mutant is active in a process. This invariant has three consequences:
- a failing test can be attributed to one mutant;
- workers can share immutable binaries safely;
- nested guards need instrumented children only in their original branch.
The last point prevents exponential source growth. If an outer mutant is active, no nested mutant
can also be active, so the outer replacement arm can contain plain original text. Deeply nested
sites can still make the encoding grow superlinearly because enclosing guards repeat parts of the
original expression.
### Semantic restraint
The operator catalog includes `%` to `+` and `+=` to `*=`, both enabled by
default. Each substitution has a distinct mutator name and is selected through
its family and presets. Compound-assignment replacements rely on compiler
convergence to exclude unsupported trait operations, including multiplication
assignment on temporal and textual types. These candidates may add convergence
rounds before tests run.
`assign.add_to_mul` skips an integer-literal right operand equal to one,
including suffixes, alternate bases, and parentheses. This limits stalled
counter mutants and applies even when explicitly selected or used with custom
assignment operators. Variables, constants, and expressions remain candidates.
Comparisons inside a match arm's `if` guard receive the same relational
mutations as comparisons elsewhere, while the pattern remains untouched.
Guard branches duplicate source text rather than eagerly evaluating and binding both alternatives.
Typed runtime helpers preserve expected-type inference and evaluate their operand once. When an
expression guard would shorten a borrowed temporary's lifetime, instrumentation lifts the edit
to its consuming expression without crossing short-circuit, closure, or async boundaries.
Attributes stay outside generated operators, and configuration-selected tails retain their
original type. Iterator-container guards route early returns through the same payload mapping as
the body's tail. The schema may affect code size, inlining, and layout, but it must preserve the
unmutated program's observable moves, borrows, evaluation order, and destruction order.
Guarded ownership interactions are not proof that an individual mutation is invalid.
When source-visible moves and restorations require it, a viable edit's guard lifts to the
nearest loop or enclosing block. The replacement applies only that edit to uninstrumented
source, while the original region retains nested guards. Independently viable restoration
and exit mutations therefore remain available without relying on the compiler to infer
runtime guard exclusivity.
The selection channel is process-local environment state captured by the runtime. On Linux the
runtime reads the immutable environment image saved by `exec`, so an earlier native constructor
cannot race the capture by starting a thread that changes the live environment. cargo-gamma sets
variables on each child before launch and never mutates its own environment to select mutants.
Absence and acquisition failure are distinct: absence selects the baseline, while an open or read
failure emits a fixed runtime marker and terminates startup. The parent recognizes that marker
before interpreting either libtest or nextest status and records the run as unmetered, so a mutant
that was requested but never activated cannot become a phantom survivor or kill.
That distinction covers the reach-observation request as well as the ordinal. An environment image
that could not be read fails at startup rather than silently disabling observation. A missing
observation file is unknown reachability, never evidence that the guard was not reached.
Signal delivery interrupts a read without failing it, so the capture retries a bounded number of
times. The budget is spent across the whole capture rather than per read, so a stream of
interruptions cannot refill it between chunks; exhausting it is an acquisition failure like any
other, never a silent absence and never an unbounded spin inside a constructor.
## Building a viable schema
Some syntactically valid mutations are not well typed. A replacement may require a trait the
original type does not implement, or may violate a type-specific operator rule. In a mutant schema,
one such mutation can prevent every test binary from being built.
cargo-gamma resolves this with a rollback fixpoint:
1. Instrument the currently admitted population.
2. Ask Cargo to check every selected test target while continuing past independent failures.
3. Attribute structured compiler diagnostics to mutation guards.
4. Withdraw every blamed mutant.
5. Rewrite only files whose admitted population changed.
6. Repeat until the schema checks or convergence can no longer be established safely.
7. Build the converged test targets once to produce the baseline binaries. If code generation or
linking exposes a failure that checking could not, converge that final build under the same
bounded attribution rules.
Withdrawn mutants are reported as unviable. They are not silently discarded, because the score is
meaningful only when its excluded population remains visible.
The source collector prevents a mutation before this loop when source evidence proves the replacement
invalid or equivalent to the original. Equivalent candidates are omitted from the population rather
than executed or counted as source suppressions. This includes token-identical replacements,
supported default-value spellings, and result-only defaulting of calls proven to return unit.
Unit-call proof requires an unambiguous local free-function declaration visible at the call site;
unresolved calls, imported names, and potentially shadowed bindings are not sufficient evidence.
Empty-constructor equivalence requires resolution to a supported standard type without custom
allocator or hasher arguments. A user-defined `new` or `default` method is not a standard constructor
merely because it has the same name. Allocation-capacity hints and application-specific invariants
remain separate policy decisions, not general equivalence proofs.
Resource-safety policy also omits mutations proven to introduce a stalled loop
or unbounded retained-memory growth. These mutations never enter the population:
they have no verdict, suppression count, or score contribution, and are not
classified as equivalent or killed. The proof is conditional on reaching the
affected loop path, not on a test exercising it. Bounded proofs cover invariant
primitive-scalar conditions, typed arithmetic fixed points, supported character
branches, standard collection-length invariance, and stable derived boundaries.
Calls require source-resolved effect evidence or an exact supported dependency
API contract; matching a helper or method name is not sufficient. Standard owned
string appends of literals can additionally establish retained growth.
Known standard character iterators additionally permit a proof conditional on
reaching exhaustion: repeated `next()` calls cannot revive an exhausted iterator.
Unknown effects and potential escapes keep the candidate eligible. Unbounded
growth does not imply a particular OOM verdict or depend on machine memory.
Proof-budget exhaustion also keeps the candidate eligible. Dependency-body
evidence must be configuration-invariant and bound to the exact source generation
read for the proof; evidence-only sources do not become mutation targets.
Its local evidence distinguishes explicit and inferred integer, unsigned, floating-point,
textual, optional, and temporal values; expected types flow through local returns, arguments,
fields, casts, indices, and initializers. Proven unsigned zeroes do not receive `-1`, proven floats
use floating-point perturbation units, and proven non-additive values do not receive integer
operators. Option-producing chains are kept out of iterator-shaped rewrites. Ambiguous external
trait implementations, constructors, moves, and overloaded operators remain candidates: guessing
that they are invalid could remove a viable survivor and improve the score incorrectly.
File and diff filters narrow the mutation population, not the package's declaration
evidence. Excluded sources still contribute module declarations and `Default`
implementations before candidate collection. Only an unanalyzable source makes its
package's negative evidence incomplete; exclusions alone do not.
`--in-diff` is changed-lines selection, not test-impact analysis. Changes or deletions in tests do
not select mutants in unchanged production code, even when those edits remove all test protection.
A test-only diff can therefore select no mutants and succeed. Passing this narrower check makes no
claim about code outside the selected mutation sites. Full mutation runs remain necessary to detect
such regressions; CI can omit `--in-diff` when tests or test configuration change, alongside periodic
full runs.
Diff path validation also recognizes selected packages' nonmutation targets,
including integration tests, examples, and their source-directory helpers.
These paths validate ordinary workspace diffs without becoming mutation targets
or declaration evidence. Each additional directory is inventoried once per
survey; build-script roots are recognized directly without recursively walking
the package root and its build artifacts. Validation-only walks prune Cargo's
resolved target directory and conventional `target` and `.git` directories at
the workspace and package roots, including for root-level tests and examples.
Ordinary mutation-source walks remain unpruned. Missing Rust paths remain errors.
Each compiler-blamed unviable mutant retains a bounded reason: rustc's primary error code when
present, a normalized and length-capped primary-message category, and whether a primary span
identified the generated replacement text. Quoted source fragments, path-shaped tokens, and
unbounded rendered diagnostics are not retained. Follow-on diagnostics cannot replace the first
attributed root cause. Proof builds that isolate a compiler failure without an attributable span
use a fixed `isolated compiler failure` category. Mutants abandoned because a build could not be
converged remain `notbuilt`, not unviable.
Historical compiler exclusions are filtered before assigning executable ordinals
or emitting guards. The cumulative `compiler-exclusions.json` inventory in the
campaign cache is seeded once from earlier completed/incomplete compiler evidence
and checked-in hints. It is independent of the current feature/toolchain context
and `--incremental`; exact matching mutants remain unviable in reports without
being recompiled. New individual failures are saved atomically as they are proved,
including during interrupted preparation. Interaction-only exclusions are never
added. To retry a repaired mutation, explicitly remove its inventory entry and
disable ordinary incremental reuse for that retry. An existing inventory is
authoritative; deleting the whole file instead permits historical evidence to be
imported again. Dry runs apply this policy without writing the inventory; raw
candidate listings do not apply it.
Known attribution and mutation-generation repairs require fresh compiler proof
before their affected historical exclusions apply. Imported `try.propagate_to_unwrap`
entries with no original diagnostic also require revalidation because added
`ControlFlow` support may make them viable. A fresh individual failure clears
that entry's pending marker without replacing its original recorded reason.
Generated whole-function replacements require positive source-visible evidence
for each `Default` construction, including recursively generated payloads.
Unknown implementations are omitted rather than classified as unviable.
Independent alternatives such as `None` and empty standard collections remain
eligible. This intentionally omits otherwise viable defaults hidden behind
unresolved dependency paths or opaque macros. No Default-only compiler pass
runs between pristine validation and full-schema validation.
All remaining selected target packages are then instrumented. cargo-gamma asks Cargo to
check their test targets together, letting Cargo schedule independent packages concurrently and one
compiler round withdraw failures from unrelated packages without repeatedly paying for code
generation and linking. After check convergence, one build produces the test binaries consumed by
baseline measurement. Example and benchmark targets are not built: cargo-gamma does not execute
them, so they are not part of its compilation oracle. A selected package with mutations but no test
target is still included in schema checking; after its source is proven buildable, its mutants are
reported uncovered rather than being mistaken for a build cargo-gamma failed to perform. A
selection preserves the exact package roots and test-target selectors validated once
on pristine source. All exclusion and isolation checks use
`cargo check --tests --keep-going`; the final code-generating invocation uses
`cargo build --tests --keep-going`, retaining features, profiles, compiler options,
and roots. `--test-lib` retains `cargo test --no-run --lib --no-fail-fast`
throughout to select only library test harnesses. No historical probe, additional
pristine control, target reduction, or scope-widening retry is performed.
During convergence, both progress and dashboard start at
`Excluding unviable mutants (0 found)` and retain the maximum confirmed compiler-unviability
count through ordinary checks and proof builds. Pending, active-subset, and interaction-only
exclusions do not inflate it. Historical exclusions are shown immediately, and
new individual proofs update the count before isolation continues. The display
omits check and proof counters.
Cargo's per-invocation unit counter and isolation diagnostics are hidden because their changing
denominators do not measure progress toward a converged schema. `--show-build` retains Cargo's raw
build narration for troubleshooting. When convergence completes, cargo-gamma reports the unviable,
viable, and not-built counts. If a diagnostic cannot be attributed directly, cargo-gamma reads the
failing Cargo package and target from the structured message. Ownership failures first prioritize
control-flow and state-update mutants in items whose instrumented guard envelopes contain a
diagnostic location, followed by the complete item. Isolation then considers diagnostic files,
the failing package, and finally that package's transitive dependency cone.
Within each tier, rustc's error code orders mutators whose compile-time effects can plausibly cause
that diagnostic ahead of type- and trait-invariant value changes. The complete tier is always tried
if the compiler does not prove the narrower heuristic sufficient, so unusual const-generic or
downstream effects cannot hide a viable cause. Unrelated and downstream packages never enter the
candidate set. Each proof keeps the full validated compiler context and activates
the requested candidate subset in the failed dependency-connected components.
Other mutations in those components are restored to prevent cross-tier interactions
from becoming individual blame. Disconnected successful components stay instrumented.
Only changed source selections are regenerated, and byte-identical outputs are not written.
The implementation retains diagnostic evidence about the admitted population, rewritten-file
count, Cargo reuse and rebuild counts, and failing targets without adding it to the normal console
display. Independent failing targets from one global check are isolated
together and withdrawn before the next global check, rather than forcing one complete workspace
round per target. Within a context, independent failures are also settled in the same bounded
search: each resolution removes its blamed members or interaction exclusion before retrying.
Diagnostic attribution uses guard locations in the instrumented text, not original line numbers.
Instrumentation changes line positions, and nested mutations can share original spans. The mutated
branches themselves do not overlap. Direct attribution requires a diagnostic span wholly contained
within generated replacement text, not a partial overlap or a span enclosing the surrounding match.
Secondary spans are considered only when that diagnostic has no exact primary attribution, and
must satisfy the same containment rule; they are not promoted to primary evidence. Flow-sensitive
diagnostics always require proof-build isolation, even when their primary span is inside a
replacement, because a different guarded edit can cause the ownership or inference failure.
Other enclosing or fallback diagnostics likewise trigger isolation before a non-empty replacement
can be reported as unviable. Deletions have no generated replacement text; positional deletion
containment remains available for non-flow-sensitive diagnostics, but a broad flow region does not
establish deletion blame. Within one compiler batch, exact generated-text attribution takes
precedence over deletion fallbacks so follow-on diagnostics cannot withdraw neighboring mutants;
any independent failure is exposed by the next rollback round. Diagnostic-local item searches
apply to type and lifetime errors as well as ownership errors. Isolation admits at most 4,096
candidates per search and performs at most 32 proof checks for one failing target. Oversized
tiers are partitioned deterministically, retaining file and item locality. Passing partitions
do not rule out interactions between them and cannot establish individual viability.
One Cargo convergence
invocation examines at most 64 diagnostic contexts and therefore launches at most 2,048 proof
checks. The candidate and per-context proof limits cover binary narrowing of a 4,096-candidate
search with interaction work left over. The invocation-wide proof ceiling is derived from the
context and local-proof limits so every admitted context can use its complete allowance; contexts
with larger dependency cones run first so diagnostic order cannot strand most of the population.
Unresolved contexts do not justify excluding their package or dependency population. Proven
causes are withdrawn and the remaining schema is checked again. If bounded recovery cannot
prove a cause and makes no progress, the command fails with the unresolved targets and advice
to inspect `--show-build` output or narrow the campaign with `--file`; it does not publish
thousands of unjudged candidates as `notbuilt`. These values may be raised only from deterministic
candidate, context, and invocation counts from representative campaigns, not from host-specific
wall-clock samples. Isolation never falls back to workspace-wide isolation. Proof builds preserve
the failed invocation's selected package roots and its Cargo feature-unification graph;
only the active mutation subset changes. The initial successful pristine validation
is the control; a failed singleton proof establishes individual unviability without
recompiling pristine source. A minimal group
that fails only in combination is recorded as a compiler interaction: one deterministic member is
excluded as `notbuilt`, without calling any member individually unviable, and the rest remain in the
schema. Direct compiler blame takes precedence if another diagnostic context also leaves that mutant
unresolved or includes it in an interaction. Isolation work does not consume the configured rollback-round allowance; a confirmed mutant
is charged to the ordinary failed round that required isolation.
Proof results are memoized within a diagnostic context. In particular, halves already proved clean
while detecting an interaction are not launched again when delta debugging begins; an
indeterminate timeout remains indeterminate and consumes its original proof-budget slot.
Exact-subset results remain available while settling independent
failures under the same Cargo command, with a separate 32-proof budget per diagnostic context.
No result survives the failed Cargo invocation boundary,
and cached retries must remove at least one candidate rather than relying on launches to consume
the budget. Development-dependency selectors for tests, examples, and benchmarks are retained
for ambiguous library/binary failures: removing them can change dependency features even with
identical package roots.
Convergence evidence is decoded only when an event consumer opts in. The normal console consumes
only the monotonic compiler-unviability count, while diagnostic reporters can request rewritten
paths, Cargo reuse/rebuild counts, and failed targets. The public event surface exposes standard
path views rather than committing consumers to the internal Camino representation.
Target-frontier acceptance was evaluated and rejected. A deterministic direct/downstream/
interaction command-count model found that a clean global check takes one invocation, whereas
checking target frontiers and still performing the required global confirmation takes at least one
invocation per frontier plus that confirmation. Removing the confirmation is unsound: Cargo feature
unification and interactions spanning targets can make individually clean frontiers fail together.
The implementation therefore retains global confirmation and does not accept packages one at a
time.
## The scratch workspace
The checkout is never instrumented in place. cargo-gamma maintains two coordinated scratch areas.
The external per-workspace area contains:
- a synchronized source workspace under `workspace/`;
- the vendored guard runtime;
- the stable process-held workspace lock.
A configured platform cache home that physically resolves inside the original workspace is
refused. In the split layout the source walk excludes the target-resident campaign cache, not the
external scratch destination; accepting that placement would let an all-files copy recurse into
the workspace it is creating.
The resolved Cargo target directory contains
`cargo-gamma/cache/<workspace-identity>/`, which contains:
- Cargo build artifacts under `target/`;
- incremental campaign records and transient execution data, including reach-observation files,
`last-gamma-run.json`, `gamma-progress.log`, and `gamma-selection.jsonl`.
The external per-workspace cache holds a small `campaign-location` locator naming the campaign
base used by the latest completed run. State-consuming commands first ask Cargo for the selected
directory's current workspace, then accept only the locator owned by that workspace. Failure to
resolve current workspace identity is an error rather than permission to adopt potentially stale
state. Completion publishes the record before the locator and advertises postprocessing commands
only after resolving the locator back to that record.
Normal runs publish `gamma-report.json`, `gamma-report.html`, `gamma-report.sarif`,
`gamma-perf-advice.md`, and `gamma-diagnostics.json` under the original workspace's
`target/cargo-gamma/`. `last-gamma-run.json`, `gamma-progress.log`, and
`gamma-selection.jsonl` remain reusable cache state rather than published artifacts. The selection
journal is append-only, flushed during the campaign, and remains non-verdict scheduling telemetry.
It retains at most 64 MiB per campaign, ending with an explicit truncation record when further
attempts are omitted; reaching that diagnostic ceiling never stops mutant execution.
Its attempt-level interpretation is described under
[Cheapest evidence first](#cheapest-evidence-first). An explicit `--cache-dir`
retains the all-in-one layout and relocates the synchronized workspace, Cargo artifacts, and
campaign state together. The default external synchronized workspace exposes the original
checkout's version-control metadata to build scripts. When `--cache-dir` explicitly relocates that
workspace, a repository whose build scripts require checkout-visible Git metadata must choose a
cache beneath the same repository. To move heavy compiler artifacts independently, leave
`--cache-dir` unset and use Cargo's `CARGO_TARGET_DIR` environment variable or `build.target-dir`
configuration; cargo-gamma resolves that target and places campaign artifacts beneath it without
relocating the source workspace. `--artifact-dir` relocates all five published artifacts together,
and its directory is created when absent. A relative configured artifact directory is interpreted
relative to the invoking process for both publication and `explain`.
The diagnostics bundle uses schema version 6 and contains aggregate algorithm-health telemetry
rather than one row per mutant. Build withdrawals are grouped by package, mutator, error code,
normalized message category, and whether the primary diagnostic span identified the replacement
site. Every group is retained so a failed run remains fully classifiable after its scratch tree is
removed; legacy omitted-group and omitted-mutant fields remain readable but new bundles leave them
at zero. Package identifiers and free-form message categories both follow the bundle's redaction
policy: names remain readable, hashed values remain groupable, and omitted values carry no
source-authored text. Exact compiler-withdrawn mutant records, including their source identities
and replacements, remain in the local completed or incomplete campaign record rather than in the
shareable diagnostics bundle. Absent replacement-site evidence in a legacy bundle remains unknown
rather than being interpreted as an observed non-replacement span. Build rounds attribute newly
withdrawn mutants to packages while retaining the round's actual workspace-wide elapsed time;
they do not invent per-package build durations. Mutator, package, and confidence breakdowns report
generated and viable populations,
kills, survivors, compiler-confirmed unviability, CPU time, and distinct killing tests observed
only in that group. Killing-test identity includes the package, Cargo target kind and name, and
harness test name, so equal test names in different binaries remain distinct. This additional detail is
diagnostic-file-only and does not add console output or campaign phases. Sweep
telemetry records whole-binary and uncovered decisions; exact-hint and inferred-test
candidate/probe/hit funnels split into item, reach, and file tiers; and each package's first mutant start and final
completion relative to the sweep start. Package CPU time remains in the package breakdown, while
the package sweep span is wall time and may overlap other packages under concurrency.
Those five files are the completed-run set. Before a build starts, cargo-gamma removes the prior
`baseline-failures/` tree; inability to remove it stops the run so stale records cannot be mistaken
for current failures. A baseline failure publishes one stable, filesystem-safe directory per
failed test beneath
`baseline-failures/<package>/<target>/tests/<test-name>/`. Unnamed binary-level failures use
categorical leaves such as `timeout`, `out-of-memory`, `enumeration-failure`,
`environment-failure`, and `anonymous-failure`. Components are length-bounded and valid on Windows
and Unix. The common path stays readable; when sanitized, truncated, case-folded, or duplicate
identities collide, a short BLAKE3 suffix derived from package, target, package-id, executable,
failure kind, and test identity disambiguates them deterministically. Each failure directory
contains `failure.json` and `diags.json`.
The baseline record uses `schemaVersion: 1` and records the failure kind and reason; package,
target, runner, executable, and working directory; cargo-gamma's explicit environment overrides;
failing and last-observed tests; termination, elapsed time, budget, peak, and memory limit; and
safely encoded stdout and stderr tails. Each stream retains at most 64 KiB and 2,000 lines, and the
record says when output was truncated. Direct libtest baselines run to completion within their time
and memory budgets instead of stopping at the first `FAILED` announcement. A record retains at most
64 named failures and reports how many additional names were omitted; libtest's trailing captured
output remains retained. Mutant attempts still stop at the first killing test. All retained
binaries settle before one concise aggregate error reports the failure count and directs the reader
to `baseline-failures/`. The canonical diagnostics and each
successfully published `failure.json` and `diags.json` are announced with a `Wrote` line using the
platform's native path separator, while the verbose per-failure summaries are not repeated. Both
early-failure artifacts are written before the failed scratch workspace is removed.
The diagnostics retain the settled plan from the completed instrumented build, so population,
unviable, pending, mutator, and package data already known before the baseline failure are not
replaced by an empty survey skeleton.
The same failure publishes compiler-confirmed unviability to
`incomplete-gamma-learning.json` before scratch cleanup. That record contains no test-derived
verdicts; a later run may adopt only its compiler outcomes, under the same source and complete
compilation-context guards as a completed record. A successful completed-record publication
atomically replaces this incomplete generation.
The source tree preserves symlinks and honors workspace ignore rules. Relative path dependencies
that leave the workspace are anchored to their original locations so moving the workspace does not
change Cargo's dependency graph.
The 0.3 compatibility boundary and adopter actions are recorded in
[the migration guide](MIGRATION.md). In particular, reach observations accompany
ordinary mutant launches, whole-binary mode explicitly disables case selection, promoted hints
use YAML, and baseline failures use a per-failure directory tree.
Only one cargo-gamma command may operate on an original workspace at a time. A process-held lock in
that workspace's default external scratch area applies even when `--cache-dir` redirects reusable state.
Conditional source, configuration, and hints publication uses this lock on every platform, so
Windows never needs persistent sibling lock files in the checkout.
The default external scratch base lives under the user cache directory as
`cargo-gamma/<identity>`. The target-resident campaign base is
`<resolved-target>/cargo-gamma/cache/<identity>`. The identity is the first sixty-four bits of the
BLAKE3 digest of the resolved physical workspace root, rendered as sixteen hex characters.
Filesystem aliases of the same existing root therefore share one cache and lock domain, while
workspaces configured to share a Cargo target directory retain separate campaign state. The
algorithm is pinned by cargo-gamma rather than borrowed from the
standard library's default hasher, which is explicitly free to change between releases: this name is
where the lock serializing every source-changing command for one workspace lives, so two binaries
that derived different names for one workspace would rewrite one tree concurrently while each
observed no contention. Sixty-four bits is a directory name a user reads, and a collision is caught
rather than trusted — the default base also carries the ownership marker, and a second workspace
landing on the same name is refused with a usage error naming both roots and suggesting
`--cache-dir`. An unmarked cache containing anything other than the lock created while claiming it
is refused rather than adopted.
Before creating, claiming, or cleaning default state, cargo-gamma independently verifies both
derived paths. The external scratch must be an identity directory directly beneath the dedicated
`cargo-gamma` namespace; the campaign cache must be the same identity directly beneath
`<resolved-target>/cargo-gamma/cache`. A malformed path is refused before a lock, ownership marker,
synchronized workspace, Cargo target, or removal can touch it. These checks are deliberately
separate from path construction so a defect there cannot redefine what counts as a valid cache.
An explicit `--cache-dir` names the cache base itself and must be empty on first use. cargo-gamma
writes an ownership marker tying it to the original workspace and takes a second process-held lock
for that cache. An unmarked non-empty directory and a cache owned by another workspace are refused
before synchronization or cleanup can alter their contents. The two lock domains prevent both
collisions: commands sharing an original workspace cannot race publication, and different
workspaces cannot race over redirected reusable state. The operating system releases both locks
automatically when their owner exits.
A redirected cache is somewhere the invoking user chose, which means it can be somewhere other
people can reach — and its contents are executed: the synchronized tree is built and its tests are
run. Three refusals apply before anything is written there. The base may not be a symbolic link or
anything other than a directory, checked both before and after it is created, so a redirect cannot
be aimed through a link at a target it does not name. On Unix, every directory on the path to the
base must be owned by the invoking user or by root and must not be group- or world-writable unless
the Unix sticky bit is set, which restricts removal and renaming of entries to their owner, the
directory owner, or a privileged process. This is the rule OpenSSH applies to a home directory, and
it allows shared temporary directories that use the sticky bit by design. The ownership marker is
created exclusively, so a marker planted in advance is a refusal rather than something to
overwrite, and it is read through the same handle its identity was taken from.
These are ownership checks, not a proof that nothing changed between the check and the use: a
directory the operating system reports as private cannot be made public by an unprivileged
stranger, but nothing here makes the sequence atomic, and none of it constrains a privileged user or
the directory's own owner. On Windows the check does not exist. Its security model is per-object
access-control lists (ACLs), which the standard library does not expose, so cargo-gamma performs no
ownership test there and claims none; a redirected cache on Windows is trusted exactly as far as
the directory the user named is.
Successful campaigns retain the external synchronized tree and target-resident build artifacts. A later campaign performs a
delta synchronization:
- byte-identical inputs remain untouched, preserving Cargo's incremental state;
- changed inputs are replaced and receive a fresh modification time;
- inputs removed from the checkout are removed from the cached workspace;
- generated artifacts remain outside the synchronized source tree.
`cargo gamma clean` takes the campaign lock and removes both the current workspace-specific
external scratch contents and target-resident campaign-cache contents. It preserves the external
lock and ownership markers and does not remove published reports under `target/cargo-gamma`,
durable hints, or source suppressions. Both cache locations are named in the output; a workspace
with nothing cached is told so. Cache identity uses the current stable naming scheme without a
migration path, and state from the former all-external layout is neither read nor migrated.
Correct invalidation is more important than avoiding a copy. Preserving an old timestamp on changed
bytes could let Cargo reuse an artifact compiled from stale source.
## Establishing the oracle
The **oracle** is the set of tests allowed to decide a mutant's verdict. It is established before
any mutant is judged. Warnings about unusually expensive test harnesses are restricted to this set,
so tests in unrelated workspace packages are not presented as costs of the run.
### Baseline
The unmutated test binaries run first. Direct libtest binaries are allowed to complete after a
failure announcement, within the existing time and memory budgets, so one pass discovers every
failure the harness reports and retains its final panic and captured-output section. A test-failing
binary is not retried: even a passing retry would make the baseline flaky and therefore unusable,
so relaunching it cannot allow the campaign to proceed. A terminal failure does not cancel other
retained binaries: every selected binary executes exactly once, all settle, then one error reports
the failure count and artifact directory. The red
baseline still stops the campaign before mutants run, because a test that already fails makes every
mutant appear detected and the mutation score meaningless.
The baseline also measures:
- duration per test binary;
- the longest legitimate period without harness progress;
- peak memory where the host can measure it.
Its completion summary reports both the number of tests that ran and the number of test binaries
across which they ran; custom harnesses that do not announce a test count still report the binary
count and elapsed time.
Timeout, stall, and memory policies derive from these observations rather than from a machine-
independent constant. The baseline and mutant runs use the same harness and execution environment.
### Harnesses
By default, cargo-gamma launches libtest binaries directly. Direct launch avoids invoking Cargo for
every mutant. Before the first launch, cargo-gamma reconstructs Cargo's test-process environment
once from the successful build stream and `cargo metadata`. Each binary receives its package's
`CARGO_MANIFEST_*` and `CARGO_PKG_*` values, build-script `OUT_DIR` and `rustc-env` values, and,
for integration tests and benchmarks, `CARGO_BIN_EXE_*` paths. The shared environment carries the
Cargo and rustup toolchain selection and reproduces Cargo's executable and dynamic-library search
paths. Test listing, the baseline, and mutant attempts all use that same environment.
Nextest mode provides process isolation for suites that cannot run safely on libtest's shared
threaded process. cargo-gamma prepares nextest's description of the already-built binaries once,
then reuses it; allowing nextest to rebuild for every mutant would reintroduce compilation into the
inner loop. Cargo-gamma supplies the same Cargo package and toolchain context to nextest, while
nextest remains responsible for its runner-specific `NEXTEST_*` values and may normalize values
according to its own compatibility contract. A resource-bearing selection limits nextest itself to
one concurrent test process, matching the one-thread direct-libtest contract.
## Selecting only tests that can matter
After removing per-mutant builds, test execution is the dominant cost. cargo-gamma narrows it using
evidence that preserves the verdict.
Every campaign first drains exact hinted cases across the population. Each worker
then adjudicates one unresolved mutant end-to-end: ranked inferred cases, all
inferred full binaries, and the remaining reachable full binaries. There are no
population-wide barriers between these last three stages.
Libtest and nextest inference keep the first six eligible distinct ranked cases
and execute them individually in priority order, without binary-level grouping.
Cases and full binaries execute sequentially per mutant, with parallelism across mutants.
Nextest preserves process-per-test isolation and runs one test process at a time
for a given mutant. Control confirmations and nextest retries do not consume
the phase-2 candidate budget.
Kills settle immediately and publish learning without waiting for
canonical predecessors. Passing named probes are not repeated during inference;
full-binary outcomes are reused between the final two phases. All required
full-binary checks must pass before survival is reported. Timeouts, resource
failures, and incomplete linkage are never treated as passing evidence.
Fallback keeps own-package binaries first, then prefers the binaries credited
with the most distinct mutant kills. Known exact hints seed this history, and
observed kills update it live without crediting the same mutant twice. Baseline
runtime and stable identity break ties. Each mutant reads the counts when it
enters fallback. Final non-killing outcome precedence retains the original
baseline-sorted reachable order.
Package dependency reachability is refined by exact compiler linkage when
available, without discarding unknown links as though they were proven absent.
Every phase admits up to the configured job count, without an additional CPU-count
cap or aggregate memory reservation. Users reduce the job count if concurrent
runs exhaust system memory. Named resources, per-binary ceilings, resource retries and enumeration controls,
and baseline harness-thread settings remain in force.
The scheduler version is frozen for testing resume; unfinished
in-memory work can be repeated after interruption, but completed verdicts and
accumulated accounting are retained. Snapshots from earlier scheduler policies
require a new campaign.
The complete operational sequence—from loading hints through mutant queue
construction, and per-mutant test selection—is summarized in
[Scheduling](SCHEDULING.md).
### Package reachability
By default, each mutant is judged only by test binaries from the package that owns it. A
whole-workspace campaign therefore behaves like one package-local campaign per member rather than
letting reverse dependents improve another package's score. `--test-package` names a different
oracle, and `--test-workspace` admits every workspace package.
`--test-lib` narrows the admitted target kinds to library unit-test harnesses. Preflight and final
test compilation use Cargo's `test --no-run --lib` selection. Fallback preserves the admitted
package scope and may broaden target kinds only within that scope; library-only selection never
adds integration, binary, example, or benchmark harnesses. A selected package with no
runnable library test harness is an error rather than a successful empty oracle. Target-pattern
validation retains package ownership: a library target declared only by another workspace member
cannot satisfy an admitted package's `--include-test` or `--exclude-test` pattern.
Within the admitted package set, a test binary cannot execute code it does not link. The Cargo
dependency graph first identifies binaries that cannot reach a mutated package. Cargo Gamma then
interposes on the successful preflight's rustc invocations and combines their exact artifact,
primary-source, dependency-file, and `--extern` relationships with Cargo's artifact messages. Test
targets proven not to link any pending mutated source are omitted from subsequent builds, baseline
measurement and mutant judgement.
Compiler capture is an optimization, never an oracle by itself. A missing or corrupt capture,
ambiguous artifact association, unsupported target kind, opaque path dependency, or untraceable
workspace `--extern` abandons target-level narrowing and retains package-level behavior. If a build
using exact Cargo target selectors fails, the same build is retried with all admitted target kinds
before the failure can affect a verdict: all test targets normally, or all selected packages'
library targets under `--test-lib`. Unknown relationships therefore run or build more tests; they
never hide one.
### Active-mutant reach observations
Package reachability is coarse: many tests link a crate but never execute a particular line.
Ordinary mutant launches request a sealed reach-observation file through `GAMMA_REACH`.
The runtime records whether the active guard was reached without adding sampling launches
or changing guard selection. Exact killer hints, persisted reach hints, and in-run item/file
learning remain checked candidates; anything they do not kill falls back to the complete
reachable binary.
A sealed, passing whole-binary observation that did not reach the guard may exclude that binary
only for another replacement at the exact same stable source site, and only while deterministic
reach narrowing is enabled. If no reachable binary remains, the mutant is uncovered.
Filtered, failed, incomplete, malformed, or missing observations never establish absence.
Unmeasured reach always means “run everything,” never “nothing was reached.”
Each observation file belongs to one process. The coordinator rejects extra records after the
seal, including a descendant's appended stream, and bounds reading before allocating from
file contents. The stream contains the active ordinal if reached, followed by a seal; a
complete observation of no reach contains only the seal. The runtime stores the file path in a
fixed-size, explicitly terminated buffer;
an unusable path cannot create valid negative evidence. Invalid observations are discarded
without changing the mutant's test outcome.
`--whole-test-binaries` suppresses case-level selection and deterministic reach narrowing.
Suites whose reachability depends on threads, the clock, or randomness can use it to keep every
reachable binary in the oracle.
### Cheapest evidence first
An eligible exact killer hint is probed before inferred tests, even when its binary is later in
canonical order. A previous killer is only a hint: its test must convict again. A kill
settles immediately, including one from a later binary, and suppresses subsequent
inferred probes and fallback for that mutant. If the hint cannot be used or does
not convict, the mutant advances to inferred selection and whole-binary fallback.
Targeted attempts are deduplicated by resolved binary and test name, including aliases and
duplicates across exact hints and inferred tests. Confirmation and necessary whole-binary fallback
are not suppressed. Stale hints can waste work but cannot settle a verdict.
Learning generalizes successful exact probes within one run and across promoted run records. Exact
tests learned from the same `(source file, enclosing item)` are ranked ahead of test binaries that
killed another mutant in the source file. Seed evidence records why a candidate may be explored;
only kills against another mutant count as transfer hits. Transfer hits, misses, and observed cost
rank candidates without economically retiring them.
Inferred named probes keep at most six eligible distinct cases under either runner,
executing each individually in priority order regardless of its binary. The
following phase runs every distinct inferred full binary without a candidate cap.
Only a test failure is a transfer hit and only a clean passing run is a transfer miss. Timeouts,
stalls, memory exhaustion, flakes, enumeration failures, metering loss, and otherwise unjudged
launches are inconclusive: they are recorded in `gamma-selection.jsonl` but do not train candidate
hit/miss rankings. A clean canonical pass contributes half-weight negative evidence to an
already-known item or file binary candidate. This evidence can lower its priority,
but never creates a candidate, excludes a test binary, or changes
a verdict. It is weaker than an observed transfer miss because the canonical run was selected for
verdict completeness rather than as a direct probe of that candidate. Two canonical misses
therefore equal one transfer miss for ranking.
Workers choose mutants at assignment time rather than advancing through a fixed queue.
Assignments preserve package turns, favor fresh effective same-item killers,
and otherwise prefer independent files and items, but same-item siblings may run
concurrently rather than waiting for a scout. Each completed mutant publishes its
exact-test and file/reach learning before releasing its assignment; subsequent
assignments observe that learning.
The durable generalized-hint schema stores seed and transfer evidence for those item and file
rankings plus interned reach sets keyed by stable source-site identity. It is independently
versioned and shared by the run record and checked-in hints artifact. Older generalized schemas are
ignored rather than migrated, without discarding exact per-mutant probes from an otherwise valid
checked-in artifact.
Diagnostics report candidates, attempts, hits, and rejected candidates separately for exact mutant
hints and for inferred item, reach, and file tiers. Every admitted inferred candidate
is rerun before use; persistence never turns inferred test selection into a verdict.
Historical compiler identities follow the separate explicit exclusion policy.
The saved knowledge remains in the `generalized` schema; checkpoint and diagnostic JSON retain
their existing `generalized*` keys for compatibility. These legacy wire names do not describe
runtime inferred selections.
Hint promotion is a projection of persisted campaign state, not another discovery pass. After
resolving and validating the current workspace identity, it uses the campaign ledger's persisted
workspace-relative source paths without a source walk, parsing, or mutant regeneration. Stale
identities remain score-neutral: killer hints are checked again, while imported
compiler identities remain explicit unviable exclusions. Incremental promotion preserves
unrelated entries; `--replace` intentionally drops them and requires a valid persisted completed
campaign record before changing the artifact, never an interrupted checkpoint.
Older records that lack an exact path preserve supported
generalized and compiler-ordering knowledge while omitting the unmappable exact entry. Incremental
promotion treats a missing, corrupt, oversized, or unsupported campaign record as no promotable
knowledge; replacement and commands that reconstruct source edits, such as `cargo gamma suppress`,
load the record strictly and report those conditions as errors.
Ordinary mutant launches use the reach-observation protocol to report whether the active guard was
reached, without another subprocess. Positive observations promote that binary for later mutants
in the same item or file. A sealed, passing, whole-binary observation that did not reach the guard
may exclude that binary only for another replacement of the exact same stable source site, and only
while deterministic reach narrowing is enabled. Filtered, failed, incomplete, or nondeterministic
observations never establish absence. Reports count launches avoided by sweep-derived reach
evidence separately from other learned-order savings.
The first observed test failure settles a mutant, so remaining tests are stopped. Direct libtest
processes are supervised and terminated when their unambiguous `FAILED` announcement is observed;
cargo-gamma does not pass libtest's unstable `--fail-fast` flag to stable harnesses. Nextest uses
its supported fail-fast option. Modes that interleave user output with harness protocol disable
early interpretation and fall back to the process exit status.
## Executing mutants safely
Workers share the immutable schema and build artifacts. For each mutant they launch a fresh process
with that mutant's ordinal selected. One process provides isolation for environment state, static
state, crashes, and resource accounting.
Before any baseline or mutant process starts, cargo-gamma discovers source-declared shared
resources from the finalized test environment and installs one admission policy. Baseline,
filtered probes, whole-binary fallbacks, and confirmations all use that same coordinator, so
calibration and verdict execution observe identical contention limits. Resource-marker discovery
uses the baseline memory policy, because it executes the same test binary before per-mutant limits
exist.
Every launch is treated as a process **tree**, not a single PID. Tests may start servers, child
tools, or nested Cargo processes. Timeout and cancellation must terminate descendants as well as the
direct test binary.
Platform containment uses the strongest suitable primitive:
- Unix process groups provide descendant-directed signaling;
- Linux cgroup v2 provides a boundary a descendant cannot leave, plus process-tree memory
accounting and enforcement;
- Windows job objects provide descendant lifetime and memory control;
- on macOS, which has no process-tree memory boundary, memory is accounted and enforced by sampling
the physical footprint of the process group and of its descendants by parentage.
A process group is escapable: one unprivileged `setsid` or `setpgid` call removes a descendant from
it, and every later signal to the group misses that descendant. cargo-nextest does this for every
test. On macOS, cleanup therefore also stops and ends every descendant found through its
parent; a descendant that has outlived its parent is no longer found. Containment is therefore not
conditional on whether memory is being measured. Every launch enters a cgroup leaf on Linux and a
job object on Windows, including launches that request no accounting at all, such as test listing;
the memory request decides only whether that boundary's readings are reported.
This yields exactly three outcomes, with no silent fourth:
- **Sealed.** The launch is inside a boundary its descendants cannot renounce.
- **Refused.** The host can seal a subtree but this launch could not be given one. The launch does
not happen; one mutant is recorded as unjudged rather than run unreachable.
- **Best effort.** The host offers no unprivileged process-tree boundary at all — every Unix that
is not Linux, and any Linux without a usable delegated cgroup. Containment silently falls back to
the process group; absence of a warning does not prove that sealed containment was available.
Containment is active before user code can escape into an untracked descendant. Cleanup follows an
observe, terminate, release, and reap lifecycle so that slots and platform resources cannot be
reused while an earlier process tree still exists. An observation that finds the leader already
reaped by somebody else revokes every capability naming it by number — its process-group id and its
retained child handle — before returning, since both may already name a replacement; the boundary
named by directory or handle is swept first, while it can still only reach this run's descendants.
Preparation happens once per launch, and ownership is what makes that true rather than a check that
could be worked around. On Linux preparation appends a pre-exec step that moves the child into one
specific leaf; a command prepared twice would walk its child through both leaves while only the last
is reported as its boundary, or through one that has since been removed, failing the spawn outright.
Preparation therefore consumes the command and yields a prepared launch, which is the only thing
that can start the child and never surrenders the command again — so there is nothing left to
prepare a second time, and no run-time mark that a caller could clear. Waiting out a transient spawn
shortage re-spawns the prepared launch already in hand; only one of the children it produces is
adopted.
The terminal-signal boundary registered for a launch is owned by that launch's boundary itself, and
released when the boundary is dropped. A signal handler must never be left holding a descriptor
whose owner has closed it, since the number it names is one the kernel is free to reissue and the
handler's next terminal signal would then act on whatever now answers to it. Tying the registration
to the boundary's own lifetime removes the possibility of a caller releasing it late, twice, or not
at all; the release waits for any handler sweep still using the descriptor before it returns.
### Timeouts and stalls
A hard timeout is calibrated from baseline duration. Suspected timeouts receive a confirmation run
with a larger budget because a loaded host can starve a healthy process, and a false timeout lowers
the score and can fail the run.
The stall detector uses harness progress rather than additional instrumentation. A mutant process
that remains silent far longer than the baseline's longest silence is terminated early. Without a
baseline there is no honest silence threshold, so stall detection is disabled. The last test the
harness announced is a landmark rather than a diagnosis because parallel harnesses announce tests
when they finish, not when they begin.
### Memory
Memory ceilings derive from each baseline binary's peak plus configured multiplier and headroom.
Exceeding the ceiling is distinct from a test failure or timeout because the remedy is different.
When the host cannot provide trustworthy process-tree enforcement, inherited defaults degrade
explicitly, while a user-requested guarantee fails rather than pretending to be active.
Enforcement is either kernel-enforced (a Linux cgroup or a Windows job) or sampled (macOS). A time
budget cannot stand in for either. A mutant that breaks a loop's termination while the loop
allocates can grow by gigabytes per second and exhaust a workstation long before any reasonable
timeout. Sampled enforcement on macOS checks the footprint of the process group and its
descendants every 10 ms and stops them past their ceiling. It reports the same `outofmem` outcome, and the run says once that its
bound is soft. The incident that motivated it and the measured overshoot are recorded in the
[macOS memory design](../../cargo-gamma-process/docs/MACOS_MEMORY.md).
### Baseline and failure attribution
Baseline validates the selected healthy tests before mutation testing. An ordinary failing mutant
execution is then accepted as a kill immediately, including exact hinted tests and failures reached
during resource retries. No unmutated kill confirmation is launched. The `--no-confirm` flag and
`no-confirm` configuration key are unsupported, not inert controls.
A passing baseline does not establish determinism: an intermittent assertion failure after baseline
can count as a kill. Timeout, stall and memory suspicions retain their resource-specific retries.
Nextest enumeration failures retain an unmutated control run; if that run fails a test, the verdict
is flaky rather than a kill. An enumeration control that times out, stalls or exceeds its memory
ceiling also occupies the score-neutral flaky bucket, with the resource cause retained rather than
a claim that the test failed twice. Enumeration controls are the only remaining producer of new
flaky verdicts; they are not general flaky-test detection. Historical flaky verdicts remain readable
and mergeable. All stay outside the score and durable killer knowledge.
## Identity and knowledge across campaigns
Line numbers are unsuitable mutant identities: formatting or inserting a function would rename the
whole population. A mutant ID instead derives from stable semantic context, including:
- workspace-relative file;
- enclosing item identity;
- mutator and replacement;
- normalized source at the site;
- occurrence and `replacement_index` where needed to distinguish repeated forms.
Comments and insignificant inter-token whitespace do not move an identity; literal contents do.
Occurrence positions are reserved before confidence-based default selection, so a site present in
both default and explicit populations keeps the same identity when an earlier optimistic site is
withheld from the default population.
Identity scheme 8 assigns those reservations in source order and distinguishes complete
implementation self-type scopes. Stated-value replacement expressions and composed return
replacements (`tuple`, `Some`, `Ok`, and one-element constructions) are token-normalized:
formatting and comments do not change their identities, but literal contents and operators do.
Including composed replacement text prevents a pruned alternative from transferring its
cached verdict to a different value at the same replacement index.
Reports stamped with an earlier scheme are incompatible and cannot contribute persisted verdicts
to the current population.
The digest is rendered as twelve hex characters. The identity joins reports, shards, suppressions,
SARIF findings, and incremental records.
`run --mutant <ID>` repeats to select an exact current population. Every requested ID must resolve
after normal discovery and selection; stale, unknown, suppressed, or filtered IDs fail rather than
silently yielding an empty campaign. Executed repair runs establish fresh verdicts and do not adopt
cached verdicts; `--dry-run` previews the exact selection without claiming a verdict. `explain <ID>`
resolves the same current identity and adds verdict context from the
current configured artifact directory's `gamma-report.json`; `--report` overrides that path. An
explicit retained report remains explainable when its source site or workspace is no longer
present, using the identity version recorded by that report.
Explicit report files are untrusted presentation input. Their paths, source fragments, test names,
and other strings receive the same control-character encoding at the terminal boundary as live
campaign data.
### Incremental knowledge
The target-resident campaign cache stores facts and hints learned by an earlier campaign:
| Compiler exclusion inventory | Exact IDs filtered before guards are emitted | Explicit policy; retained until the entry is removed |
| Prior killer | Test is tried first, then must fail again | A stale hint changes cost only |
| Unviability | Reused only under matching compilation context and source | Otherwise a valid mutant could disappear from the score |
Test verdicts are never reused. A kill is one observation of a potentially nondeterministic test
suite; unchanged source, configuration, toolchain, and environment cannot prove that the next
observation will agree. Each run therefore re-establishes every score-bearing outcome.
When `--re` selects only part of a file, the completed campaign record retains valid earlier
outcomes and killer hints for unselected mutant IDs in that unchanged file. Its latest campaign
population still names only the selected mutants, so post-run hint promotion stays scoped to them.
Compiler convergence is the semantic type-and-trait oracle for facts syntax cannot establish. It
checks the active target, feature set, compiler, configuration, and dependency graph rather than
depending on a separate analyzer whose view might differ. When convergence completes but a later
baseline phase fails, its exact compiler outcomes and ordering are written to the incomplete
learning record. A later run can therefore reuse definitive negative answers without turning
uncertain external `Default`, constructor, arithmetic, or move behavior into source-level
suppression.
Build incremental mode captures compilation inputs before execution. It reads and hashes regular
workspace files and external path dependencies while excluding generated build, version-control, and
cached workspaces.
Build scripts can read paths Cargo does not declare. When their complete input set cannot be known,
context-validated unviability reuse is disabled rather than guessed. The explicit
historical exclusion inventory remains applicable independently of this optimization.
`--incremental no` does not probe cache-reuse context, resolve cache-only external inputs, or adopt a
prior campaign record for reuse. A real completed run still snapshots its workspace source generation and
writes the outcome ledger used by explicit postprocessing commands; that evidence is never adopted
by the run that records it. Because every measured run publishes this state, including
`--incremental no`, measured runs for one workspace are serialized by its process-held workspace
lock. Dry runs do not publish campaign state or take that lock; they read and
apply the explicit historical exclusion inventory.
Incremental-cache eligibility and completed-campaign publication are separate decisions. An
incomplete conservative input snapshot—for example, because a build script can read undeclared
paths—prevents later verdict reuse, but does not discard source bytes and outcomes the completed
campaign already observed. The completed ledger is published from that captured generation and
postprocessing commands validate each affected current source site when they consume it. Publishing
learning first must not leave a knowledge-only record in place of the final ledger; publication
failure preserves the prior complete generation, is reported explicitly, and suppresses advice for
a command whose required outcomes were not saved.
An existing readable campaign-state generation with an unsupported top-level version is never
treated as an empty record for publication: cargo-gamma leaves it byte-for-byte intact and reports
that a compatible tool or explicit archival/removal is required. Measured campaigns check this
publication compatibility before expensive work, including when incremental reuse is disabled.
Corrupt cache state may still be replaced
because it carries no readable versioned contract.
Compiler-unviability reuse also includes a compiler-evidence policy epoch in its context identity.
Changing attribution policy invalidates older exclusions even when the package version is
unchanged. Older records and frozen checkpoints remain readable, and publication preserves their
original context rather than relabeling old exclusions with the current policy. Exact killer and
ordering knowledge retains its original provenance. This epoch governs ordinary
context-validated reuse, not the independently maintained explicit exclusion inventory.
### Durable hints and suppressions
Compiler-failure identities and killer hints can be promoted into a version-controlled artifact.
Killer hints never settle a verdict without being checked. Historical compiler
identities seed a campaign's explicit exclusion inventory only when it is absent;
an existing inventory remains authoritative. `gamma-hints.yaml` groups exact hints
by source file and interns repeated killer identities.
Ordinary promotion updates knowledge produced by the selected run and preserves everything else;
`--replace` is the explicit request to rebuild from only that selected population. The artifact's
context is provenance, not an admission gate: it records the repository HEAD SHA and UTC generation
date. Killer hints require fresh execution; imported compiler exclusions follow
the explicit inventory policy instead.
`cargo gamma run --update-hints` performs that ordinary incremental promotion after the campaign
reaches a reportable outcome. A successful campaign returns a promotion failure, while normal
nonzero mutation and gate outcomes retain their campaign status if promotion fails. Operational
errors do not promote. The option conflicts with `--dry-run`, and a delegated Linux resume owns
both the campaign and promotion so the parent does not promote a second time.
Automatic hint consumption remains best-effort: malformed or unsupported knowledge is ignored
because an optimization cannot be allowed to stop a run. Explicit incremental promotion has the
opposite failure policy. It refuses an existing artifact it cannot understand rather than replacing
unknown knowledge with a partial generation; `--replace` is the explicit permission to discard it.
That refusal includes an unsupported independently versioned generalized section, whose future
fields cannot be preserved by today's typed serializer. Publication compares against the exact
YAML bytes used for the merge, so another writer's intervening generation
causes a conflict instead of being overwritten. YAML is published atomically and read back before
success is reported, so interruption cannot leave a partially written artifact.
Suppression is different. It is a reviewed policy decision and therefore lives in source or
configuration, not in an ephemeral cache. A cache directory must always be safe to delete without
losing accepted policy. A run that produces a timeout or out-of-memory verdict points to
`cargo gamma suppress`, which reads the persisted outcome ledger and previews the corresponding
reviewed suppression after using Cargo metadata to validate the current workspace identity, but
without synchronization, compilation, baselining, or test execution. Preview performs the same
external-source and recoverability preflight as apply, but describes the diff as proposed because
only the applied tree can be rediscovered and checked for missed or collateral suppressions.
`--apply` writes the proposed edits, verifies the resulting mutant population, and reverts them if
that verification fails, matching `unsuppress`. The ledger records stable
identity, verdict, workspace-relative source, mutator, source-site identity and location,
source-generation evidence, and the optional bounded compiler reason for unviable mutants.
Suppression edits only
uniquely matched unchanged or moved sites, reports stale or ambiguous sites instead of guessing,
and verifies the source-level effect transactionally without altering the ledger or progress log.
### Sharding and merging
Mutants are assigned to shards with stable hashing so changing the shard count moves only the
necessary fraction of the population. Reports merge by mutant identity, keeping freshness and
withdrawal explicit.
A report records the mutant-ID scheme used to produce its identities. A report that omits this
metadata is treated as using the current scheme. When inputs explicitly identify different
schemes, merge isolates the newest scheme and reports every excluded input rather than counting
identities from different namespaces together. A rotation that spans an identity change must
therefore be restarted or completed with reports from the new scheme. When either `--min-score`
or `--max-flaky` is requested, any excluded input fails the gate because the requested population
is incomplete. The same fail-closed rule rejects missing shards and inputs that disagree about the
shard count: neither a score nor a flaky count over a partial or inconsistent rotation describes
the selected population.
A shard describes only its slice and cannot prove that a missing mutant was withdrawn. Only an
unsharded, complete population can withdraw identities from an accumulated report.
The source used to render a mutant can differ from the latest source generation in which it was
seen. An unfiltered source presentation takes precedence over a filtered one even when older,
keeping unmatched historical verdicts renderable; a current unfiltered listing provides current
presentations. If a filtered source introduces a mutant absent from that presentation, merge
requires a complete listing for the current source before producing a staged report; otherwise
the new verdict would be lost in a later merge. Merged reports retain each mutant's latest source
sighting separately from the rendered file source, so a later complete report withdraws it only
when that report is at least as recent as the sighting. A merged report's timestamp includes that
sighting even when its mutant was only pending. An unfiltered, unsharded report can still establish
withdrawals.
## Verdicts and scoring
A verdict states what evidence the campaign obtained:
| `killed` | A relevant test failed with the mutant active | Detected |
| `timeout` | The mutant exceeded a confirmed time or stall budget before an assertion rejected it | Undetected |
| `outofmem` | The mutant exceeded its memory ceiling before an assertion rejected it | Undetected |
| `survived` | Every relevant test passed | Undetected |
| `uncovered` | No test reached the mutation site | Undetected |
| `unviable` | The mutation could not compile | Excluded |
| `ignored` | Explicit policy suppressed the mutant | Excluded |
| `notbuilt` | The selected build did not compile that source | Excluded |
| `flaky` | An unmutated nextest enumeration control failed or was inconclusive, or a historical report recorded a flaky verdict | Excluded; retried in later campaigns |
| `pending` | The run ended without judging the mutant | Excluded; makes a requested score or flaky gate incomplete |
The mutation score is:
```text
detected / (detected + undetected)
```
Only `killed` enters the numerator. Timeouts and memory exhaustion establish that the mutant
changed resource behavior, but they remain undetected because no test assertion rejected the
change. Consequently, `--min-score 100` fails closed on either outcome.
Flaky outcomes remain outside that percentage because they are inconclusive, but `--max-flaky`
provides an independent run and merge gate. The default has no flaky budget; a configured value
fails when the current result contains more flaky findings than allowed. Ordinary kills do not
perform a flakiness check, so this gate does not detect intermittent assertion failures after
baseline. Nextest enumeration controls and historical reports can still supply flaky outcomes.
Excluded mutants remain visible but make no claim about test quality because they were never
validly judged.
An `--in-diff` run that selects no mutants is a successful no-op, even with score or flaky gates
enabled. This applies to the final selection after combining diff scope with other filters,
including name filters and sharding; the reason that selection is empty does not affect success.
The exception applies to an empty final selection, not to a nonempty population whose mutants
were all suppressed, unviable, or otherwise ungradeable. Selection errors and builds that got stuck
remain failures.
Otherwise, an empty denominator is printable but not gradeable. A requested score or flaky gate
fails structurally when no mutant was judged rather than treating it as a perfect score or zero
flaky outcomes. Likewise, either gate fails when any mutant remains pending:
neither a score nor a flaky count over only the completed subset describes the selected population.
The distinctions between `survived`, `uncovered`, `unviable`, and `notbuilt` are architectural:
they may all involve no failing test, but they prescribe different action. Collapsing them would
make the report easier to serialize and harder to use correctly.
## Reports and integrations
`cargo gamma run --resume-testing` continues an interrupted mutant-testing
sweep against the already copied and instrumented tree and compiled test
binaries. It does not repeat discovery, copying, building, or baseline measurement.
Selection and execution settings stay frozen, including
mutant ordinals, budgets, resources, oracle, and test arguments. Every durably
recorded verdict is retained; unfinished mutants are retried. Repeated
interruptions are supported. Missing, pre-testing, completed, incompatible, or
corrupt state cause an error instead of a fresh run. Pending execution additionally
requires unchanged prepared artifacts. Live checkout edits do not alter the frozen campaign. Live shard flags are
not validated on resume; the saved campaign's validated shard selection is used.
Prepared snapshot schema version 3 rejects older snapshots, whose reachability or kill-confirmation
policies differ. Start a fresh campaign rather than mixing verdict policies on resume.
If interruption occurs after the last verdict is recorded but before the
completed ledger is durably published, resume still closes the planning phase and displays the
restored testing status. It launches no mutants and preserves prior sweep
accounting before completing the campaign. This publication-only recovery does
not require prepared execution artifacts or host execution capabilities. The
snapshot, journal checksums, campaign identity, and captured report-source
provenance remain validated; current checkout contents cannot replace the
generation that the campaign measured.
Prepared snapshots capture regular-file type and Unix executable bits as
well as content digests. Symlink artifacts record their target spelling;
links to regular files also validate target bytes and executable bits.
Directory symlinks retain both their target spelling and resolved target identity.
Their target must lie in an independently captured real input directory, outside
excluded generated output. The ordinary input scan freezes the target files.
Aliases and ancestor cycles are supported without following directory links.
Links to uncaptured external directories or unscanned linked roots are rejected
with guidance to move the target into a captured input tree. Resume checks this
coverage again, along with link identity and recorded target-file contents.
Captured dynamic libraries remain integrity-checked even beneath generated
target directories. Snapshots freeze each artifact's generated-output
classification at capture time; adding a manifest later cannot turn recorded
source beneath a directory named `target` into exempt output.
Snapshots retain candidate confidence separately from
the public mutant serialization, so resumed diagnostics preserve its buckets.
Resumed diagnostics and advice use the saved worker count, prepared Cargo
profile, and original shard identity, not live execution flags. Snapshots
also retain the selected mutator catalog for diagnostics. Artifact paths, diagnostic verbosity and
redaction remain live reporting options. Live `gamma.toml` supplies only the
`min-score` and `max-flaky` reporting gates, with command-line gates taking
precedence. Its narrow schema ignores values of known execution settings;
unknown top-level keys are rejected, and reporting gates and TOML syntax still
require valid input. Pending tests run
only if the current inherited environment matches
its saved digest; a changed or unrecorded environment refuses resume rather than
mixing verdicts from different execution contexts. The environment digest is
saved independently of compiler identity, even when incremental reuse cannot
be enabled. Recovery with no pending mutants publishes retained verdicts
without requiring the current inherited environment to match.
A failed checkpoint append or sync truncates and syncs back to the prior
journal length; a rollback failure is reported explicitly. Restoring measured
state does not append previously logged verdicts again.
Each new verdict checkpoint also stores cumulative sweep accounting. Resume
adds the unfinished segment's counters and active elapsed time to that saved
prefix; package spans use this same active-time axis, excluding downtime.
Counters sample all worker effort observed at checkpoint publication, including
in-flight work; package mutant counts include only completed verdicts. Work
after the last durable checkpoint cannot be recovered. Older journals without
accounting remain resumable, but their prior sweep costs cannot be reconstructed.
Diagnostic schema 7 includes optional `phases.sweep.accounting` without changing terminal
display. Its frozen `initialPopulation` and `initialExactHints` cover the Pending,
nonzero-ordinal population after compilation and baseline filtering, before assignments.
`initialKillerConfirmed` counts distinct mutants with an observed accepted failure of the
originally recorded test, regardless of execution tier or final canonical failure credit.
Original hint identities remain in the private checkpoint, not the diagnostic report.
The `exact`, `inferred`, `fallback`, and `confirmation` buckets each contain
`launchAttempts`, `executionMs`, `hits`, `misses`, and `inconclusive`. Attempts are runner
invocations, including inconclusive pre-spawn refusals, not physical child creations.
Confirmation work is disjoint; reused executions add zero attempts.
`duplicateLaunchesAvoided` records deduplication, and execution milliseconds sum concurrent
attempts rather than measuring campaign wall time. The private campaign base's atomically
refreshed `gamma-sweep-live.json` contains `schemaVersion: 1`, `sweepWallMs`, and `accounting`.
This telemetry is generated-file/log output only; it adds no screen metrics.
Checksummed accounting-only snapshots replace earlier cumulative counters during journal replay.
Confirmation checkpoints store only newly confirmed identities; replay unions them with the
prepared snapshot and legacy full-set entries. Other checkpoints do not copy the growing set,
and final summaries retain complete identity evidence. Entries are serialized once, preserving
the existing checksum format and durable sync. Existing journals are not rewritten or compacted.
Initial-killer confirmations are deduplicated across resume, unavailable legacy coverage
remains unknown, and `complete` distinguishes partial accounting. Interrupted attempts may
lack finalized duration/result data. Live-file refresh failures increment `liveRefreshFailures`
and preserve the error chain in journal diagnostics; failures to publish the journal propagate.
The snapshot also retains the original report timestamp and shard identity;
completion never substitutes the resume command's selection or current time.
Legacy snapshots lacking this provenance still resume, with an explicit warning.
Their reports use timestamp zero and the non-authoritative population marker
used for merged reports, so they cannot refresh stale evidence or withdraw
mutants from other shards.
The private cache stores a versioned prepared snapshot and a synced verdict
journal independently of the diagnostic progress log. `hints` and `suppress`
can consume this partial knowledge after CTRL+C; pending mutants supply no
verdict. Partial records also merge generalized item, file, and reach ordering
knowledge saved by the interrupted sweep. The verdict journal remains the
authority for outcomes and exact probes. `unsuppress` remains based on current
source policy. Resume appends the
existing progress and selection logs, discarding only incomplete trailing lines
before appending new events, and publishes final reports only after testing
finishes. Reporting flags and gates remain configurable; live
`gamma.toml` and new mutation/testing settings are not applied on resume.
One verdict model feeds every output surface:
- the console emphasizes actionable survivors and exceptional resource or enumeration-control outcomes;
- the mutation-testing-elements JSON report is the interchange artifact;
- the HTML report provides a browsable, self-contained view;
- SARIF and CI annotations place survivors on changed source;
- the diagnostics bundle records campaign phases and measurements;
- the progress journal preserves completed verdict lines if a campaign is interrupted.
Report source content comes from a retained snapshot inside the synchronized campaign workspace,
validated against the digest discovery recorded before publication. The original checkout may
change or delete files after discovery without changing the report. Artifact keys, locations,
diagnostics, console descriptions, and `projectRoot` nevertheless retain original
workspace-relative and original-project identities; cache paths are never published.
`--only-survivors` reads `gamma-report.json` from the effective artifact directory and intersects
its genuine survivor IDs with the population discovered from the current source. The effective
directory is `target/cargo-gamma` by default or the value of `--artifact-dir` when configured.
Timeout and memory-limit outcomes are not selected even though the interchange schema exports them
as `Survived`. This supports focused confirmation after tests are added without carrying any prior
verdict forward.
Command help follows the workspace Cargo-tool convention: green bold headings and usage,
cyan bold literals, cyan placeholders, and package author/version metadata.
The ordinary live display uses one active phase line at a time. Workspace discovery begins with
`Analyzing the workspace` and completes as `Analyzed the workspace`. Compiler convergence replaces
Cargo's invocation-local unit counters with the monotonic number of unviable mutants found. After
baseline measurement it
opens a `Planning` phase, reports how many pending mutants have had their scheduling work
constructed while reachability, hints, and the sweep queue are
prepared, and closes that phase before workers start. `--dashboard` replaces the planning bar and
subsequent testing bar with a multiline, in-place dashboard containing mutant outcomes,
test-selection effectiveness, and test-process costs. The completed
`Planned N/N mutants planned` line remains above the live testing dashboard just as the completed
baseline line does. It uses the same terminal eligibility as `--progress`,
selects small, medium, or large layouts from the current terminal width, adapts when the window is
resized, and redraws no more than once per second.
While sweep workers are quiet, the coordinator emits a one-second heartbeat so a repaint deferred
by that rate limit is eventually flushed and wall-clock-derived values remain current. Matching the
heartbeat to the repaint limit avoids coordinator wakeups that cannot produce a visible update.
The layouts use Unicode box drawing and hierarchy rather than ASCII approximations. Headings,
separators, and outcome rows use semantic color when the resolved `--color` policy permits it and
remain structurally identical without color.
The testing header is the ordinary cargo-style progress line. In the large layout, a mutant panel
and equal-height test-execution panel share the first row, with a centered hints panel beneath.
Each panel derives its width from its longest rendered row, retaining one space of inner padding
rather than reserving a fixed right margin; metric values therefore remain inside the border.
Label and value columns use a consistent three-space gutter across all panels.
The mutant panel uses an adaptive waffle: one cell per mutant through 100 mutants and a fixed
10-by-10 percentage grid above that. The fixed grid keeps large populations readable at ordinary
terminal widths while giving each cell an approximately one-percent meaning; the gutter is the
smallest spacing that keeps label and value columns visually distinct across all layouts. Its
legend reports unviable, pending, killed, survived,
timed-out, out-of-memory, flaky, and uncovered counts in that order, plus the mutation score.
The hints panel reports only explicit and inferred hint counts and hit rates. The execution panel
reports binary count, busy workers, recent throughput, whole and filtered selections,
average and maximum selection runtime, selections per completed mutant, the measured baseline memory peak
when available, and estimated time avoided by successful hints. Execution metric values share one
right edge even when a value, such as recent throughput with its unit, is wider than the usual
numeric column.
Each refresh is emitted as one synchronized terminal update so the previous frame is not visibly
erased before its replacement is ready. The dashboard records the visible width of every rendered
row, moves back by the corresponding physical row count, and erases to the end of the screen.
Recomputing that count at the current terminal width handles resize reflow without relying on the
terminal's global saved-cursor slot, which another program may overwrite or a terminal may not
implement.
`SURVIVED`, `TIMEOUT`, `OUTOFMEMORY`, and `FLAKY` verdicts remain durable one-line announcements
above the dashboard rather than transient table content. When progress is disabled or standard
error is not a terminal, the dashboard is not drawn and the completed summary remains the complete
console account.
The JSON report is written straight from the verdict model rather than rebuilt through a generic
tree first, and is validated in that form. Object keys follow schema declaration order, and
per-file entries follow path order, making output byte-for-byte reproducible across runs.
After findings, one blank line introduces a compact ordered footer: `Summary:`, `Stats :` for an
executed campaign, an actionable `Note :` and indented entries for idle directives, the
conditional suppression note, the conditional hints-promotion reminder, then one `Wrote :` line
per published artifact. The stats line reports elapsed wall time, distinct participating test
binaries, mutant-test subprocess launches, and exact-hint plus inferred-test probes that killed
their mutant. It is absent for dry runs and runs that never reached test execution. All labels
start in column one, use the same width and styled emphasis, and artifact paths use the host's
native separators. Hints promotion is recommended only when the successfully persisted campaign
state is resolvable through the same locator used by the command and would add, correct, or remove
exact killer or compiler-ordering knowledge in the checked-in hints artifact. Generalized
seed/transfer counter churn alone does not trigger the reminder. Suppression is recommended only
when that same resolvable state contains the timeout or out-of-memory outcomes the command will
consume.
Incomplete, truncated, or unsaved output is labelled `warning`. Routine internal adjustments are
silent.
Everything a rendered artifact shows that the repository controls — paths, source fragments, test
names, mutator notes, and a build tool's own diagnostics — is control-character encoded before it
reaches a terminal or a CI log. The campaign writes real escape sequences of its own to draw the
progress display, so a filename able to write them too could erase the line above it, forge a
verdict, or attach a terminal hyperlink to someone else's URL. A build tool's color and bold SGR
parameters are allowed through, followed by a trusted reset that contains the style to the relayed
line. Other presentation effects and every non-SGR terminal control — cursor motion, erasure,
operating-system commands, and the C0/C1 controls including newline — are rendered visibly as an
escape rather than executed.
The HTML report loads no code from anywhere: the viewer is embedded, always. A report carries the
complete source of every file it describes, so loading an external viewer would allow remotely
supplied JavaScript to read and disclose the embedded source and results. The viewer bundle is
vendored, reviewed, and versioned in-tree; a remote artifact could only be trusted if it were
pinned by an integrity digest established here, which the vendored build cannot establish for a
published one. No external-viewer mode is exposed.
A mutant's published reason is built from what the campaign observed about it — an outcome, a
killing test, a memory ceiling — never from a child process's own captured output. When nextest
cannot enumerate a binary's tests with a mutant active, the campaign confirms the mutant caused it
before scoring a kill, but the raw enumeration output stays out of every published report: a test
or a nextest extension inherits this run's environment, and repeating that output verbatim would
extend a run's secret-retention boundary into every uploaded artifact. That output is still raised
locally as a best-effort console diagnostic: only a bounded tail is submitted, and the command-wide
note cap may discard it when earlier diagnostics have filled the queue. Any retained text remains
local to the run.
Reports are projections, not independent calculations. The console, JSON, HTML, merged report, and
score gate must agree because they consume the same outcomes and scoring rules.
Process exit status is another projection of that model: `0` means the command and every requested
gate or expectation passed, `1` is invalid usage or configuration, `2` is a completed command whose
correctness, score, or flaky gate failed, `3` means the requested answer could not be completed, and
`70` marks an internal cargo-gamma panic. An empty final `--in-diff` selection is a successful
no-op even with score or flaky gates enabled. Outside that exception, a score or flaky gate
also fails with `2` when no mutant was judged or selected mutants remain pending; without
either gate those states are reportable results rather than implicit grading policy.
Limits imposed by CI platforms are explicit. Truncating annotations or SARIF silently would make a
successful upload look like the complete result.
## Correctness principles
The architecture is governed by a small set of rules.
### Never improve the score on uncertainty
Ambiguous source stays in scope. Unknown reachability runs more tests. Missing reach evidence runs the
whole binary. Untrusted cache state is ignored. These choices may cost time but cannot hide a
test-suite gap.
### Measure before deriving policy
Timeouts, stall thresholds, memory ceilings, and test ordering derive from the same baseline and
host that judge the mutants. Cross-machine constants would be simultaneously too tight and too
loose.
### Keep policy separate from mechanism
The source engine does not decide verdicts. The process supervisor does not decide score treatment.
The runtime does not know campaign policy. This keeps safety-critical platform behavior from being
entangled with user-facing choices.
### One mutant, one process, one explanation
Exactly one active mutant keeps causality clear. A process failure, timeout, or memory event belongs
to one source change, and a report can explain that change without disentangling interactions among
mutants.
### Source names shared resources; execution policy sizes them
Tests declare stable semantic resource names with `#[gamma::resource("name")]`.
Function declarations require a test attribute such as `#[test]` or
`#[tokio::test]`; inline module declarations apply to the containing test target.
The coordinator discovers declarations through an ignored-only libtest listing, so an ordinary
test whose name resembles the reserved marker encoding cannot acquire resources or alter harness
threading without a resource attribute.
Declarations do not embed a machine-dependent concurrency value. `gamma.toml` and
repeatable command-line overrides assign capacities, with an unspecified
declared resource defaulting to one. Admission applies consistently to
baseline, mutant, and confirmation launches so calibration and verdict
execution observe the same contention policy. Gamma discovers declarations
through ignored test-harness markers and refuses to run when a selected libtest harness
cannot be enumerated, because source text cannot reliably reveal imported or renamed attributes.
Enumeration failures distinguish process launch and containment errors, exit statuses
(including hexadecimal Windows codes), the 30-second timeout, and incomplete or invalid output.
They show bounded `Test Stderr` and `Test Stdout` sections using the same formatting as Cargo
failures, followed by a shell-quoted retry command with its working directory. Configured
environment values are not printed. The workspace and executable are kept automatically;
retry in the same configured test environment before another campaign replaces them.
A custom harness that has no libtest registry cannot contain those markers and contributes no
declarations. A launch that may consume a declared resource runs its libtest harness with one test
thread, keeping the admitted process from consuming the same resource concurrently from several
tests.
### Caches may save time, never supply faith
Durable verdicts require matching source, build, execution context, workspace inputs, and test
evidence. Weaker information may influence order but not outcome. Deleting all cached state changes
performance only.
### Refuse silent configuration failures
Unknown configuration, unmatched selectors, and impossible test filters are errors. A campaign that
quietly ignores user intent can produce a precise score for the wrong population.
### Coverage that a host cannot provide is reported, not skipped
Containment, metering, and terminal-signal handling depend on capabilities supplied by the host
environment, and no host is obliged to offer all of them. Such a test is marked ignored, so that
every runner names it as missing coverage, and fails outright when it is asked for by name on a host
that cannot supply what it needs. A test that returns early and reports success hides the absence of
the coverage, not just the absence of the capability.
Tests that change process-wide state which cannot be restored — an interrupt registry that has been
told a run is ending, a fixed set of watch slots — must run in a process of their own or against an
isolated injected instance. Left in the shared one, they decide what unrelated tests are able to do
next, and which tests those are depends on the harness's scheduling rather than on anything the
suite states. Production-handler exercises therefore run in child processes, while registry and
cgroup lifetime tests use isolated registries with injected recording killers.
### Testing progress
The testing progress display reports completed and total mutants plus observed
verdict counts. It deliberately omits an ETA because scheduling, resource
contention, timeouts, confirmation runs, and learned test selection make a
live completion forecast misleading.
## Costs and limitations
The mutant-schema design makes large campaigns practical, but it is not free.
- **The instrumented build is larger and slower.** Source duplication and guards increase compile
time and may affect inlining and code layout. Instrumented binaries are not suitable for
benchmarking application performance.
- **One bad mutation can affect the shared check.** The rollback loop contains this cost, but each
convergence round is sequential fixed work. Unattributed failures add bounded, target-scoped
proof checks before the final test-binary build.
- **Process launch remains per mutant.** Activating several mutants together would confound
causality, so launch overhead is the floor left after compilation is removed.
- **Survivors remain expensive.** Proving that nothing detects a mutant requires exhausting all
relevant tests. Better-tested code is generally faster to mutation-test because kills terminate
early.
- **Deterministic reachability is an assumption.** Nondeterministic suites may need whole-
binary execution, trading speed for a conservative oracle.
- **Doctests are outside the model.** Rust compiles them as separate programs, which would
reintroduce per-mutant compilation.
- **Guards perturb generated code.** They preserve intended unmutated semantics but alter size and
layout, and may expose stack or compiler limits in unusually deep code.
- **Resource containment depends on the host.** Linux and Windows provide strong process-tree
facilities; other environments may provide less, and requested guarantees are refused when they
cannot be honored. Where no sealed boundary exists, containment silently falls back to best
effort, and a descendant that deliberately leaves its process group can still outlive the run
(on macOS, only once it has also outlived its parent).
Linux additionally treats the memory interface as part of the same capability, so a kernel too
old to expose it falls back to best effort even though its cgroup could have held the subtree.
- **The workspace environment must be sound on platforms without an immutable startup image.**
Linux reads the environment snapshot captured by `exec`; other Unix targets rely on constructor-
time capture and therefore cannot support an earlier native initializer concurrently mutating
the process environment.
The central trade remains favorable for large Rust workspaces: pay fixed compiler-convergence and
test-binary generation costs up front to remove compilation from thousands of mutant decisions,
then spend effort only where evidence is still needed.