Module test_effectiveness 

Test Coverage and Test Effectiveness

Test coverage measures the percentage of code executed by a test suite — line, branch, or path coverage. It is cheap to compute and easy to visualize as a single percentage, which also makes it one of the most frequently gamed metrics in software engineering: coverage measures whether code executed during a test run, not whether the test actually checked that the code behaved correctly. Mutation testing answers that gap directly: it deliberately introduces small, artificial faults into the code and checks whether the test suite catches (“kills”) them, and is coverage’s necessary complement, not an optional extra.

Formula

Test coverage %     = lines (or branches) covered / total lines × 100
Mutation kill rate % = mutants killed / mutants total × 100

covered / killed = execution or detection count
total            = denominator (lines, branches, or mutants); zero is undefined

Why it matters

A coverage target with no effectiveness check is a textbook Goodhart’s law setup: the number improves while genuine quality does not. A test suite with high line coverage but a low mutation-kill rate is executing code without meaningfully checking it — pairing the two is the single most effective guardrail against coverage-target gaming.

Example

The topic doc’s enterprise example: a company-wide 95% coverage requirement, enforced as a hard CI gate, coexisted with a mutation-kill rate under 40% across much of the codebase — tests executed code without meaningfully asserting on its behaviour. The revised policy required an 80% mutation-kill-rate threshold for payment and authentication code.

use software_engineering::test_effectiveness::{
    mutation_kill_rate_percent, test_coverage_percent,
};

// A company-wide "95% coverage requirement" met at face value...
let coverage = test_coverage_percent(95.0, 100.0);
assert_eq!(coverage, Some(95.0));

// ...while the mutation-kill rate is "under 40%" on the same code —
// high coverage, weak verification.
let kill_rate = mutation_kill_rate_percent(38.0, 100.0).unwrap();
assert!(kill_rate < 40.0);

// The revised policy's "80% kill-rate threshold" for critical code.
let critical_kill_rate = mutation_kill_rate_percent(85.0, 100.0).unwrap();
assert!(critical_kill_rate >= 80.0);

// An empty denominator leaves both rates undefined.
assert_eq!(test_coverage_percent(1.0, 0.0), None);
assert_eq!(mutation_kill_rate_percent(1.0, 0.0), None);

Pitfalls

• Treating coverage percentage as a direct quality verdict — it measures execution, not verification.
• Writing tests primarily to satisfy a coverage gate, producing the threshold-gaming pattern a coverage-only target invites.
• Disabling or deleting failing tests instead of fixing the underlying problem, which removes real protection while the reported number barely moves.
• Applying a uniform coverage target regardless of code risk, wasting effort on low-risk code and under-investing in critical paths.
• Treating a coverage-mutation gap as evidence of nothing: a large gap between a high coverage number and a low mutation-kill rate is the clearest sign that coverage alone is not telling you what you think.

Sources

• Chapter 4.2, Test coverage and test effectiveness.
• Jia, Yue, and Mark Harman, “An Analysis and Survey of the Development of Mutation Testing,” IEEE Transactions on Software Engineering (2011).

Topic doc: software-engineering-metrics/locales/en-001/chapters/04-02-test-coverage-and-test-effectiveness.md

Functions

mutation_kill_rate_percent

Mutation kill rate as a percentage: mutants killed / mutants total × 100.

test_coverage_percent

Test coverage as a percentage: covered lines (or branches) / total × 100.