# Assertion coverage
Line coverage shows which code ran. Assertion coverage shows which of that code
the tests would catch breaking.
## A passing test can miss a wrong result
This function confirms an order and calculates its total:
```js
export function checkout(price, quantity) {
const total = price * quantity;
return { status: 'confirmed', total };
}
```
The test checks the status, but not the total:
```js
import assert from 'node:assert/strict';
import test from 'node:test';
import { checkout } from './checkout.js';
test('confirms an order', () => {
const order = checkout(25, 2);
assert.equal(order.status, 'confirmed');
});
```
Every line runs and the test passes. It would also pass if `total` were
`undefined`, so the line computing it is covered but not asserted.
## What counts as asserted
A statement is asserted when changing it would make at least one passing test
that runs it fail: an assertion fails, test or helper code throws, the test
times out, or the process crashes. The change depends on the statement:
| `if` (and `elif`, `elsif`, `unless`, `else if`) | the condition inverted, each way separately (asserted if either is caught); a Rust `if let` is made not to match |
| `return x`, and a block's value in Ruby, Rust and Kotlin | returns a stand-in for nothing, without evaluating `x` |
| a `when`, `switch`, `try`, `loop` or Kotlin `throw` a function ends on | every value it returns becomes that stand-in (skipping it would leave the function nothing to return) |
| declaration or assignment with a value | the value becomes that stand-in, without being evaluated |
| JSX expression | the value becomes undefined |
| `return true`/`false`, or a declaration of `true`/`false` (JavaScript, TypeScript, Python) | the boolean becomes the other one: `false` becoming `undefined` changes nothing a truthiness check reads |
| `return undefined` or `return None` | skipped, since returning the stand-in would be the same code |
| `throw` in TypeScript | throws a bare `new Error()` instead: deleting a throw seldom compiles, because it usually guards the narrowing the next line relies on |
| anything else | skipped: the statement does not run |
A constructor's `super(...)` is not assessed: skipping it does not compile.
The stand-in is `undefined` in JavaScript and TypeScript, `None` in Python and
`nil` in Ruby. Go and the JVM languages replace a value after it is computed,
as their mutation testers do: the zero value in Go, and `null`, `0` or `false`
in Java and Kotlin. Rust has no value that fits every type, so it takes a
different value of the same type, from the function's signature or the
`let`'s annotation where there is one: a boolean negated, an `Ok` turned into
an `Err` and back, a literal swapped (`false` for `true`, `1` for `0`), an
`Ordering` reversed, `None`, `""` or `0`. Where nothing is annotated, the type
comes from the crate's own signatures, the parameters and `let`s in scope, or
a `match`'s arms (`let (major, text) = numeric(text)?;` takes `numeric`'s
return type): a tuple gets one element changed, an enum another of its
variants, a struct literal one field, a closure its result, integer
arithmetic its lowest bit flipped, a comparison its order reversed, a slice
or split the empty one, and `let Some(x) = .. else` gets `None`. A value of a
type the crate defines is built from its own constants, constructors and
enums (`Err(Error::new(ErrorKind::Empty))`, `Ok(Version::new(0, 0, 0))`).
`Default::default()` is left only where nothing says what the type is.
Imports, declarations without a value, and function, class and type
declarations are not assessed; in Python neither are `pass`, `global`,
`nonlocal` and docstrings, and in Ruby neither are the calls that declare
(`require`, `attr_reader`, `private`, `include` ...). **Assertions** is the
share of the executed statements that are asserted.
## Assess and read
```bash
npx supercov -- npm test
npx supercov runs latest assertions assess
npx supercov runs latest assertions
```
`assertions assess` works it out and saves the result to the run. For each test,
Supercov renders what the test ran -- its code and the source it executed --
and asks [Jev](https://typesafe.ai) whether the test would fail for each
statement's change. In JavaScript, TypeScript and Python the source shown is
the function each asked statement is in; in the other languages it is all the
test ran in the statement's file, with the helper files the test imports. It prints an estimate first; `--dry-run` sends nothing.
The estimate gives the first round and the most the pass can send, which is
what it sends when no test is judged to catch anything.
It needs a TypeSafe AI API key in `TYPESAFE_API_KEY`.
`assertions` reads the saved result with no network and no key, a page at a
time, from the run down to one statement:
```bash
npx supercov runs latest assertions # the share, and the files by statements not asserted
npx supercov runs latest assertions src/adapters.ts # that file's statements not asserted
npx supercov runs latest assertions src/adapters.ts:100 # one statement: each asked test's answer
npx supercov runs latest assertions --test "closes the stream" # what one test catches, and what it misses
```
```text
Run run_4f2a: 93.8% asserted (2731 of 2913 executed statements)
Files, most statements not asserted first: a statement is not asserted when none of the tests asked about it was judged to fail if it changed. Up to 18 of the tests that run a statement are asked, so a test never asked can still catch it.
NOT ASSERTED ASSERTED FILE
41 388/429 src/adapters.ts
...
showing 1-20 of 57
next page: npx supercov runs 'run_4f2a' assertions --offset 20
inspect a file: npx supercov runs 'run_4f2a' assertions 'src/adapters.ts'
```
Each listing ends with the command for its next page and for the next step
down, so an agent reads only as much as it needs. `runs latest file` and
`runs latest line` show the same verdicts beside the coverage, and
`npx supercov runs latest` and the HTML report show the same share.
A statement that is not asserted is a place to add a check: tests run it, but
nothing the asked tests assert depends on what the statement does. Five of the
tests that run a statement are asked first, and up to 15 when none of those
catches it: tests whose file quotes text the statement writes (a log line, a
message), then tests named like the source file, then one per test file and
describe block. When none of the 15 catches it, up to three more are asked, the
ones most likely written for the statement: tests whose name shares words with
the statement's line (`corsOrigin: a false value` for
`return corsOrigin(origin)`), and among those the tests that ran the least
other code. A statement many tests run can still be caught by one that was
never asked; the statement view (`assertions <file>:<line>`) says how many were,
and names the test files none of whose tests were asked, which is where to look
before writing a test that may already exist.
An asserted statement is a judgment too: some test was judged to fail if it
changed, and the pass stops at the first that is. How strong that answer was
says how far to trust it. Checked by applying each change and running the
tests, an asserted verdict was wrong 22 times in 1,169 on an answer of 75% or
more, and 33 times in 711 on a weaker one. The overview says how many of the
asserted rest on a weaker answer, and `--weak` lists them:
```bash
npx supercov runs latest assertions --weak # the files by statements asserted on a weak answer
npx supercov runs latest assertions src/adapters.ts --weak # those statements, the least certain first
```
What reads as covered and is not is most often a deferred effect: a flag set
or an entry removed whose only consequence is what happens on a later event,
run by a test that asserts on the immediate reply.
## Which tests check a change
`npx supercov runs latest tests affected` uses the assessment to say which of
the tests a change reaches were judged to catch it, and lists them first;
`assertions assess --changed` asks every test that ran the changed code. See
[Find the tests a change affects](cli.md#find-the-tests-a-change-affects).
## Later runs ask only about what changed
Answers are kept in `.supercov/assertions/`. Each is stored under the exact text
Jev saw for that statement and test, and reused only when a new run would ask
the same question with the same text. Nothing decides whether a change could
matter: a change to the statement, to code around it that the test ran, to the
test, or to which tests run it, asks again. A dependency or
configuration change asks everything again. After a typical commit only a few
dozen questions are asked; the rest are reused.
A reused answer is the one Jev gave before, right or wrong. When you have shown
an answer wrong, `assess --refresh` asks every question again and keeps the new
answers.
## Cost, speed and accuracy
- **Cost:** about 0.006 cents per statement for a first assessment: 14 projects
in six languages (8,356 statements) cost $0.48 together. A project of 3,000
statements costs about $0.18; a later commit usually well under a cent.
- **Speed:** about 20 seconds for 1,500 statements with 24 requests in flight
(`--workers`, default 24; it changes how long a pass takes, not what it
costs).
Measured on four JavaScript and TypeScript projects (7,393 executed statements):
- **Accuracy:** against ground truth from applying each change and running the
tests, on a fresh sample of 78 statements: 69 right; of the 13 not asserted it
found 8, and 8 of the 12 it flagged were truly not asserted. The share it
reported was 84.6% against a true 83.3%.
- **Python:** on h11 (532 executed statements, pytest), every statement checked
the same way: 502 right; of the 67 not asserted it found 45, and 45 of the 53
it flagged were truly not asserted. The share it reported was 90.0% against a
true 87.4%, for about $0.04.
- **Go, Rust, Java, Ruby and Kotlin**, checked the same way. These were used
to shape how changes and tests are shown to Jev, so they are flattering:
| Project | Checked | Right | Flagged (truly not asserted) | Share reported / true |
| --- | --- | --- | --- | --- |
| google/uuid (Go) | 339 | 298 | 34 (28) | 90.0% / 81.4% |
| hashicorp/go-version (Go) | 257 | 253 | 21 (19) | 91.8% / 91.8% |
| dtolnay/semver (Rust) | 200 of a sample of 200 | 190 | 7 (2) | 96.5% / 96.5% |
| apache/commons-cli (Java) | 196 of a sample of 200 | 167 | 49 (33) | 75.0% / 76.5% |
| dry-rb/dry-inflector (Ruby 3.4) | 157 | 148 | 13 (8) | 91.7% / 92.4% |
| sporkmonger/addressable (Ruby) | 198 of a sample of 200 | 182 | 25 (14) | 87.4% / 90.4% |
| square/moshi (Kotlin) | 60 of a sample of 60 | 52 | 9 (6) | 85.0% / 81.7% |
- **Held out:** five projects nothing was tuned on, each assessed once by the
shipped version and its verdicts frozen before any change was run, scored on
a pre-registered sample of 60 statements (and on every statement where the
suite is fast enough):
| Project | Checked | Right | Flagged (truly not asserted) | Share reported / true |
| --- | --- | --- | --- | --- |
| pallets/itsdangerous (Python) | 60 of 60 | 56 | 3 (2) | 95.0% / 91.7% |
| — every statement | 238 of 238 | 224 | 13 (9) | 94.5% / 92.0% |
| Masterminds/semver (Go) | 60 of 60 | 55 | 7 (7) | 88.3% / 80.0% |
| — every statement | 497 of 498 | 472 | 46 (44) | 90.7% / 86.5% |
| hashie/hashie (Ruby 3.4) | 60 of 60 | 57 | 3 (1) | 95.0% / 96.7% |
| — every statement | 700 of 701 | 655 | 26 (10) | 96.3% / 94.4% |
| FasterXML/java-classmate (Java) | 60 of 60 | 58 | 9 (7) | 85.0% / 88.3% |
| bluss/arrayvec (Rust) | 54 of 60 | 48 | 1 (1) | 98.1% / 87.0% |
Assessing all five (2,582 statements) cost $0.15, about 0.006 cents per
statement. Go reads high: Jev takes a dropped error (`return false,
fmt.Errorf(..)` becoming `return false, nil`) as caught where the tests check
only the boolean. Rust reads high too on arrayvec, whose code is generic over
its element type; six of its sampled changes cannot be made, because no value
of an unknown `T` can be written.
The share is a good measure of a project; a single statement's verdict is a
judgment worth checking before acting on it. Supercov does not run mutated code
for this: nothing is executed, and your tests are never slowed.
## Requirements
Runs of any language Supercov measures: JavaScript and TypeScript, Python
(pytest, unittest), Ruby (RSpec, Minitest, test-unit, Cucumber), Go, Rust
(libtest, nextest, doctests), and Java and Kotlin (JUnit, TestNG); and
`TYPESAFE_API_KEY` for `assess`. Reading a saved result needs neither.