# Objective Observability — a design rule for numerical verification
By **Henos D** ([@4ktLuffy](https://github.com/4ktLuffy)) · <henosd19@gmail.com>
`design note` · Written after the v4 comparator was validated against the existing corpus.
Read this before adding a verification case.
**The rule in §3 is the subject of this note. The defect described below is only the story
that motivated it.** The rule is short: a verification objective must depend on the outputs
being verified.
The defect is worth recording because it was invisible for the entire life of the v3
corpus, it was a defect in how we *designed experiments* rather than in any framework or in
the checker, and the thing that concealed it is still a tempting design choice.
---
## 1. Observation
**The purpose of this re-adjudication was not to find new framework defects. It was to
measure whether the v4 comparator changed the meaning of evidence already collected.** The
corpus was re-judged, not re-run: no framework code executed that had not executed before.
Read the tables below as a statement about our evidence, not about the framework.
The v4 comparator was run over an existing 349-case cross-backend corpus. No framework
code changed, no inputs changed, no operation under test changed. The only difference is
that v4 can report a third outcome — `Unchecked` — where v3 could only say pass or fail.
Case verdicts:
| v3 | 340 | — | 9 |
| v4 | 230 | **110** | 9 |
Two things to note before anything else.
**Mismatches are identical.** Nothing that v3 rejected is now accepted. The stricter
comparator lost no detection; it only stopped claiming things it could not support.
**Nothing was discovered about the framework.** 110 cases moved because our own experiment
could not have detected a fault in them, not because their behaviour changed.
At component level, 7,716 of 9,663 comparisons were adjudicated (79.8%); 20.2% abstained.
The per-operation breakdown is where the problem shows up:
| `softmax` (dim 0) | 0 | 97 | **0.0%** |
| `softmax` (dim 1) | 0 | 97 | **0.0%** |
| `softmin` | 0 | 97 | **0.0%** |
| `softmax` on a transposed view | 0 | 98 | **0.0%** |
| `softmax` on a sliced view | 0 | 98 | **0.0%** |
| broadcast divide by row sum | 0 | 95 | **0.0%** |
| `log_softmax` | 97 | 0 | 100.0% |
| `quiet_softmax` | 97 | 0 | 100.0% |
| `max` over a dim | 24 | 71 | 25.3% |
| `max_pool2d` | 180 | 612 | 22.7% |
| `relu` | 50 | 41 | 54.9% |
Six entries at exactly zero. Not one component of the softmax family had ever been
compared. The corpus had reported all of them as clean.
---
## 2. Root cause
The objective being differentiated was
```
J(x) = Σ f(x)
```
For a row-normalizing operation, the outputs of each row sum to one by construction. For a
tensor of shape `[r, c]` with softmax taken along the row axis:
```
J(x) = Σ softmax(x) = r (a constant — one per row, r rows)
```
A constant has zero derivative. Therefore
```
∂J/∂x ≡ 0
```
independently of whether the implementation is correct. The framework returns zeros. The
numerical oracle perturbs an input, observes no change in `J`, and also returns zeros.
Zero agrees with zero, so the comparison succeeds — while carrying no information at all.
The same holds for softmin (a softmax of the negated input) and for dividing a tensor by
its own row sum: both produce rows summing to one, so `J` is again constant. Taking a
transposed or sliced view changes nothing, because normalization is preserved.
### The controls
Two operations in the same sweep, from the same family, verified completely:
- **`log_softmax`** — `Σ log softmax(x)` is a sum of log-probabilities. Not constant.
**100% verified.**
- **`quiet_softmax`** — adds one to the denominator, so outputs deliberately do *not* sum
to one. Not constant. **100% verified.**
These two are the controls that matter. If the abstention were caused by tolerance settings,
tensor shapes, dtype or backend, `log_softmax` and `quiet_softmax` would have abstained
too — same shapes, same data, same comparator, adjacent code. They did not. The property
that separates the two groups is exactly whether the objective is invariant to the
operation.
### What the earlier verdicts meant
A `Pass` from v3 on these cases did not mean *"this operation is correct."* It meant
*"the quantity we differentiated does not depend on this operation."* Those are entirely
different statements, and the comparator had no vocabulary to distinguish them.
The partial abstentions have the same cause in weaker form. `max`, `relu` and `threshold`
have gradients that are structurally zero over part of their domain — a one-hot gradient
for `max`, a half-zero gradient for `relu`. Those components are unobservable for the same
reason; the operation is merely not *entirely* unobservable.
---
## 3. The rule
> **Objective observability.** A verification objective must depend on the outputs being
> verified. If the objective is invariant to the operation under test, no comparison
> against it can distinguish a correct implementation from an incorrect one, and a
> favourable result carries no information.
This is a restatement of the test-oracle problem in a numerical setting, and it is close to
observability in the control-theory sense: a quantity you cannot observe at the output
cannot be inferred from it. Nothing here is new as an idea. It is worth writing down anyway,
because this failure is quiet and looks exactly like success.
### The at-risk pattern, stated generally
Any objective that is invariant — wholly or partly — to the operation under test:
- **Normalizing operations.** Anything whose outputs sum, or are constrained, to a fixed
value: softmax and its variants, `x / sum(x)`, probability simplex projections, attention
weights, partition functions.
- **Telescoping compositions.** A sum over a cumulative difference collapses to its
endpoints, so interior behaviour is unobservable.
- **Antisymmetric compositions.** Terms that cancel pairwise under a symmetric reduction.
- **Round trips.** `g(g⁻¹(x))` reduces to the identity and tests neither direction.
- **Operations with structurally zero gradients** over part of their domain: `max`, `min`,
`relu`, hard thresholds, and any gate. Partially unobservable, and easy to overlook
because the remaining components do produce signal.
The unifying question is not "is this operation exotic?" but **"can a change in this
operation's output change my objective?"**
---
## 4. Detection
A principle nobody can check won't get followed, so there are two mechanisms for catching
this: one automatic, one manual.
### Automatic
The comparator treats an unobservable objective as an absence of evidence rather than as
evidence of correctness:
> If the analytic gradient is identically zero across an entire case, the objective does
> not depend on the operation. Emit `Unchecked`; never `Pass`.
This is what produced the table in §1. Under the three-way rule this falls out naturally:
components at magnitude zero are below the certification floor, so they abstain rather than
certify.
### The diagnostic that made it visible
Abstention plotted against component magnitude, from the same run:
| 0 | 0 | 1635 | 100% |
| 1e-10 … 1e-7 | 0 | 312 | 100% |
| ≥ 1e-4 | 7449 | 0 | **0%** |
Every abstention sits at magnitude ≤ 1e-7 — numerically zero — and nothing abstains above
the floor. That shape is what distinguishes *honest abstention* from *a broken tolerance*.
Run this diagnostic whenever the abstention rate changes:
- Abstentions concentrated at tiny magnitudes → the objective is unobservable there.
Expected, and the fix is §5.
- Abstentions at large magnitudes → **the comparator is misconfigured.** That is a
tolerance bug, not honesty, and must be fixed before any conclusion is drawn.
### Manual, before adding a case
Ask whether the objective can change when the operation's output changes. When in doubt,
compute the analytic gradient once and look at it: an all-zero gradient on well-conditioned
input is the signature. Do not reach for a larger sweep — a wider sweep of an unobservable
objective produces more cases and no more information, which is precisely how 349 cases
accumulated around six operations that were never tested.
---
## 5. Resolution, and what it costs
Replace the scalar objective with a weighted one:
```
J_w(x) = ⟨w, f(x)⟩
```
With a non-constant `w`, a normalizing operation is no longer invisible: `⟨w, softmax(x)⟩`
varies with the individual outputs even though they sum to a constant, so perturbing an
input changes the objective and the derivative is non-trivial.
This restores observability **without weakening the correctness requirement.** The
comparison is still analytic-versus-numerical on the same quantity, adjudicated by the same
rule and the same tolerances. Nothing is relaxed; a previously invisible axis becomes
visible.
### What it costs
- **`w = ones` is exactly the degenerate case.** The scalar objective is the weighted
objective at the worst possible weight. Adding weights does not help unless the weights
are actually varied.
- **A single fixed `w` can be blind to a specific error direction.** If an implementation's
error happens to be orthogonal to `w`, `⟨w, error⟩` is zero and the check is silent
again. This is why multiple independent `w` are used rather than one.
- **The claim narrows.** A weighted check verifies `wᵀJ`, not the full Jacobian. Any
statement about coverage must say which weights were used. "Verified" without a stated
weight set is not a meaningful claim.
- **Cost scales with the weight count.** Each additional `w` is another full set of
perturbations.
---
## 6. Lessons
Three distinct failure categories, worth keeping separate because they are diagnosed and
fixed differently:
| **Framework bug** | the code under test computes the wrong value | a backward pass returning unwritten memory |
| **Oracle bug** | the checker reaches a wrong verdict on correct data | an all-NaN gradient reported as a pass |
| **Experimental-design bug** | checker and framework are both fine, and the experiment could not have detected a fault | this note |
**This finding is the third category.** The framework was not wrong. The comparator
arithmetic was not wrong. The experiment was incapable.
### The two defects compounded
If you take one thing from this note, take this part rather than the softmax result.
The design defect existed for the entire life of the v3 corpus and left no trace, because
**a binary verdict cannot express "this experiment proved nothing."** Forced to choose
between pass and fail on an all-zero comparison, v3 chose pass — the only available answer,
and a wrong one.
So an experimental-design bug hid behind an oracle-design limitation. Fixing either alone
would not have surfaced it:
- A third verdict without re-adjudicating the corpus would have changed future runs and
left the existing claim standing.
- Re-examining the corpus under a binary comparator would have produced the same 340 passes.
The general form: **a verification system that cannot report absence of evidence will
report absence of evidence as evidence of absence.** Any checker with only two outcomes
has this property. That is the argument for the third verdict.
---
## 7. Falsifiers
How we would know this rule is wrong, or the fix insufficient:
- **An operation whose gradient is genuinely zero everywhere.** Correctly abstaining on it
is right, not a defect. If the rule flags such a case as a problem to fix, the rule is
over-applied.
- **An error direction orthogonal to every weight used.** Then a weighted objective
reports agreement on a wrong implementation, and the weight set — not the principle —
needs revising. This is why the weight set is recorded with every claim.
- **Abstention appearing at large magnitudes.** That would indicate a comparator
misconfiguration rather than an unobservable objective, and would invalidate the
diagnostic in §4 rather than confirm it.
- **Verified fractions that do not rise once weights are introduced.** If the softmax
family stays at 0% under a non-constant `w`, the diagnosis in §2 is wrong and something
else is causing the abstention.
---
## Before you add a verification case
Ask one question:
> **If this operation returned the wrong answer, would my objective change?**
If the answer is no — or only for some components — the case cannot detect a fault in the
part that does not move, and a favourable result from it means nothing. Fix the objective
before widening the sweep.