# How glmm is validated
## Two independent reference engines
Every validated model is fit three ways: on the same data, with the same
formula, in R `lme4` and in Julia `MixedModels.jl`, then compared against
`glmm`. Two independently-implemented engines agreeing with each other
within tolerance is the truth condition `glmm` is held to — not agreement
with either engine alone.
## The references are frozen; disagreement is documented
The reference data and reference results are frozen and never move to
accommodate `glmm`. A reference result is only regenerated when the
reference's own model spec is proven wrong (wrong formula, family, or link),
and that requires a recorded justification. Tolerances are never relaxed to
make `glmm` pass.
The comparison is a reference check rather than a pass/fail gate. Where
`glmm` and a reference disagree beyond the agreement band, the disagreement
has to be written up and registered in
[`../validation/divergences.json`](../validation/divergences.json) before it
passes; an unregistered one, one that outgrows its registered magnitude, and
a registered one that stops firing all fail the run. Every reported
divergence is printed with its direction, so none of this is silent. Where
the two reference engines disagree with each other beyond tolerance, that is
recorded as a flag to investigate — the harness never silently picks
whichever one is closer to `glmm`, and no `glmm`-side entry may excuse it.
## What is covered
The suite spans Gaussian, binomial, Poisson, and Gamma models across a
range of random-effect shapes (intercept-only, correlated slopes, nested
and crossed grouping, canonical and non-canonical links, dense and sparse
routing). The exact dataset list and per-dataset model
spec is the single source of truth in
[`../validation/manifest.json`](../validation/manifest.json); prior (case)
weights get their own manifest rungs (`tier: "weights"`, rungs 29-43).
The Python and R packages are not compared against lme4/MixedModels.jl
directly — they wrap the same Rust kernel, so they are gated against the
Rust engine's own results at a round-off tolerance, confirming the wrapper
introduces no numerical drift.
## Tolerances and known exemptions
Tolerances are per-quantity, not a single global threshold, because point
estimates and standard errors have different natural agreement bands: fixed
effects and variance-component standard deviations are compared at a
relative tolerance, the log-likelihood at an absolute tolerance on the
shared scale (looser for GLMM than LMM, since GLMM compares two different
Laplace-approximation optimizers on the same objective rather than a
near-exact profiled criterion), and standard errors similarly at a relative
tolerance. All bands were set from the measured worst case at freeze plus a
margin, and are never widened to accommodate a failure.
The one recorded, permanent exemption is the GLMM standard-error method
split: `glmm` and lme4 both compute a Hessian-based standard error (which
keeps the θ–β coupling), but MixedModels.jl computes only the Rx variant
(conditional on the estimated variance components). The comparison matches
method to method — Rx against Rx, Hessian against Hessian — rather than
comparing across methods, which would manufacture a spurious disagreement.
Where lme4 and MixedModels.jl disagree with each other on a shared
quantity, that is recorded as a flag for investigation, never resolved by
picking whichever reference happens to sit closer to `glmm`.
## Running it yourself
From `validation/`, `./run.sh` fits `glmm` (Rust) and its Python and R ports
and compares them against the existing reference results on disk — R and
Julia are not refit, so this is the fast path for iterating on `glmm`
itself. `./run.sh --oracles` refits all engines, including the R and Julia
references, for when the oracle itself needs regenerating. See
[`../validation/README.md`](../validation/README.md) for the full directory layout,
result schema, and running instructions.