# symplex test suite
~245 integration-test files, ~10 300 `#[test]`s, plus the SymPy oracle
described below. Everything runs with `cargo test --tests`; individual files
run with `cargo test --test <file-stem>`.
## Layout
| `v02_oracle_*.rs` | **SymPy oracle for the 0.2 API** (see below). One `#[test]` per fixture subcategory. |
| `v02_oracle_common/mod.rs` | Shared oracle runner (statuses, tolerances, hang guard, strict xfail). Not a test target. |
| `v02_oracle_meta.rs` | Consistency checks on the fixture files (every subcategory has a consumer, ids/keys, size). |
| `test_sympy_cross_validation.rs` | Original 263-fixture SymPy oracle for the 0.1 surface (`fixtures/sympy_cross_validation.json`). |
| `test_correctness_audit.rs` | Definite-integral / FTC / Gosper / series audit against `fixtures/new_capabilities.json`. |
| `v02_*.rs` (non-oracle) | Feature tests written alongside the 0.2 API (matrices, sets, transforms, …). |
| `test_*.rs` | Unit-style tests per module/feature. |
| `round*_*.rs`, `bugfinder*_*.rs`, `math*_*.rs` | Historical bug-hunting rounds. Many tests here are **intentionally duplicated** across rounds (see "Known overlap"). |
| `proptest_*.rs`, `round3_parser_fuzz.rs`, `fuzz/` | Property-based / fuzz tests. "No panic" *is* the assertion for these. |
| `simplify_perf_test.rs`, `perf_analysis.rs` | Benchmarks. `#[ignore]`d by default; run with `--ignored --nocapture --release`. |
| `ui_tests.rs` | `trybuild` compile-fail tests for the macros (~20 s: compiles a crate). |
| `fixtures/*.json` | Oracle data. Committed, regenerated by `scripts/*.py`, < 2 MB total. |
## The SymPy oracle
Reference values come from SymPy 1.14 in the venv at
`/Users/chris.gorski/repos/math/symplex/.venv` (do not create another one).
### Regenerating fixtures
```sh
PY=/Users/chris.gorski/repos/math/symplex/.venv/bin/python
$PY scripts/generate_v02_fixtures.py # tests/fixtures/v02_cross_validation.json (~45 s)
$PY scripts/generate_sympy_fixtures.py # tests/fixtures/sympy_cross_validation.json (~2 s)
$PY scripts/generate_new_fixtures.py # tests/fixtures/new_capabilities.json (~2 s)
$PY scripts/gen_new_fixtures.py # tests/fixtures/new_features_cross_validation.json
# each accepts --check: exit 1 if the committed file would change
```
All four generators are deterministic (fixed seeds, `sort_keys=True`, no
timestamps), so `--check` is a valid CI step. Every SymPy computation runs
under a per-fixture `SIGALRM` timeout; a fixture whose oracle computation
times out / raises / returns an unevaluated object is **kept** with
`sympy_timeout` / `sympy_error` / `sympy_unevaluated` set, never dropped.
### Statuses (honesty policy)
| `PASS` | symplex agrees with the oracle within tolerance | no |
| `FAIL` | symplex produced a **wrong** value/set/structure | **yes** |
| `KNOWN_BUG` | wrong, but listed in the file's `KNOWN_BUGS` table with a `// BUG:` reason | no — but if it starts passing the test fails (strict xfail) |
| `NOT_IMPLEMENTED` | symplex returned an error / unevaluated form | no (informational) |
| `UNSUPPORTED_API` | no public API for the operation | no (informational) |
| `SKIPPED_ORACLE` | SymPy itself could not produce a reference | no (printed) |
Every fixture runs on its own thread with a 4 s budget; a hang is a `FAIL`
("HANG") so one pathological case cannot stall CI. Each `#[test]` covers one
`(category, subcategory)` and must finish in well under 10 s.
Comparison rules of thumb used by the consumers:
* constants: complex `f64` with relative tolerance `1e-6` (special functions
`1e-12`, plus a 18-digit `eval_decimal` check against `N(…, 30)`);
* expressions: evaluated at 3–4 exact rational sample points from the fixture;
* antiderivatives / sums with symbolic bounds: compared through
differences or brute force at `n = 1..6`, never by string;
* sets / inequalities: membership at ~20 sample points;
* boolean logic: full 16-row truth tables;
* multi-valued results (`sqrt_mod`, `primitive_root`, `nroots`, eigenvalues):
verified by congruence / membership / sorted multiset, not by SymPy's choice.
### Recording a library bug
1. Add `(category, subcategory, key, reason)` to the consumer's `KNOWN_BUGS`
with a `// BUG:` comment stating the wrong and the correct answer.
2. Add an `#[ignore = "BUG: …"]` reproducer test next to it.
3. When the bug is fixed the oracle test fails with `UNEXPECTED_PASS` —
remove the entry and un-ignore the reproducer.
Do **not** fix `src/` from a test-campaign branch; report the bug.
## Conventions
* **One concept per test**, named after the mathematical fact it checks.
* **Assert values, not existence.** `assert!(r.is_ok())` and
`eprintln!`-only bodies are not tests; compute the expected value
independently (SymPy in a comment is fine) and assert it.
* **No silent skips.** `if let Ok(v) = … { assert!(…) }` passes vacuously
when the operation fails. Either the operation is expected to succeed
(`expect`) or the test documents that it is optional and reports the skip.
(`if let Some(bad) = check() { panic!() }` — failure iff `Some` — is fine.)
* **Display strings** are a legitimate assertion only when the display *is*
the feature (pretty-printer, LaTeX, canonical form tests). For
mathematical results prefer structural equality or a numeric check.
* **Tolerances** should reflect the math: exact results `1e-12`, `f64`
special functions `1e-10`, truncated series what the remainder bound says.
`1e-2` needs a comment explaining why.
* **Timeouts.** Wrap every cargo/python invocation in `timeout` when
iterating; keep each `#[test]` under 10 s; put exhaustive sweeps behind
`#[ignore]` with a sampled variant that runs by default.
* **Benchmarks are not tests** — `#[ignore]` them with a reason and run them
in `--release` when needed.
## Known overlap
The `round*`, `bugfinder*`, `math*`, `test_cycle*` and `test_stage*` files
were written in independent bug-hunting rounds and contain ~130 groups of
byte-identical test bodies (~930 tests, e.g. `test_stage1` ↔ `test_stage3`,
`test_math_rules` ↔ `test_rule_application`). They are kept deliberately
(cheap, and each round's file is self-contained); do not add new duplicates.