# symplex test suite
~275 integration-test source files (~8 500 `#[test]`s) compiled into **ten
test binaries**, plus the SymPy oracle described below. Everything runs with
`cargo test --tests`; the recommended fast path is
```sh
cargo nextest run # one process per test, per-test timeouts
# from .config/nextest.toml
cargo nextest run -E 'test(/^v03::/)' # one binary
cargo nextest run -E 'test(/^unit::test_parser::/)' # one former file
```
Each former top-level file is now a module of one of the group binaries below,
so a test's full name is `<module>::<test>` inside the `<group>` binary
(`cargo test --test unit test_parser::` runs everything from the former
`tests/test_parser.rs`; `cargo test --test unit poly_` runs every test whose
name contains `poly_`). Why: 277 separately linked debug binaries took
~6.5 min to link and ~13 GB of `target/`; ten take seconds. A new
`v0N.rs` group is added per minor release; everything else goes into an
existing group.
## Layout
| Binary / path | What it is |
|---|---|
| `v04.rs` → `v04/v04_*.rs` | Feature tests written alongside the 0.4 API: `v04_polyhedron` (parametric polyhedron certificates; the emitted Lean is pinned byte-for-byte to `fixtures/polyhedron_certificates.lean`, which was compiled against Mathlib with `linter.style.longLine` on), `v04_polytope` (exact polytopes: vertices, volume, cuts, the bridge to the certificate search). |
| `v03.rs` → `v03/v03_*.rs` | Feature tests written alongside the 0.3 API (`poly_view`, `poly_symbolic_coeffs`, `ratsimp`, `linprog`, `normalforms`, `matrix_ergonomics`, `optimize`, `certificates`, `assumptions_poly`, `user_notes`; 0.3.5: `exact_matrix` — `QMatrix`/`ZMatrix` against textbook Gauss–Jordan, the `Matrix` fast paths, the numeric `linsolve` route, the integer-pivoting simplex against exact KKT conditions). |
| `v03_oracle.rs` → `v03_oracle/v03_oracle_*.rs` | **SymPy oracle for the 0.3 API**: `v03_oracle_poly` (`Poly`/`as_poly`: `as_dict`, degrees, `LC`, `all_coeffs`, exact `eval`, `nroots`; symbolic-coefficient `degree`/`coeff`; `ratsimp` vs `cancel`; `poly_fit_exact`; Brent roots), `v03_oracle_linprog` (exact LP vs `sympy.solvers.simplex`; objective compared exactly, the point checked for feasibility/optimality since LP vertices are not unique), `v03_oracle_normalforms` (column/row HNF, Smith form, integer kernel, lattice determinant, `gcd_many`/`lcm_many`; handles SymPy's dropped zero columns), `v03_oracle_meta` (consistency of `fixtures/v03_cross_validation.json`: consumers, `fixture_count`, unique keys, size). |
| `v02.rs` → `v02/v02_*.rs` | Feature tests written alongside the 0.2 API (matrices, sets, transforms, …). |
| `v02_oracle.rs` → `v02_oracle/v02_oracle_*.rs` | **SymPy oracle for the 0.2 API** (see below). One `#[test]` per fixture subcategory. `v02_oracle_meta` checks the fixture files (every subcategory has a consumer, ids/keys, size). |
| `v02_oracle_common/mod.rs` | Shared oracle runner (statuses, tolerances, hang guard, strict xfail). Not a test target; declared once in `v02_oracle.rs` and once (as `oracle_common`, via `#[path]`) in `v03_oracle.rs`. |
| `unit.rs` → `unit/test_*.rs` | Unit-style tests per module/feature. Includes `test_sympy_cross_validation` (original 263-fixture SymPy oracle for the 0.1 surface, `fixtures/sympy_cross_validation.json`) and `test_correctness_audit` (definite-integral / FTC / Gosper / series audit against `fixtures/new_capabilities.json`). |
| `legacy.rs` → `legacy/{round*_*,bugfinder*_*,math*_*,*_validation,regular_joe_tests,fixpoint_convergence_test}.rs` | Historical bug-hunting rounds and one-off validations. Many tests here are **intentionally duplicated** across rounds (see "Known overlap"). `round3_parser_fuzz` is the parser fuzzer. |
| `proptests.rs` → `proptests/{proptest_*,test_proptest_new,test_quality_props,test_units_proptest}.rs` | Property-based tests (plus `fuzz/` outside this tree). "No panic" *is* the assertion for these. Each module's `<stem>.proptest-regressions` file sits next to its source and is still picked up. |
| `perf.rs` → `perf/{simplify_perf_test,perf_analysis}.rs` | Benchmarks. `#[ignore]`d by default; run with `cargo test --test perf --release -- --ignored --nocapture`. |
| `ui_tests.rs` | `trybuild` compile-fail tests for the macros (~20 s: compiles a crate). Toolchain-pinned; CI runs it separately. |
| `common/mod.rs` | Shared helpers (`assert_math_eq`, …). Declared once per group binary; modules use it via `use super::common;`. |
| `fixtures/*.json` | Oracle data. Committed, regenerated by `scripts/*.py`, < 2 MB total. Modules reach it with `include_str!("../fixtures/…")`. |
### Adding a test file
Create `tests/<group>/<name>.rs` and add
`#[path = "<group>/<name>.rs"] mod <name>;` to `tests/<group>.rs` (keep the
list alphabetical — `cargo fmt` enforces it). Use `use super::common;`
instead of `mod common;`, and `../fixtures/` in `include_str!`. Do **not**
add new top-level `tests/*.rs` files: each one is another linked binary.
## The SymPy oracle
Reference values come from SymPy 1.14 in the venv at
`/Users/chris.gorski/repos/math/symplex/.venv` (do not create another one).
### Regenerating fixtures
```sh
PY=/Users/chris.gorski/repos/math/symplex/.venv/bin/python
$PY scripts/generate_v02_fixtures.py # tests/fixtures/v02_cross_validation.json (~45 s)
$PY scripts/generate_v03_fixtures.py # tests/fixtures/v03_cross_validation.json (~3 s)
$PY scripts/generate_sympy_fixtures.py # tests/fixtures/sympy_cross_validation.json (~2 s)
$PY scripts/generate_new_fixtures.py # tests/fixtures/new_capabilities.json (~2 s)
$PY scripts/gen_new_fixtures.py # tests/fixtures/new_features_cross_validation.json
# each accepts --check: exit 1 if the committed file would change
```
All five generators are deterministic (fixed seeds, `sort_keys=True`, no
timestamps), so `--check` is a valid CI step. Every SymPy computation runs
under a per-fixture `SIGALRM` timeout; a fixture whose oracle computation
times out / raises / returns an unevaluated object is **kept** with
`sympy_timeout` / `sympy_error` / `sympy_unevaluated` set, never dropped.
### Statuses (honesty policy)
| Status | Meaning | Fails the test? |
|---|---|---|
| `PASS` | symplex agrees with the oracle within tolerance | no |
| `FAIL` | symplex produced a **wrong** value/set/structure | **yes** |
| `KNOWN_BUG` | wrong, but listed in the file's `KNOWN_BUGS` table with a `// BUG:` reason | no — but if it starts passing the test fails (strict xfail) |
| `NOT_IMPLEMENTED` | symplex returned an error / unevaluated form | no (informational) |
| `UNSUPPORTED_API` | no public API for the operation | no (informational) |
| `SKIPPED_ORACLE` | SymPy itself could not produce a reference | no (printed) |
Every fixture runs on its own thread with a 4 s budget; a hang is a `FAIL`
("HANG") so one pathological case cannot stall CI. Each `#[test]` covers one
`(category, subcategory)` and must finish in well under 10 s.
Comparison rules of thumb used by the consumers:
* constants: complex `f64` with relative tolerance `1e-6` (special functions
`1e-12`, plus a 18-digit `eval_decimal` check against `N(…, 30)`);
* expressions: evaluated at 3–4 exact rational sample points from the fixture;
* antiderivatives / sums with symbolic bounds: compared through
differences or brute force at `n = 1..6`, never by string;
* sets / inequalities: membership at ~20 sample points;
* boolean logic: full 16-row truth tables;
* multi-valued results (`sqrt_mod`, `primitive_root`, `nroots`, eigenvalues):
verified by congruence / membership / sorted multiset, not by SymPy's choice;
* exact results of the 0.3 API (rational coefficients, LP optima, integer
normal forms) are compared as `Ratio<BigInt>` / `BigInt` **equality**;
symbolic coefficients at 3 parameter points (`1e-9`).
Oracle caveats found while building the v03 fixtures (worked around in
`scripts/generate_v03_fixtures.py`): SymPy 1.14's `simplex.linprog`
mishandles non-default `bounds` (a negative lower bound or a free variable
is still forced `≥ 0`), so `lpmin`/`lpmax` with explicit relational
constraints are the LP reference; `Poly.nroots` does not converge on
repeated roots, so `all_roots(radicals=False)` is used there.
### Recording a library bug
1. Add `(category, subcategory, key, reason)` to the consumer's `KNOWN_BUGS`
with a `// BUG:` comment stating the wrong and the correct answer.
2. Add an `#[ignore = "BUG: …"]` reproducer test next to it.
3. When the bug is fixed the oracle test fails with `UNEXPECTED_PASS` —
remove the entry and un-ignore the reproducer.
Do **not** fix `src/` from a test-campaign branch; report the bug.
## Conventions
* **One concept per test**, named after the mathematical fact it checks.
* **Assert values, not existence.** `assert!(r.is_ok())` and
`eprintln!`-only bodies are not tests; compute the expected value
independently (SymPy in a comment is fine) and assert it.
* **No silent skips.** `if let Ok(v) = … { assert!(…) }` passes vacuously
when the operation fails. Either the operation is expected to succeed
(`expect`) or the test documents that it is optional and reports the skip.
(`if let Some(bad) = check() { panic!() }` — failure iff `Some` — is fine.)
* **Display strings** are a legitimate assertion only when the display *is*
the feature (pretty-printer, LaTeX, canonical form tests). For
mathematical results prefer structural equality or a numeric check.
* **Tolerances** should reflect the math: exact results `1e-12`, `f64`
special functions `1e-10`, truncated series what the remainder bound says.
`1e-2` needs a comment explaining why.
* **Timeouts.** Wrap every cargo/python invocation in `timeout` when
iterating; keep each `#[test]` under 10 s; put exhaustive sweeps behind
`#[ignore]` with a sampled variant that runs by default.
* **Benchmarks are not tests** — `#[ignore]` them with a reason and run them
in `--release` when needed.
## Known overlap
The `round*`, `bugfinder*`, `math*`, `test_cycle*` and `test_stage*` files
were written in independent bug-hunting rounds and contain ~130 groups of
byte-identical test bodies (~930 tests, e.g. `test_stage1` ↔ `test_stage3`,
`test_math_rules` ↔ `test_rule_application`). They are kept deliberately
(cheap, and each round's file is self-contained); do not add new duplicates.