# Testing Weir
Weir's test suite is organized around a **4-vector strategy** that maximizes coverage while minimizing duplication. This document explains the philosophy, the tools, and the practical commands needed to run, extend, and debug tests.
---
## Quick start
```sh
# Fastest check, lib-only unit tests (~1,400 tests, ~20 s)
cargo test --lib
# Full host check including doc tests and examples
cargo test --lib && cargo test --doc && cargo test --examples
# Full parity gate (requires CUDA-capable host + `cpu-parity` feature)
cargo test --features cpu-parity
# Run a specific integration test
cargo test --test df_parity_exact_primitives
# Run property-based tests with expanded cases
PROPTEST_CASES=100000 cargo test --test df_property_bitset_oracles
```
---
## The 4-vector strategy
| **Test Density** | Every public API surface has at least one direct caller in `#[cfg(test)]` or `tests/`. | ~1,443 test functions across `src/**/*.rs` module tests and `tests/*.rs` integration targets. |
| **Performance & Innovation** | Hot paths must have regression baselines; new algorithms must prove they do not regress. | Criterion benches in `benches/` with fixed baselines and regression factors in `README.md` and `PERF.md`. |
| **Deduplication & Legoblock** | No raw Vyre IR construction in production paths; reusable test fixtures are centralized. | `test_harness/` provides `CsrFactory`, `DominatorFixture`, `IfdsProblemBuilder`, `FakeResidentBackend`. |
| **Corpus & Fuzz** | Input-validation boundaries must be exercised by generated inputs, not just hand-written cases. | 9 `libfuzzer` targets in `fuzz/fuzz_targets/` plus proptest suites (`proptest_*` integration tests). |
---
## Test categories
### Unit tests
Embedded `#[cfg(test)]` modules inside `src/`. These test single functions, edge cases, and internal invariants. Run with `cargo test --lib`.
### Integration tests
`tests/*.rs` binaries that exercise full analysis pipelines end-to-end. Many are feature-gated behind `cpu-parity` because they require a CUDA backend for GPU parity verification.
### Property-based tests
Proptest suites (e.g. `tests/proptest_oracle_bitset_laws.rs`) generate thousands of random inputs to verify algebraic laws: commutativity, idempotence, empty-set identity, etc.
### Contract tests
Tests that enforce structural policies rather than numerical correctness:
- `feature_gate_contract.rs`: optional modules are strictly behind feature flags.
- `release_surface_contract.rs`: public API does not leak crate-private types.
- `fixed_point_scratch_contract.rs`: scratch reuse invariants hold across dispatches.
### Doc tests
Rustdoc ` ``` ` examples in `src/lib.rs` and public modules. Verified with `cargo test --doc`.
### Example tests
`tests/examples_build.rs` compiles and runs every example in `examples/` with a 30-second timeout per example.
### Adversarial tests
`tests/df_adversarial_oracles.rs` and `tests/adversarial_stress_suite.rs` feed hostile inputs (empty vectors, length mismatches, `u32::MAX` boundaries) to every oracle and verify well-defined outputs without panic or OOB indexing.
---
## Test harness fixtures
### `FakeResidentBackend` v2
A mock `vyre::VyreBackend` that tracks every allocate/upload/free/dispatch call. It lives in `weir::test_harness::fake_backend` and is gated behind the `test-harness` feature.
```rust
use weir::test_harness::fake_backend::{FakeResidentBackend, InjectedFailure};
let backend = FakeResidentBackend::new();
// or with a custom id:
let backend = FakeResidentBackend::with_id("my_test");
// After running a resident analysis:
assert_eq!(backend.alloc_count(), 3);
assert_eq!(backend.upload_count(), 2);
assert_eq!(backend.dispatch_count(), 1);
// Inspect recorded calls
let dispatches = backend.take_dispatches();
let allocs = backend.take_allocs();
```
**Failure injection**, queue failures for the next operation of a given type:
```rust
backend.inject_next_allocate(InjectedFailure::DeviceOutOfMemory {
requested: 1 << 30,
available: 1 << 20,
});
```
**Lifecycle invariants enforced automatically:**
- Use-after-free panics.
- Double-free panics.
- Ranged upload/download bounds are checked against allocation size.
### `CsrFactory`
Generates random and adversarial CSR graph fixtures with known CPU reachability answers.
```rust
use weir::test_harness::csr_factory::CsrFactory;
let fixture = CsrFactory::random_valid(100, 0.5, 42);
assert_eq!(fixture.node_count, 100);
assert!(!fixture.edge_offsets.is_empty());
// Adversarial shapes
let bad = CsrFactory::adversarial_non_monotonic_offsets();
let oob = CsrFactory::adversarial_oob_targets();
```
### `IfdsProblemBuilder`
Parametric IFDS supergraph generator with a known CPU oracle answer.
```rust
use weir::test_harness::ifds_builder::IfdsProblemBuilder;
let problem = IfdsProblemBuilder::random_valid(
4, // num_procs
8, // blocks_per_proc
4, // facts_per_proc
20, // edge_count
5, // seed_count
2, // summary_count
123, // seed
);
assert!(!problem.expected_reached.is_empty());
```
Adversarial shapes are also available:
```rust
let zero = IfdsProblemBuilder::adversarial_zero_dimensions();
let overflow = IfdsProblemBuilder::adversarial_overflow_shape();
```
### `DominatorFixture`
Random CFG generator with a known immediate-dominator map computed via Lengauer–Tarjan.
```rust
use weir::test_harness::dominator_fixture::DominatorFixture;
let cfg = DominatorFixture::random_dag(50, 0.3, 99);
assert!(cfg.idoms.contains_key(&1));
// Irreducible (cyclic) CFGs
let irred = DominatorFixture::random_irreducible(50, 0.2, 99);
```
---
## Adding a new analysis family
Weir's bitset fixed-point analyses (`reaching`, `live`, `points_to`, `slice`) share a single generic driver ladder. The `define_analysis_family!` macro generates the full wrapper suite.
### 1. Implement the core pieces
You need:
- A `Program` builder: `fn my_step(shape: ProgramGraphShape, input: &str, output: &str) -> Program`
- A graph prep function: `fn prepare_my_graph_with_scratch(..., scratch: &mut FixedPointScratch) -> Result<FixedPointForwardGraph, String>`
- A `FixedPointAnalysisKind` discriminant.
### 2. Invoke the macro
```rust
crate::define_analysis_family! {
family: MyAnalysisFamily,
kind: crate::fixed_point_graph::FixedPointAnalysisKind::MyAnalysis,
name: "my_analysis_closure",
program_builder: crate::my_analysis::my_analysis_step,
graph_prep: crate::my_analysis::plan::prepare_my_graph,
graph_prep_scratch: crate::my_analysis::plan::prepare_my_graph_with_scratch,
input_buffer: "fin",
output_buffer: "fout",
closure_prefix: my_analysis_closure,
resident_prefix: my_analysis_closure_resident,
word_capacity: "my_analysis_closure",
frontier_output: "frontier",
resident_capacity: "my_analysis_closure resident",
resident_output: "frontier",
visibility: pub,
}
```
### 3. What the macro generates
- `MyAnalysisFamily` marker type + `AnalysisFamily` trait impl.
- `my_analysis_closure_via`: owned-dispatch entry point.
- `my_analysis_closure_borrowed_via`: borrowed-dispatch entry point.
- `my_analysis_closure_borrowed_into_*` variants (caller-owned result storage).
- `my_analysis_closure_prepared_*` variants (pre-built graph hot path).
- `my_analysis_closure_plan_*` variants (pre-built plan hot path).
- `my_analysis_closure_resident_plan_with_scratch*`: resident GPU closure variants.
- `my_analysis_closure_resident_plan_with_reusable_frontier_scratch*`: reusable frontier variants.
All generated wrappers preserve the same names as the original hand-written ones so call sites do not change during migration.
### 4. Seal the trait
`AnalysisFamily` is sealed (`pub(crate) mod private { pub trait Sealed {} }`). Only families defined inside the `weir` crate can implement it, ensuring that downstream consumers cannot accidentally violate the invariants shared by the generic drivers.
---
## Running the full test suite
```sh
# 1. Lib-only (fastest, no GPU required)
cargo test --lib
# 2. Doc tests
cargo test --doc
# 3. Examples
cargo test --examples
# 4. Integration tests that do NOT require cpu-parity
cargo test --tests -- --skip df_ --skip parity --skip gpu
# 5. Full parity gate (requires CUDA runtime + `cpu-parity`)
cargo test --features cpu-parity
# 6. Fuzz targets (individual)
cargo fuzz run csr_normalize
cargo fuzz run fixed_point_converge
cargo fuzz run ifds_shape_validate
# 7. Property-based regression replay
PROPTEST_DISABLE_FAILURE_PERSISTENCE=1 cargo test proptest_
```
**CI gate:** Every subagent must run `cargo test --lib` and `cargo clippy --lib -D warnings` before returning.