# Test Suite
rsleigh has a multi-layer test suite spanning unit tests, integration tests, real-binary validation, fuzz testing, and Spectra backend integration tests.
Test counts and timings vary with source revision, feature flags, fixture
availability, and host platform. Check the test runner's executed/skipped counts;
historical totals are not a compatibility or coverage guarantee.
## Running Tests
```bash
# rsleigh — generated decoder harness
make test # generate + run test-harness
# decompiler library and agent-facing CLI contracts
cargo test --release -p rsleigh-decompile --lib --tests
cargo test --release -p rsleigh-cli --lib --tests
# rsleigh — test-harness only (fast, no codegen)
cargo test --release -p test-harness
# benchmarks
python3 scripts/benchmark.py # function count vs Ghidra baselines
make decomp-bench # local pseudocode quality gate
python3 scripts/decomp-regress.py --binary ./some.bin --sample 12
```
---
## Layer 1: Golden P-code Tests (`test-harness`)
Fresh decoder generation is part of release validation. Subtables must emit
P-code in SLEIGH pattern operand order across repeated parses; declaration
order and randomized hash-map iteration must not change the lift. The focused
regression runs before generation in CI:
```bash
cargo test --release -p rsleigh --no-default-features --test codegen_lift_order
```
CI builds the workspace in release mode so the generated decoder libraries
are reused by the release test suites.
**Test:** `x86_64_golden`
Validates that specific x86-64 instruction byte sequences produce the exact expected P-code operations. Each test encodes known instruction bytes, decodes them, and asserts on:
- Instruction length (bytes consumed)
- Disassembly text
- P-code operation types and varnode values
- Register offsets and sizes
Coverage: MOV, ADD, PUSH, POP, RET, Jcc, JMP, CALL, CMP, SUB, XOR, NOP, LEA, MOV mem, MOV imm, MOVSXD, MOVZX, IMUL, TEST, SHL/SHR/SAR, CMOV, SETCC, CDQ, REP MOVSB, XCHG, NOT, NEG, INC/DEC, BSF/BSR, BT, POPCNT, LZCNT, conditional set instructions, SSE/AVX moves, multi-byte prefixes, sign/zero extension edge cases, and more.
~770 assertions covering correct P-code emission for the x86-64 instruction set.
---
## Layer 2: Edge Case / Robustness Tests
**Test:** `truncated_and_garbage_input`
Validates that all 5 architecture decoders (x86-64, AArch64, ARM32, MIPS32, RISC-V) handle pathological input without panicking:
- Empty byte arrays
- Truncated instructions (1-byte fragments)
- Byte sequences of `0xFF` (all ones)
- Random single bytes (0x00 through 0xFF)
- 5,000 random multi-byte fuzz sequences
This test runs random byte sequences through both the decoder and the decompiler, asserting zero panics across all architectures.
### Agent interface contract
`rsleigh-cli/tests/agent_interface.rs` builds synthetic PE fixtures and invokes
real CLI processes. Coverage includes brief → selected address → JSON/text card,
byte and P-code evidence, instruction/operation pagination against full dumps,
partial decode recovery, invalid inputs, generation identity/checksums,
stale/corrupt/missing artifacts, write failures, semantic selectors, typed raw
origins, reusable snapshots, execution budgets, and bounded dependency queries.
CLI unit tests inject decompiler panics and interrupted artifact publication.
Decompiler tests exercise phi cycles, conservative store forwarding, helper
invocation contexts, missing variables/arguments, native return layouts, and
traversal caps. CI runs decompiler library tests, focused integration suites,
and all CLI unit/integration tests in release mode.
```bash
cargo test --release -p rsleigh-cli --lib --test agent_interface
cargo test --release -p rsleigh-decompile --lib --tests
```
### Deterministic agent RE evaluation
The runner covers 18 tasks using original hand-encoded fixtures across six
native architectures: 14 recovered-fact tasks and four explicit-unresolved tasks.
It checks answers and raw instruction origins over three repetitions in each
of three cache states, including graph equivalence and zero new decode/SSA
work on warm hits. The corpus needs no external compiler, JVM, or hosted model.
```bash
cargo build --release -p rsleigh-cli
cargo run --release -p rsleigh-cli --example agent_re_eval -- \
target/release/rsleigh agent-re-results.json --full-corpus
cargo run --release -p rsleigh-cli --example agent_re_eval -- \
target/release/rsleigh agent-re-cache-results.json --cache-benchmark
```
See the [task catalog](../test-harness/fixtures/agent-re/corpus.md),
[measurement report](agent-re-evaluation.md), and
[completion audit](llm-re-completion-audit.md) for reproducibility, raw results,
and baseline limitations. The `ecfd1cd` audit recorded 162/162 corpus checks and
365 regression tests passing locally; those are revision-specific results.
Origin checks use the same decoder API and are separate from Ghidra oracle
parity. Latency and byte totals are measurements, not portable CI thresholds.
---
## Layer 3: Decompiler Validation
**Test:** `decompiler_validation`
Compiles a C source file with known functions (`add`, `factorial`, `reverse_string`, `main`), then decompiles the resulting binary and validates:
- Function signatures are generated
- Return statements are present
- String literals are recovered from read-only sections
- Function calls are resolved (printf, strcpy, strlen)
- Parameter annotations appear at call sites (`/* format */`, `/* ptr */`)
- DWARF debug info is parsed (macOS dSYM auto-discovery)
Uses the host `cc` compiler; skips gracefully on systems without a C toolchain.
---
## Layer 4: Pseudocode Quality Regression Tests
**Tests:** Quality regression tests covering the 14-point pseudocode audit fixes.
These prevent regressions in pseudocode output quality across architectures:
- **CDQ+IDIV simplification** — verifies signed division doesn't produce 64-bit concatenation noise
- **Sub-register Zext deferral** — array access expressions use correct base pointers
- **Smart array base validation** — only pointer-like names converted to array syntax
- **Call return tracking** — call results inlined correctly without duplication or loss
- **Format string preservation** — variadic args match format specifier count
- **AArch64 stack/prologue noise** — sp[] references and prologue/epilogue lines eliminated
- **Return type inference** — works across all 6 architectures
- **Struct field naming** — heuristic naming produces meaningful field names without debug info
- **Cast removal** — redundant same-type casts eliminated
- **Assignment folding** — sequential assignments to same variable collapsed
- **Memory SSA** — stack-spilled values correctly forwarded through loads
- **For-loop init** — pre-header assignments recovered into for-loop initializers
- **Loop counter naming** — induction variables named i, j, k
- **Named expression substitution** — intermediate named values propagated into expressions
---
## Layer 5: Spectra API Contract Tests
**Tests:** `spectra_decoder_api`, `spectra_decompile_api`, `spectra_analysis_api`, `spectra_multi_arch_decode`
These verify the rsleigh API surface that Spectra depends on:
### Decoder API (`spectra_decoder_api`)
- `Decoder::new()` for all 6 architectures (X86_64, X86_32, AArch64, ARM32, MIPS32, RiscV64)
- `decode()` produces correct disassembly and P-code for known instruction bytes
- `CALL` instruction produces `PcodeOp::Call` (Spectra uses this for recursive descent)
- `addr_size()` returns 4 or 8 per architecture
- Empty/invalid input returns `Err` without panic
### Decompile API (`spectra_decompile_api`)
- `decompile()` from raw instruction tuples produces valid C-like output
- `decompile_with_binary()` with compiled Mach-O produces output with function signatures, return statements, and braces
- Empty instruction list doesn't panic
### Analysis API (`spectra_analysis_api`)
- `extract_function_meta()` extracts calls, strings, behavioral tags, complexity from pseudocode
- `scan_vulns()` detects strcpy/gets/system as HIGH/CRIT severity findings
- Safe code produces zero HIGH/CRIT findings
- `FunctionMeta`, `VulnFinding`, `CallGraphEntry` all implement `serde::Serialize` (Spectra uses JSON transport)
### Multi-Architecture (`spectra_multi_arch_decode`)
- Decode + decompile a minimal function on all 6 architectures
- Each produces non-empty output without panic
---
## Layer 6: Spectra Native Backend Integration
This section describes a separate Spectra checkout, not tests shipped in this
repository. It is optional downstream integration context and is not required
to use or validate rsleigh's CLI.
**Location:** `spectra/src-tauri/tests/native_backend_tests.rs`
These test the exact code paths Spectra uses when `analysis_backend = rsleigh`:
### Decoder Integration
- **`native_decoder_x86_64_sequence`** — Decode a full x86-64 function (push/mov/add/pop/ret), verify instruction lengths, disassembly, P-code ops, and return detection. Tests the `decode_range` loop pattern Spectra uses.
- **`native_decoder_aarch64_function`** — Decode STP/MOV/ADD/LDP/RET AArch64 sequence, verify 4-byte fixed instruction length.
- **`native_decoder_mips32_big_endian`** — Decode MIPS32 big-endian prologue (addiu sp/sw ra/jr ra/nop).
### Decompilation
- **`native_decompile_x86_64_function`** — Decompile from raw bytes, verify function body structure.
- **`native_decompile_with_real_binary`** — Full Spectra flow: compile C → parse Mach-O → symbol lookup → VA-to-file-offset mapping → decode until RET → `decompile_with_binary()` → verify output contains return statements.
### Analysis Pipeline
- **`native_analysis_metadata_extraction`** — `extract_function_meta()` on pseudocode with strcpy/printf/strlen calls. Verify call list, string extraction, behavioral tags, JSON serialization.
- **`native_analysis_vulnscan`** — `scan_vulns()` on code with recv/sprintf/system/gets. Verify HIGH/CRIT severity count, JSON serialization.
### P-code Frontend Contract
- **`pcode_json_round_trip`** — Decode x86-64 instruction, format each P-code op as JSON with address/seq/mnemonic/inputs fields. Validates the frontend contract for Spectra's P-code view.
### Function Discovery
- **`native_function_discovery_macho`** — Compile C with foo/bar/main call chain → extract Mach-O symbols → recursive descent via P-code `Call` ops → verify CALL targets are discovered. Tests Spectra's `open_binary()` function discovery flow.
### End-to-End Pipeline
- **`native_end_to_end_pipeline`** — Full pipeline: compile C with -g → discover functions from symbols → decode each → decompile with binary+DWARF → extract metadata → vulnscan → serialize to JSON. Tests every stage of Spectra's native analysis backend.
---
## Layer 7: Benchmarks
`scripts/decomp-regress.py` is the fast pseudocode regression bench for normal
decompiler iteration. It compiles `test-harness/fixtures/bench/pseudocode_core.c`
at `-O0` and `-O2`, runs `rsleigh` over representative functions, scores empty
bodies, unresolved temporary/name leaks, line volume, return coverage, and
control-flow surface, then compares against the checked-in baseline:
```bash
make decomp-bench
python3 scripts/decomp-regress.py --update-baseline # intentional baseline refresh
python3 scripts/decomp-regress.py --binary ./some.bin --sample 12
```
Reports are written under ignored `results/decomp-bench/`. Use the `--binary`
mode for quick exploratory runs against real samples; baseline comparison is
only applied to the deterministic source-built fixture.
`scripts/benchmark.py` runs rsleigh on all binaries in the test corpus and compares function discovery counts against Ghidra baselines:
| PE x86-64 | main.exe, 4RMMaster.exe, ChocolateFactory.exe, crackme_shroud.exe, etc. |
| PE x86-32 | TRYCRACKME.EXE, FLRSCRNSVR.SCR, masoncrackmev2.exe |
| ELF x86-64 | elf-Linux-x64-bash (stripped, 1,242 functions) |
| Mach-O | Compiled test binaries |
A historical run reported **rsleigh 15 — Ghidra 6** on function discovery
across 21 compared binaries. This is not a current parity or correctness claim;
rerun comparisons for the revision and corpus under investigation.
---
## Real-World Binary Validation
Validated against external binary corpora (not in CI, manual testing):
### MIPS32 (darkerego/mips-binaries)
13 binaries tested: busybox (stripped, 5,405 functions — up from 9 before MIPS discovery fixes), bash, nmap, openssl, tor, curl, wget, tcpdump, lua, htop, netcat, socat, dnsmasq. Mix of statically/dynamically linked, stripped/unstripped, big-endian. PIC indirect call resolution: 77% resolved via GP-relative GOT tracing (423→98 unresolved on busybox).
### AArch64 (polaco1782/linux-static-binaries)
13 binaries tested: tor (10,021 functions), openssl, curl, wget, bash, busybox, tcpdump, dnsmasq, socat, htop, netcat, objdump, readelf. All statically linked and stripped.
---
## CI Pipeline
Four GitHub Actions jobs in [ci.yml](../.github/workflows/ci.yml):
1. **test** — Generate decoders, build the workspace, run the harness, decompiler
unit and focused integration tests, CLI contracts, and deterministic seed,
six-architecture corpus, and card-cache gates.
2. **seh-smc-windows** — Execute the PE64 SEH-SMC fixtures on Windows.
3. **clippy** — Lint core crates and CLI; currently allows lint failures.
4. **check** — Fast compile check on pcode-ir without decoder generation.
```
make test-all # full pipeline
make check # quick compile check
make release # optimized CLI build
make benchmark # function count regression check
```
## Documentation and agent contracts
The existing `agent_interface` CLI tests build a minimal PE fixture and check
brief caps/trust labels, function-card evidence sections, and index artifacts:
```bash
cargo test --release -p rsleigh-cli --lib --test agent_interface
```
For documentation changes, also check the examples against the command routing
in [`cli.rs`](../rsleigh-cli/src/cli.rs), run the relevant
[artifact validators](output-formats.md), and check local Markdown links.
Exercise malformed and mixed-version NDJSON as well as empty valid streams;
filtering out invalid records is not validation. No JVM or Z3 is needed for
these agent-interface checks. See [SMT tests](smt-backend.md#rust-implementation-and-tests)
for optional solver validation.