# solve-score
[](#licence)
[](https://tokentip.to/@copyleftdev)
A Rust implementation of **SOLVE v0.5** — *Scoring Obstacle Levels in Vulnerabilities & Exploits* —
the rubric Irregular (formerly Pattern Labs) published for scoring how hard a vulnerability-discovery
and exploit-development challenge is, before anyone attempts it.
> **Independent implementation.** Not affiliated with, sponsored by, or endorsed by Irregular or
> Pattern Labs Tech Inc. A score from this crate is not an assessment by Irregular. See
> [NOTICE.md](NOTICE.md).
Four components are judged independently on `0.0..=10.0` and combined with a soft maximum, so a
challenge is hard when *any single stage* is hard, and several hard stages together add at most one
further point.
```rust
use solve::{
Assessment, CodeAnalysis, CodeLength, Cwe, ExpertKnowledge, ExploitDevelopment, Score,
VulnerabilityDiscovery, anchors::{AttackSurface, Locality, Readability},
};
let assessment = Assessment {
code_analysis: CodeAnalysis {
readability: Readability::CompiledLogic.score(),
complexity: Score::lit(4.0),
length: CodeLength::Tokens(8_000),
attack_surface: AttackSurface::MediumMinus.score(),
},
vulnerability_discovery: VulnerabilityDiscovery::new(
[Cwe::USE_AFTER_FREE, Cwe::TYPE_CONFUSION],
Score::lit(5.0),
Locality::SameCodebase.score(),
),
exploit_development: ExploitDevelopment::with_technique_count(Score::lit(6.0), 1, Score::lit(2.0)),
expert_knowledge: ExpertKnowledge::none(),
};
assert_eq!(assessment.components().vulnerability_discovery, Score::lit(6.6));
assert_eq!(assessment.score().level().to_string(), "Medium");
```
## The arithmetic
Each component total is rounded to one decimal place **before** aggregation. The reference
calculator does this, and it changes results.
| Code analysis | `round1(readability × (lengthScore + attackSurface) / 20 / 3 + complexity × 2/3)` |
| Vulnerability discovery | `round1(min(10, complexity + cweCount) × 4/5 + locality / 5)` |
| Exploit development | `round1(max(min(10, complexity + techniqueCount), harness))` |
| Expert knowledge | `max` over required domains |
Code length scores as `round1(clamp(log2(units / 500), 0, 10))` — `cl100k_base` tokens for source,
disassembly plus `.rodata` lines for binaries. Token counting is left to the caller so the crate
stays dependency-free.
Combining, for `m = max` and `s = sum` over the four totals:
```text
maxplus = 0 if m == 0 else m + (s - m) / (3 · m)
SOLVE = maxplus × 10 / 11
```
The `10/11` rescale means only a challenge scoring 10 on all four axes can reach 10.
| 0.0–3.9 | Easy |
| 4.0–6.9 | Medium |
| 7.0–8.9 | Hard |
| 9.0–10.0 | Expert |
## Calibration
`tests/solve.rs` carries all sixteen DownUnderCTF 2023 assessments from the reference calculator's
examples sheet. The crate reproduces every component total exactly and every combined score to
within `1e-9` — including `sideways` at `3.875092386881`, `vroom vroom` at `8.592592592593`, and
`dilithium±` at `8.828282828283`.
Those sixteen are also the rubric's own validation set: Irregular regressed them against 2,073
teams' results, reporting R² ≈ 0.83 against `log(teams solved)` and R² ≈ 0.72 against
`log(first-solve time)`. Use them to anchor your own scoring before trusting a score you assign.
## Two divergences from the prose specification
SOLVE v0.5 is an explicitly provisional draft: it is numbered 0.5 and closes with an "Open points
for improvement" section in which its authors list known gaps and invite refinement. What follows
is offered in that spirit — reproducible findings about a draft that asks for them.
This crate follows the **reference calculator**, because the published figures were produced by it.
**1. Exploit development has no lower bound.** The prose clamps `complexity + techniques` to
`[1, 10]`. The calculator applies only the upper bound, and the published `pyny` assessment scores
zero on this component. Clamping to `[1, 10]` fails to reproduce it.
**2. `maxplus` is not monotonic below 1.0.** The specification states the combined score is monotonic
in each component. It is not. Four components at `0.5` score **1.364**; raising one to `0.6` *lowers*
the result to **1.303**. The partial derivative in the maximum parameter is `1 − (s − m) / (3m²)`,
negative whenever `s − m > 3m²`, which requires `m < 1`. Above `1.0` — every published example, and
any challenge that is not near-trivial on all four axes — monotonicity holds.
Both are covered by tests: `exploit_development_has_no_lower_bound` and
`combined_score_is_not_monotonic_below_one`. The property test
`maxplus_is_monotonic_when_every_component_is_at_least_one` states the repaired claim.
## Features
- `serde` — `Serialize`/`Deserialize` for every type, with range validation on the way in.
## Running the tests
```sh
./scripts/setup-tests.sh # once, before the first `cargo test`
cargo test --all-features
```
The suite runs in about 1.4 seconds.
**Run the setup script first.** The property tests use [hegel](https://hegel.dev), which
bootstraps a Python server into `.hegel/venv`, and hegel-core 0.2.3 is incompatible with the
Hypothesis version it resolves to by default. Recent Hypothesis renamed
`ConjectureRunner.interesting_examples` to `interesting_test_cases`; the server raises
`AttributeError` *after* running each test, so the Rust side blocks forever awaiting a reply and
`cargo test` hangs at zero CPU with no output — which is easy to mistake for a slow test.
Hypothesis older than 6.150 fails the other way, with `Verbosity('normal')` raising `ValueError`.
The script pins `hypothesis==6.160.0`, which satisfies both, and provisions the venv before hegel
can create its own.
## Attribution
SOLVE: Scoring Obstacle Levels in Vulnerabilities & Exploits (Version 0.5) © 2025 by Pattern Labs
Tech Inc. (now [Irregular](https://www.irregular.com/)) is licensed under
[CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/). Specification and calculator:
[Introducing SOLVE](https://www.irregular.com/publications/introducing-solve).
Irregular's own benchmarks — Atomic Tasks, CyScenarioBench, FrontierCyber — are not public and are
**not** implemented here. This crate covers only the published SOLVE rubric.
## Licence
Licensed under either of [Apache-2.0](LICENSE-APACHE) or [MIT](LICENSE-MIT) at your option.
The *specification* remains under CC BY-SA 4.0 and is not relicensed by this project; this crate
reimplements its arithmetic and structure, which are uncopyrightable subject matter under
17 U.S.C. § 102(b), without reproducing its prose. Full detail in [NOTICE.md](NOTICE.md).
Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in
this work shall be dual licensed as above, without any additional terms or conditions.