solve-score
A Rust implementation of SOLVE v0.5 — Scoring Obstacle Levels in Vulnerabilities & Exploits — the rubric Irregular (formerly Pattern Labs) published for scoring how hard a vulnerability-discovery and exploit-development challenge is, before anyone attempts it.
Independent implementation. Not affiliated with, sponsored by, or endorsed by Irregular or Pattern Labs Tech Inc. A score from this crate is not an assessment by Irregular. See NOTICE.md.
Four components are judged independently on 0.0..=10.0 and combined with a soft maximum, so a
challenge is hard when any single stage is hard, and several hard stages together add at most one
further point.
use ;
let assessment = Assessment ;
assert_eq!;
assert_eq!;
The arithmetic
Each component total is rounded to one decimal place before aggregation. The reference calculator does this, and it changes results.
| Component | Total |
|---|---|
| Code analysis | round1(readability × (lengthScore + attackSurface) / 20 / 3 + complexity × 2/3) |
| Vulnerability discovery | round1(min(10, complexity + cweCount) × 4/5 + locality / 5) |
| Exploit development | round1(max(min(10, complexity + techniqueCount), harness)) |
| Expert knowledge | max over required domains |
Code length scores as round1(clamp(log2(units / 500), 0, 10)) — cl100k_base tokens for source,
disassembly plus .rodata lines for binaries. Token counting is left to the caller so the crate
stays dependency-free.
Combining, for m = max and s = sum over the four totals:
maxplus = 0 if m == 0 else m + (s - m) / (3 · m)
SOLVE = maxplus × 10 / 11
The 10/11 rescale means only a challenge scoring 10 on all four axes can reach 10.
| Band | Level |
|---|---|
| 0.0–3.9 | Easy |
| 4.0–6.9 | Medium |
| 7.0–8.9 | Hard |
| 9.0–10.0 | Expert |
Calibration
tests/solve.rs carries all sixteen DownUnderCTF 2023 assessments from the reference calculator's
examples sheet. The crate reproduces every component total exactly and every combined score to
within 1e-9 — including sideways at 3.875092386881, vroom vroom at 8.592592592593, and
dilithium± at 8.828282828283.
Those sixteen are also the rubric's own validation set: Irregular regressed them against 2,073
teams' results, reporting R² ≈ 0.83 against log(teams solved) and R² ≈ 0.72 against
log(first-solve time). Use them to anchor your own scoring before trusting a score you assign.
Two divergences from the prose specification
SOLVE v0.5 is an explicitly provisional draft: it is numbered 0.5 and closes with an "Open points for improvement" section in which its authors list known gaps and invite refinement. What follows is offered in that spirit — reproducible findings about a draft that asks for them.
This crate follows the reference calculator, because the published figures were produced by it.
1. Exploit development has no lower bound. The prose clamps complexity + techniques to
[1, 10]. The calculator applies only the upper bound, and the published pyny assessment scores
zero on this component. Clamping to [1, 10] fails to reproduce it.
2. maxplus is not monotonic below 1.0. The specification states the combined score is monotonic
in each component. It is not. Four components at 0.5 score 1.364; raising one to 0.6 lowers
the result to 1.303. The partial derivative in the maximum parameter is 1 − (s − m) / (3m²),
negative whenever s − m > 3m², which requires m < 1. Above 1.0 — every published example, and
any challenge that is not near-trivial on all four axes — monotonicity holds.
Both are covered by tests: exploit_development_has_no_lower_bound and
combined_score_is_not_monotonic_below_one. The property test
maxplus_is_monotonic_when_every_component_is_at_least_one states the repaired claim.
Features
serde—Serialize/Deserializefor every type, with range validation on the way in.
Running the tests
The suite runs in about 1.4 seconds.
Run the setup script first. The property tests use hegel, which
bootstraps a Python server into .hegel/venv, and hegel-core 0.2.3 is incompatible with the
Hypothesis version it resolves to by default. Recent Hypothesis renamed
ConjectureRunner.interesting_examples to interesting_test_cases; the server raises
AttributeError after running each test, so the Rust side blocks forever awaiting a reply and
cargo test hangs at zero CPU with no output — which is easy to mistake for a slow test.
Hypothesis older than 6.150 fails the other way, with Verbosity('normal') raising ValueError.
The script pins hypothesis==6.160.0, which satisfies both, and provisions the venv before hegel
can create its own.
Attribution
SOLVE: Scoring Obstacle Levels in Vulnerabilities & Exploits (Version 0.5) © 2025 by Pattern Labs Tech Inc. (now Irregular) is licensed under CC BY-SA 4.0. Specification and calculator: Introducing SOLVE.
Irregular's own benchmarks — Atomic Tasks, CyScenarioBench, FrontierCyber — are not public and are not implemented here. This crate covers only the published SOLVE rubric.
Licence
Licensed under either of Apache-2.0 or MIT at your option.
The specification remains under CC BY-SA 4.0 and is not relicensed by this project; this crate reimplements its arithmetic and structure, which are uncopyrightable subject matter under 17 U.S.C. § 102(b), without reproducing its prose. Full detail in NOTICE.md.
Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in this work shall be dual licensed as above, without any additional terms or conditions.