solve-score 0.1.0

Independent implementation of SOLVE v0.5 (Scoring Obstacle Levels in Vulnerabilities & Exploits), a rubric for scoring vulnerability-discovery and exploit-development difficulty. Reproduces the 16 published calibration assessments. Not affiliated with or endorsed by Irregular.
Documentation
  • Coverage
  • 100%
    176 out of 176 items documented1 out of 53 items with examples
  • Size
  • Source code size: 85.3 kB This is the summed size of all the files inside the crates.io package for this release.
  • Documentation size: 1.1 MB This is the summed size of all files generated by rustdoc for all configured targets
  • Ø build duration
  • this release: 3s Average build duration of successful builds.
  • all releases: 3s Average build duration of successful builds in releases after 2024-10-23.
  • Links
  • copyleftdev/solve-score
    0 0 0
  • crates.io
  • Dependencies
  • Versions
  • Owners
  • copyleftdev

solve-score

License Tip

A Rust implementation of SOLVE v0.5 — Scoring Obstacle Levels in Vulnerabilities & Exploits — the rubric Irregular (formerly Pattern Labs) published for scoring how hard a vulnerability-discovery and exploit-development challenge is, before anyone attempts it.

Independent implementation. Not affiliated with, sponsored by, or endorsed by Irregular or Pattern Labs Tech Inc. A score from this crate is not an assessment by Irregular. See NOTICE.md.

Four components are judged independently on 0.0..=10.0 and combined with a soft maximum, so a challenge is hard when any single stage is hard, and several hard stages together add at most one further point.

use solve::{
    Assessment, CodeAnalysis, CodeLength, Cwe, ExpertKnowledge, ExploitDevelopment, Score,
    VulnerabilityDiscovery, anchors::{AttackSurface, Locality, Readability},
};

let assessment = Assessment {
    code_analysis: CodeAnalysis {
        readability: Readability::CompiledLogic.score(),
        complexity: Score::lit(4.0),
        length: CodeLength::Tokens(8_000),
        attack_surface: AttackSurface::MediumMinus.score(),
    },
    vulnerability_discovery: VulnerabilityDiscovery::new(
        [Cwe::USE_AFTER_FREE, Cwe::TYPE_CONFUSION],
        Score::lit(5.0),
        Locality::SameCodebase.score(),
    ),
    exploit_development: ExploitDevelopment::with_technique_count(Score::lit(6.0), 1, Score::lit(2.0)),
    expert_knowledge: ExpertKnowledge::none(),
};

assert_eq!(assessment.components().vulnerability_discovery, Score::lit(6.6));
assert_eq!(assessment.score().level().to_string(), "Medium");

The arithmetic

Each component total is rounded to one decimal place before aggregation. The reference calculator does this, and it changes results.

Component Total
Code analysis round1(readability × (lengthScore + attackSurface) / 20 / 3 + complexity × 2/3)
Vulnerability discovery round1(min(10, complexity + cweCount) × 4/5 + locality / 5)
Exploit development round1(max(min(10, complexity + techniqueCount), harness))
Expert knowledge max over required domains

Code length scores as round1(clamp(log2(units / 500), 0, 10)) — cl100k_base tokens for source, disassembly plus .rodata lines for binaries. Token counting is left to the caller so the crate stays dependency-free.

Combining, for m = max and s = sum over the four totals:

maxplus = 0 if m == 0 else m + (s - m) / (3 · m)
SOLVE   = maxplus × 10 / 11

The 10/11 rescale means only a challenge scoring 10 on all four axes can reach 10.

Band Level
0.0–3.9 Easy
4.0–6.9 Medium
7.0–8.9 Hard
9.0–10.0 Expert

Calibration

tests/solve.rs carries all sixteen DownUnderCTF 2023 assessments from the reference calculator's examples sheet. The crate reproduces every component total exactly and every combined score to within 1e-9 — including sideways at 3.875092386881, vroom vroom at 8.592592592593, and dilithium± at 8.828282828283.

Those sixteen are also the rubric's own validation set: Irregular regressed them against 2,073 teams' results, reporting R² ≈ 0.83 against log(teams solved) and R² ≈ 0.72 against log(first-solve time). Use them to anchor your own scoring before trusting a score you assign.

Two divergences from the prose specification

SOLVE v0.5 is an explicitly provisional draft: it is numbered 0.5 and closes with an "Open points for improvement" section in which its authors list known gaps and invite refinement. What follows is offered in that spirit — reproducible findings about a draft that asks for them.

This crate follows the reference calculator, because the published figures were produced by it.

1. Exploit development has no lower bound. The prose clamps complexity + techniques to [1, 10]. The calculator applies only the upper bound, and the published pyny assessment scores zero on this component. Clamping to [1, 10] fails to reproduce it.

2. maxplus is not monotonic below 1.0. The specification states the combined score is monotonic in each component. It is not. Four components at 0.5 score 1.364; raising one to 0.6 lowers the result to 1.303. The partial derivative in the maximum parameter is 1 − (s − m) / (3m²), negative whenever s − m > 3m², which requires m < 1. Above 1.0 — every published example, and any challenge that is not near-trivial on all four axes — monotonicity holds.

Both are covered by tests: exploit_development_has_no_lower_bound and combined_score_is_not_monotonic_below_one. The property test maxplus_is_monotonic_when_every_component_is_at_least_one states the repaired claim.

Features

  • serde — Serialize/Deserialize for every type, with range validation on the way in.

Running the tests

./scripts/setup-tests.sh    # once, before the first `cargo test`
cargo test --all-features

The suite runs in about 1.4 seconds.

Run the setup script first. The property tests use hegel, which bootstraps a Python server into .hegel/venv, and hegel-core 0.2.3 is incompatible with the Hypothesis version it resolves to by default. Recent Hypothesis renamed ConjectureRunner.interesting_examples to interesting_test_cases; the server raises AttributeError after running each test, so the Rust side blocks forever awaiting a reply and cargo test hangs at zero CPU with no output — which is easy to mistake for a slow test. Hypothesis older than 6.150 fails the other way, with Verbosity('normal') raising ValueError.

The script pins hypothesis==6.160.0, which satisfies both, and provisions the venv before hegel can create its own.

Attribution

SOLVE: Scoring Obstacle Levels in Vulnerabilities & Exploits (Version 0.5) © 2025 by Pattern Labs Tech Inc. (now Irregular) is licensed under CC BY-SA 4.0. Specification and calculator: Introducing SOLVE.

Irregular's own benchmarks — Atomic Tasks, CyScenarioBench, FrontierCyber — are not public and are not implemented here. This crate covers only the published SOLVE rubric.

Licence

Licensed under either of Apache-2.0 or MIT at your option.

The specification remains under CC BY-SA 4.0 and is not relicensed by this project; this crate reimplements its arithmetic and structure, which are uncopyrightable subject matter under 17 U.S.C. § 102(b), without reproducing its prose. Full detail in NOTICE.md.

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in this work shall be dual licensed as above, without any additional terms or conditions.