solve-score 0.1.0

Independent implementation of SOLVE v0.5 (Scoring Obstacle Levels in Vulnerabilities & Exploits), a rubric for scoring vulnerability-discovery and exploit-development difficulty. Reproduces the 16 published calibration assessments. Not affiliated with or endorsed by Irregular.
Documentation
# solve-score

[![License](https://img.shields.io/badge/license-MIT%20OR%20Apache--2.0-blue)](#licence)
[![Tip](https://img.shields.io/badge/tip-tokentip.to-6f42c1)](https://tokentip.to/@copyleftdev)

A Rust implementation of **SOLVE v0.5** — *Scoring Obstacle Levels in Vulnerabilities & Exploits* —
the rubric Irregular (formerly Pattern Labs) published for scoring how hard a vulnerability-discovery
and exploit-development challenge is, before anyone attempts it.

> **Independent implementation.** Not affiliated with, sponsored by, or endorsed by Irregular or
> Pattern Labs Tech Inc. A score from this crate is not an assessment by Irregular. See
> [NOTICE.md]NOTICE.md.

Four components are judged independently on `0.0..=10.0` and combined with a soft maximum, so a
challenge is hard when *any single stage* is hard, and several hard stages together add at most one
further point.

```rust
use solve::{
    Assessment, CodeAnalysis, CodeLength, Cwe, ExpertKnowledge, ExploitDevelopment, Score,
    VulnerabilityDiscovery, anchors::{AttackSurface, Locality, Readability},
};

let assessment = Assessment {
    code_analysis: CodeAnalysis {
        readability: Readability::CompiledLogic.score(),
        complexity: Score::lit(4.0),
        length: CodeLength::Tokens(8_000),
        attack_surface: AttackSurface::MediumMinus.score(),
    },
    vulnerability_discovery: VulnerabilityDiscovery::new(
        [Cwe::USE_AFTER_FREE, Cwe::TYPE_CONFUSION],
        Score::lit(5.0),
        Locality::SameCodebase.score(),
    ),
    exploit_development: ExploitDevelopment::with_technique_count(Score::lit(6.0), 1, Score::lit(2.0)),
    expert_knowledge: ExpertKnowledge::none(),
};

assert_eq!(assessment.components().vulnerability_discovery, Score::lit(6.6));
assert_eq!(assessment.score().level().to_string(), "Medium");
```

## The arithmetic

Each component total is rounded to one decimal place **before** aggregation. The reference
calculator does this, and it changes results.

| Component | Total |
|---|---|
| Code analysis | `round1(readability × (lengthScore + attackSurface) / 20 / 3 + complexity × 2/3)` |
| Vulnerability discovery | `round1(min(10, complexity + cweCount) × 4/5 + locality / 5)` |
| Exploit development | `round1(max(min(10, complexity + techniqueCount), harness))` |
| Expert knowledge | `max` over required domains |

Code length scores as `round1(clamp(log2(units / 500), 0, 10))` — `cl100k_base` tokens for source,
disassembly plus `.rodata` lines for binaries. Token counting is left to the caller so the crate
stays dependency-free.

Combining, for `m = max` and `s = sum` over the four totals:

```text
maxplus = 0 if m == 0 else m + (s - m) / (3 · m)
SOLVE   = maxplus × 10 / 11
```

The `10/11` rescale means only a challenge scoring 10 on all four axes can reach 10.

| Band | Level |
|---|---|
| 0.0–3.9 | Easy |
| 4.0–6.9 | Medium |
| 7.0–8.9 | Hard |
| 9.0–10.0 | Expert |

## Calibration

`tests/solve.rs` carries all sixteen DownUnderCTF 2023 assessments from the reference calculator's
examples sheet. The crate reproduces every component total exactly and every combined score to
within `1e-9` — including `sideways` at `3.875092386881`, `vroom vroom` at `8.592592592593`, and
`dilithium±` at `8.828282828283`.

Those sixteen are also the rubric's own validation set: Irregular regressed them against 2,073
teams' results, reporting R² ≈ 0.83 against `log(teams solved)` and R² ≈ 0.72 against
`log(first-solve time)`. Use them to anchor your own scoring before trusting a score you assign.

## Two divergences from the prose specification

SOLVE v0.5 is an explicitly provisional draft: it is numbered 0.5 and closes with an "Open points
for improvement" section in which its authors list known gaps and invite refinement. What follows
is offered in that spirit — reproducible findings about a draft that asks for them.

This crate follows the **reference calculator**, because the published figures were produced by it.

**1. Exploit development has no lower bound.** The prose clamps `complexity + techniques` to
`[1, 10]`. The calculator applies only the upper bound, and the published `pyny` assessment scores
zero on this component. Clamping to `[1, 10]` fails to reproduce it.

**2. `maxplus` is not monotonic below 1.0.** The specification states the combined score is monotonic
in each component. It is not. Four components at `0.5` score **1.364**; raising one to `0.6` *lowers*
the result to **1.303**. The partial derivative in the maximum parameter is `1 − (s − m) / (3m²)`,
negative whenever `s − m > 3m²`, which requires `m < 1`. Above `1.0` — every published example, and
any challenge that is not near-trivial on all four axes — monotonicity holds.

Both are covered by tests: `exploit_development_has_no_lower_bound` and
`combined_score_is_not_monotonic_below_one`. The property test
`maxplus_is_monotonic_when_every_component_is_at_least_one` states the repaired claim.

## Features

- `serde` — `Serialize`/`Deserialize` for every type, with range validation on the way in.

## Running the tests

```sh
./scripts/setup-tests.sh    # once, before the first `cargo test`
cargo test --all-features
```

The suite runs in about 1.4 seconds.

**Run the setup script first.** The property tests use [hegel](https://hegel.dev), which
bootstraps a Python server into `.hegel/venv`, and hegel-core 0.2.3 is incompatible with the
Hypothesis version it resolves to by default. Recent Hypothesis renamed
`ConjectureRunner.interesting_examples` to `interesting_test_cases`; the server raises
`AttributeError` *after* running each test, so the Rust side blocks forever awaiting a reply and
`cargo test` hangs at zero CPU with no output — which is easy to mistake for a slow test.
Hypothesis older than 6.150 fails the other way, with `Verbosity('normal')` raising `ValueError`.

The script pins `hypothesis==6.160.0`, which satisfies both, and provisions the venv before hegel
can create its own.

## Attribution

SOLVE: Scoring Obstacle Levels in Vulnerabilities & Exploits (Version 0.5) © 2025 by Pattern Labs
Tech Inc. (now [Irregular](https://www.irregular.com/)) is licensed under
[CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/). Specification and calculator:
[Introducing SOLVE](https://www.irregular.com/publications/introducing-solve).

Irregular's own benchmarks — Atomic Tasks, CyScenarioBench, FrontierCyber — are not public and are
**not** implemented here. This crate covers only the published SOLVE rubric.

## Licence

Licensed under either of [Apache-2.0](LICENSE-APACHE) or [MIT](LICENSE-MIT) at your option.

The *specification* remains under CC BY-SA 4.0 and is not relicensed by this project; this crate
reimplements its arithmetic and structure, which are uncopyrightable subject matter under
17 U.S.C. § 102(b), without reproducing its prose. Full detail in [NOTICE.md](NOTICE.md).

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in
this work shall be dual licensed as above, without any additional terms or conditions.