Module provider 

The provider boundary. skilltest never talks to a model directly; a Provider runs the skill, plays the simulated user, and judges the transcript.

There are two real implementations. OneharnessProvider (the default) runs each prompt on a harness through the oneharness CLI and parses its JSON. CommandProvider speaks a small JSON-lines protocol (see docs/protocol.md) and backs both the deterministic skilltest-fake-provider used by the gate and any custom provider you write. The Provider trait also lets the runner be unit-tested against an in-memory fake.

Structs

ApiJudgeProvider

A judge-only Provider that scores evals and plays the simulated user with a direct model API call (Anthropic or OpenAI), rather than running them through a harness.

AssistantTurn

An assistant/skill turn produced by the provider.

CommandProvider

A Provider backed by an external command speaking the JSON protocol.

JudgeQuery

A judge query: the criterion, its kind, and (for numeric) the scale.

JudgeVerdict

A judge verdict: the raw value (bool or number) plus the stated reason.

OneharnessProvider

The default Provider: runs each prompt on a harness through the oneharness CLI (targets v0.3.8+ — the release carrying opt-in run history: run --history/--history-dir/--history-name, a history_file in the report, and the oneharness history verb).

SkillRef

A borrowed view of the skill under test, as sent to the provider.

SplitProvider

A Provider that runs skills with one provider and judges with another: respond (and supports_resume) go to the skill-running provider; judge and simulate_user go to the judge. This keeps harness fidelity for the thing under test while letting the judge run on a fast, cheap, deterministic backend (typically ApiJudgeProvider).

Usage

Token / cost usage for one provider call.

UserTurn

A simulated-user turn produced by the provider.

Enums

JudgeKind

The kind of judgement requested.

Traits

Provider

The provider boundary.

Functions

default_history_dir

The centralized directory oneharness run history is written to, shared across every skilltest invocation so past runs accumulate in one reviewable place (rather than scattering per project the way oneharness’s own default would).

supports_resume

The harnesses oneharness’s adapter table marks supports_resume = true (claude-code’s --resume, opencode’s --session, cursor’s --resume). Kept in sync with the oneharness list registry — when a new harness ships session continuation, add it here so the runner threads session_id.