Skip to main content

Module provider

Module provider 

Source
Expand description

The provider boundary. skilltest never talks to a model directly; a Provider runs the skill, plays the simulated user, and judges the transcript.

There are two real implementations. OneharnessProvider (the default) runs each prompt on a harness through the oneharness CLI and parses its JSON. CommandProvider speaks a small JSON-lines protocol (see docs/protocol.md) and backs both the deterministic skilltest-fake-provider used by the gate and any custom provider you write. The Provider trait also lets the runner be unit-tested against an in-memory fake.

Structs§

AssistantTurn
An assistant/skill turn produced by the provider.
CommandProvider
A Provider backed by an external command speaking the JSON protocol.
JudgeQuery
A judge query: the criterion, its kind, and (for numeric) the scale.
JudgeVerdict
A judge verdict: the raw value (bool or number) plus the stated reason.
OneharnessProvider
The default Provider: runs each prompt on a harness through the oneharness CLI.
SkillRef
A borrowed view of the skill under test, as sent to the provider.
Usage
Token / cost usage for one provider call.
UserTurn
A simulated-user turn produced by the provider.

Enums§

JudgeKind
The kind of judgement requested.

Traits§

Provider
The provider boundary.

Functions§

supports_resume
The harnesses oneharness’s adapter table marks supports_resume = true (claude-code’s --resume, opencode’s --session, cursor’s --resume). Kept in sync with the oneharness list registry — when a new harness ships session continuation, add it here so the runner threads session_id.