Struct ApiJudgeProvider 

pub struct ApiJudgeProvider { /* private fields */ }

A judge-only Provider that scores evals and plays the simulated user with a direct model API call (Anthropic or OpenAI), rather than running them through a harness.

Why this exists: routing the judge through a full agentic harness pays an agent-loop cold start on every short verdict. A direct API call is one HTTP round trip — faster and cheaper on API-key auth — and still reuses the exact same judge/user prompts and tolerant verdict parsing as OneharnessProvider, so the two are directly comparable.

It does not run skills: respond returns an error. Compose it with a skill-running provider via SplitProvider so the harness under test still drives respond, while the judge runs on the API.

The request is sent with curl (Rust has no official vendor SDK). The API key is read from an env var and passed through a private (0600) curl config file, so it never appears in argv / ps.

Implementations

impl ApiJudgeProvider

pub fn new(config: &ApiJudgeConfig) -> Self

Build a provider from its configuration, resolving per-vendor defaults for the API-key env var and endpoint.

Trait Implementations

impl Provider for ApiJudgeProvider

fn respond( &self, _platform: &str, _model: &str, _skill: &SkillRef<'_>, _messages: &[Message], _session: Option<&str>, ) -> Result<AssistantTurn>

Run one assistant/skill turn given the conversation so far. session, when Some, is a handle returned by a previous respond call on this run that the provider may use to continue the same harness session (e.g. via oneharness run --resume); providers that don’t support continuation should ignore it.

fn simulate_user( &self, model: &str, persona: &str, messages: &[Message], ) -> Result<UserTurn>

Produce one simulated-user turn.

fn judge( &self, model: &str, query: &JudgeQuery<'_>, messages: &[Message], ) -> Result<JudgeVerdict>

Score a criterion against the conversation.

fn respond_streaming( &self, platform: &str, model: &str, skill: &SkillRef<'_>, messages: &[Message], session: Option<&str>, on_event: &mut dyn FnMut(&ToolEvent) -> ControlFlow<()>, ) -> Result<AssistantTurn>

Like Provider::respond, but delivers each normalized tool event to on_event as it is observed, so a caller can stream events live and short-circuit. on_event returns ControlFlow::Break to abort the turn — the provider tears down the harness and returns the partial turn.

fn respond_with_mocks( &self, platform: &str, model: &str, skill: &SkillRef<'_>, messages: &[Message], session: Option<&str>, mocks: Option<&MockPlan<'_>>, ) -> Result<AssistantTurn>

Like Provider::respond, but with a tool mock/spy plan: the provider must enforce the plan’s compiled ruleset on the turn’s tool calls and return the observed-call records on the turn (mock_calls).

fn respond_streaming_with_mocks( &self, platform: &str, model: &str, skill: &SkillRef<'_>, messages: &[Message], session: Option<&str>, mocks: Option<&MockPlan<'_>>, on_event: &mut dyn FnMut(&ToolEvent) -> ControlFlow<()>, ) -> Result<AssistantTurn>

Like Provider::respond_streaming, with a tool mock/spy plan. Same contract as Provider::respond_with_mocks: the default supports no mocking and errs loudly on a present plan.

fn supports_resume(&self, _platform: &str) -> bool

True iff respond on platform will faithfully continue a prior session when given its session_id. The default is false; providers that support resume override this so the runner knows to thread the session id through.