Skip to main content

Provider

skilltest_core::provider

Trait Provider 

Source 
pub trait Provider {
    // Required methods
    fn respond(
        &self,
        platform: &str,
        model: &str,
        skill: &SkillRef<'_>,
        messages: &[Message],
        session: Option<&str>,
    ) -> Result<AssistantTurn>;
    fn simulate_user(
        &self,
        model: &str,
        persona: &str,
        messages: &[Message],
    ) -> Result<UserTurn>;
    fn judge(
        &self,
        model: &str,
        query: &JudgeQuery<'_>,
        messages: &[Message],
    ) -> Result<JudgeVerdict>;

    // Provided methods
    fn respond_streaming(
        &self,
        platform: &str,
        model: &str,
        skill: &SkillRef<'_>,
        messages: &[Message],
        session: Option<&str>,
        on_event: &mut dyn FnMut(&ToolEvent) -> ControlFlow<()>,
    ) -> Result<AssistantTurn> { ... }
    fn respond_with_mocks(
        &self,
        platform: &str,
        model: &str,
        skill: &SkillRef<'_>,
        messages: &[Message],
        session: Option<&str>,
        mocks: Option<&MockPlan<'_>>,
    ) -> Result<AssistantTurn> { ... }
    fn respond_streaming_with_mocks(
        &self,
        platform: &str,
        model: &str,
        skill: &SkillRef<'_>,
        messages: &[Message],
        session: Option<&str>,
        mocks: Option<&MockPlan<'_>>,
        on_event: &mut dyn FnMut(&ToolEvent) -> ControlFlow<()>,
    ) -> Result<AssistantTurn> { ... }
    fn supports_resume(&self, _platform: &str) -> bool { ... }
}
Expand description

The provider boundary.

Required Methods§

Source

fn respond( &self, platform: &str, model: &str, skill: &SkillRef<'_>, messages: &[Message], session: Option<&str>, ) -> Result<AssistantTurn>

Run one assistant/skill turn given the conversation so far. session, when Some, is a handle returned by a previous respond call on this run that the provider may use to continue the same harness session (e.g. via oneharness run --resume); providers that don’t support continuation should ignore it.

§Errors

Error::Provider if the command fails or returns malformed output.

Source

fn simulate_user( &self, model: &str, persona: &str, messages: &[Message], ) -> Result<UserTurn>

Produce one simulated-user turn.

§Errors

Error::Provider if the command fails or returns malformed output.

Source

fn judge( &self, model: &str, query: &JudgeQuery<'_>, messages: &[Message], ) -> Result<JudgeVerdict>

Score a criterion against the conversation.

§Errors

Error::Provider if the command fails or returns malformed output.

Provided Methods§

Source

fn respond_streaming( &self, platform: &str, model: &str, skill: &SkillRef<'_>, messages: &[Message], session: Option<&str>, on_event: &mut dyn FnMut(&ToolEvent) -> ControlFlow<()>, ) -> Result<AssistantTurn>

Like Provider::respond, but delivers each normalized tool event to on_event as it is observed, so a caller can stream events live and short-circuit. on_event returns ControlFlow::Break to abort the turn — the provider tears down the harness and returns the partial turn.

The default implementation runs the buffered Provider::respond and replays the finished turn’s events once; providers that can stream (like OneharnessProvider, via oneharness --stream) override it so events arrive — and an abort takes effect — mid-turn.

§Errors

Error::Provider if the command fails or returns malformed output.

Source

fn respond_with_mocks( &self, platform: &str, model: &str, skill: &SkillRef<'_>, messages: &[Message], session: Option<&str>, mocks: Option<&MockPlan<'_>>, ) -> Result<AssistantTurn>

Like Provider::respond, but with a tool mock/spy plan: the provider must enforce the plan’s compiled ruleset on the turn’s tool calls and return the observed-call records on the turn (mock_calls).

The default implementation supports no mocking: a present plan is a loud error — a provider silently ignoring mocks would let a mocked suite pass vacuously — and an absent one delegates to Provider::respond. CommandProvider and OneharnessProvider override this.

§Errors

Error::Provider if the command fails, returns malformed output, or a plan was given and this provider cannot enforce it.

Source

fn respond_streaming_with_mocks( &self, platform: &str, model: &str, skill: &SkillRef<'_>, messages: &[Message], session: Option<&str>, mocks: Option<&MockPlan<'_>>, on_event: &mut dyn FnMut(&ToolEvent) -> ControlFlow<()>, ) -> Result<AssistantTurn>

Like Provider::respond_streaming, with a tool mock/spy plan. Same contract as Provider::respond_with_mocks: the default supports no mocking and errs loudly on a present plan.

§Errors

As Provider::respond_with_mocks.

Source

fn supports_resume(&self, _platform: &str) -> bool

True iff respond on platform will faithfully continue a prior session when given its session_id. The default is false; providers that support resume override this so the runner knows to thread the session id through.

Dyn Compatibility§

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors§