Crate paigasus_helikon_evals 

Evaluation harness for Paigasus Helikon agents: datasets, evaluators, deterministic replay, and trace recording.

The core loop: load an EvalDataset, point an eval run at an agent, attach evaluators, and collect a report of trajectory and final-response scores.

Structs

CaseOutcome

What one case’s agent run produced.

CaseResult

One case’s result: its agent-run outcome (or error) plus every evaluator’s score.

EvalCase

One evaluation case: an input plus optional expectations.

EvalDataset

A named collection of EvalCases.

EvalReport

The full result of an EvalRunBuilder::run call.

EvalRun

Entry point for configuring and running an evaluation.

EvalRunBuilder

Builds an EvalRun. See the individual setters for what’s required vs. optional; EvalRunBuilder::run validates at call time.

EvalSummary

Aggregate statistics over an EvalReport’s results.

EvaluatorScore

One evaluator’s named score for one case.

EvaluatorSummary

One evaluator’s aggregate over every case it scored.

ExactMatch

Compares a run’s final output against EvalCase::expected.

JsonSchemaConformance

Validates a run’s final output against a JSON Schema, independent of anything on the case itself (never skips).

LlmJudge

Judges a run’s final output with a model call, scored against a rubric.

MockModel

A scripted Model that replays pre-recorded ModelEvents: one script per invoke call, in order. Running out of scripts yields a ModelError — deterministic by construction.

RunMeta

Run-level metadata recorded once per EvalRunBuilder::run call.

Score

One evaluator’s verdict on one case.

ScriptFile

A recorded script file: per-invoke scripts, optionally keyed by case id.

ToolUseTrajectory

Compares the observed tool-call sequence in a run’s events against EvalCase::expected_tools.

Enums

EvalError

Errors produced by dataset loading, evaluation, and eval runs.

ScoreOutcome

Pass/fail/skip classification of one score.

ScriptEvent

Serde mirror of core’s ModelEvent (same five variants).

ScriptFinishReason

Serde mirror of core’s FinishReason.

TraceError

Errors from trace sinks.

Traits

Evaluator

Scores one case’s outcome. Implementations must be side-effect free.

TraceSink

Receives each case’s result during an eval run.