Skip to main content

Crate paladin_eval

Crate paladin_eval 

Source
Expand description

§paladin-eval — a deterministic evaluation harness for Paladin agent graphs

paladin-eval provides a serde scenario file format (.eval.yaml, see scenario::Scenario), a scripted paladin_ports::output::llm_port::LlmPort implementation (scripted_llm::ScenarioLlm) that makes a run deterministic without touching a real provider, an assertion library (assertion) evaluated over the captured trace record and the final Battlefield, and runner::ScenarioRunner – the cargo test-integrable driver plans 28-12/28-17’s [[test]] name = "evals" harness = false target and paladin-cli eval run both build on.

§Hexagonal position (D-27)

paladin-eval is a composition-tier tool crate, not an extracted leaf crate: it depends downward on paladin-core, paladin-ports, paladin-battalion, paladin-llm (mock feature) and paladin-storage (sqlite feature), and never on the facade paladin-ai — the same shape doc-examples uses, recorded in .planning/decisions/0048-paladin-eval-composition-crate.md so ADR-0031’s leaf-crate-independence invariant (scoped to the extracted leaf crates) is not misread as violated by these edges. Nothing in the workspace depends on paladin-eval except as a [dev-dependencies] entry, WITH ONE NAMED EXCEPTION (D-33, plan 28-12): the facade’s own OPTIONAL cli feature adds paladin-eval as an optional [dependencies] entry too, so paladin-cli eval run can build a ScenarioRunner – the facade’s DEFAULT build (no cli feature) still gains no edge to this crate. Outside that one opt-in exception, a downstream team building their own agent graph on Paladin takes this crate as a dev-dependency to write deterministic evaluation scenarios for it.

§Safety (T-28-05-01)

A scenario file is untrusted structured input. It carries only deserialized assertion and script parameters — no shell command, no path that gets executed, no dynamically-loaded code. See scenario’s module docs for the closed-format argument in full.

§Example

use paladin_eval::scenario::{Scenario, ScenarioTarget};

let yaml = r#"
schema_version: "1"
target:
  graph_doc: "fixtures/linear.json"
cases:
  - name: happy_path
    assertions:
      - run_status: completed
"#;

let scenario: Scenario = serde_yaml::from_str(yaml).unwrap();
assert_eq!(
    scenario.target,
    ScenarioTarget::GraphDoc {
        graph_doc: "fixtures/linear.json".into(),
    }
);

Re-exports§

pub use assertion::AssertionContext;
pub use assertion::AssertionFailure;
pub use assertion::AssertionOutcome;
pub use assertion::CustomAssertion;
pub use assertion::evaluate;
pub use runner::AssertionResult;
pub use runner::CaseOutcome;
pub use runner::CaseReport;
pub use runner::GraphConstructor;
pub use runner::LiveModeError;
pub use runner::PALADIN_EVAL_LIVE_ENV;
pub use runner::RegistriesFactory;
pub use runner::RunOptions;
pub use runner::RunnerError;
pub use runner::ScenarioRunner;
pub use runner::ScriptedPorts;
pub use runner::Verdict;
pub use runner::check_live_mode;
pub use scenario::Assertion;
pub use scenario::Case;
pub use scenario::EVAL_SCHEMA_VERSION;
pub use scenario::LiveOptions;
pub use scenario::LlmErrorKind;
pub use scenario::LlmScript;
pub use scenario::MatchRule;
pub use scenario::ParleyScript;
pub use scenario::RunStatusValue;
pub use scenario::Scenario;
pub use scenario::ScenarioError;
pub use scenario::ScenarioTarget;
pub use scenario::ScriptEntry;
pub use scenario::StoreKind;
pub use scenario::Times;
pub use scripted_llm::CapturedRequest;
pub use scripted_llm::ScenarioLlm;
pub use scripted_llm::ScenarioLlmError;

Modules§

assertion
The assertion library (OBS-04, D-29, PRD 07 §3 acceptance 5, OBS-FR-12).
runner
ScenarioRunner – drives a crate::scenario::Scenario case through a real WarEngine, captures its trace record stream, and evaluates its assertions (D-29, D-31, D-32, D-34).
scenario
The .eval.yaml scenario file format (OBS-04, D-28).
scripted_llm
ScenarioLlm – the scripted LlmPort a running scenario substitutes for a real provider (D-30).

Macros§

eval_scenarios
Expands to a fn main() for a harness = false [[test]] target (D-32): globs $pattern at runtime and hands one libtest_mimic::Trial per (file, case) pair to the custom harness, so cargo test --test <name> <filter> filters exactly like any other Rust test.