paladin_eval/lib.rs
1//! # `paladin-eval` — a deterministic evaluation harness for Paladin agent graphs
2//!
3//! `paladin-eval` provides a serde scenario file format (`.eval.yaml`, see
4//! [`scenario::Scenario`]), a scripted [`paladin_ports::output::llm_port::LlmPort`]
5//! implementation ([`scripted_llm::ScenarioLlm`]) that makes a run deterministic without
6//! touching a real provider, an assertion library ([`assertion`]) evaluated over the
7//! captured trace record and the final `Battlefield`, and [`runner::ScenarioRunner`] --
8//! the `cargo test`-integrable driver plans 28-12/28-17's `[[test]] name = "evals"
9//! harness = false` target and `paladin-cli eval run` both build on.
10//!
11//! ## Hexagonal position (D-27)
12//!
13//! `paladin-eval` is a **composition-tier tool crate**, not an extracted leaf crate: it
14//! depends downward on `paladin-core`, `paladin-ports`, `paladin-battalion`,
15//! `paladin-llm` (`mock` feature) and `paladin-storage` (`sqlite` feature), and never on
16//! the facade `paladin-ai` — the same shape `doc-examples` uses, recorded in
17//! `.planning/decisions/0048-paladin-eval-composition-crate.md` so ADR-0031's
18//! leaf-crate-independence invariant (scoped to the *extracted* leaf crates) is not
19//! misread as violated by these edges. Nothing in the workspace depends on
20//! `paladin-eval` except as a `[dev-dependencies]` entry, WITH ONE NAMED EXCEPTION
21//! (D-33, plan 28-12): the facade's own OPTIONAL `cli` feature adds `paladin-eval` as
22//! an optional `[dependencies]` entry too, so `paladin-cli eval run` can build a
23//! `ScenarioRunner` -- the facade's DEFAULT build (no `cli` feature) still gains no
24//! edge to this crate. Outside that one opt-in exception, a downstream team building
25//! their own agent graph on Paladin takes this crate as a dev-dependency to write
26//! deterministic evaluation scenarios for it.
27//!
28//! ## Safety (T-28-05-01)
29//!
30//! A scenario file is untrusted structured input. It carries only deserialized
31//! assertion and script parameters — no shell command, no path that gets executed, no
32//! dynamically-loaded code. See [`scenario`]'s module docs for the closed-format
33//! argument in full.
34//!
35//! ## Example
36//!
37//! ```rust
38//! use paladin_eval::scenario::{Scenario, ScenarioTarget};
39//!
40//! let yaml = r#"
41//! schema_version: "1"
42//! target:
43//! graph_doc: "fixtures/linear.json"
44//! cases:
45//! - name: happy_path
46//! assertions:
47//! - run_status: completed
48//! "#;
49//!
50//! let scenario: Scenario = serde_yaml::from_str(yaml).unwrap();
51//! assert_eq!(
52//! scenario.target,
53//! ScenarioTarget::GraphDoc {
54//! graph_doc: "fixtures/linear.json".into(),
55//! }
56//! );
57//! ```
58
59#![warn(missing_docs)]
60
61pub mod assertion;
62pub mod runner;
63pub mod scenario;
64pub mod scripted_llm;
65
66pub use assertion::{
67 AssertionContext, AssertionFailure, AssertionOutcome, CustomAssertion, evaluate,
68};
69pub use runner::{
70 AssertionResult, CaseOutcome, CaseReport, GraphConstructor, LiveModeError,
71 PALADIN_EVAL_LIVE_ENV, RegistriesFactory, RunOptions, RunnerError, ScenarioRunner,
72 ScriptedPorts, Verdict, check_live_mode,
73};
74pub use scenario::{
75 Assertion, Case, EVAL_SCHEMA_VERSION, LiveOptions, LlmErrorKind, LlmScript, MatchRule,
76 ParleyScript, RunStatusValue, Scenario, ScenarioError, ScenarioTarget, ScriptEntry, StoreKind,
77 Times,
78};
79pub use scripted_llm::{CapturedRequest, ScenarioLlm, ScenarioLlmError};