1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
//! # `paladin-eval` — a deterministic evaluation harness for Paladin agent graphs
//!
//! `paladin-eval` provides a serde scenario file format (`.eval.yaml`, see
//! [`scenario::Scenario`]), a scripted [`paladin_ports::output::llm_port::LlmPort`]
//! implementation ([`scripted_llm::ScenarioLlm`]) that makes a run deterministic without
//! touching a real provider, an assertion library ([`assertion`]) evaluated over the
//! captured trace record and the final `Battlefield`, and [`runner::ScenarioRunner`] --
//! the `cargo test`-integrable driver plans 28-12/28-17's `[[test]] name = "evals"
//! harness = false` target and `paladin-cli eval run` both build on.
//!
//! ## Hexagonal position (D-27)
//!
//! `paladin-eval` is a **composition-tier tool crate**, not an extracted leaf crate: it
//! depends downward on `paladin-core`, `paladin-ports`, `paladin-battalion`,
//! `paladin-llm` (`mock` feature) and `paladin-storage` (`sqlite` feature), and never on
//! the facade `paladin-ai` — the same shape `doc-examples` uses, recorded in
//! `.planning/decisions/0048-paladin-eval-composition-crate.md` so ADR-0031's
//! leaf-crate-independence invariant (scoped to the *extracted* leaf crates) is not
//! misread as violated by these edges. Nothing in the workspace depends on
//! `paladin-eval` except as a `[dev-dependencies]` entry, WITH ONE NAMED EXCEPTION
//! (D-33, plan 28-12): the facade's own OPTIONAL `cli` feature adds `paladin-eval` as
//! an optional `[dependencies]` entry too, so `paladin-cli eval run` can build a
//! `ScenarioRunner` -- the facade's DEFAULT build (no `cli` feature) still gains no
//! edge to this crate. Outside that one opt-in exception, a downstream team building
//! their own agent graph on Paladin takes this crate as a dev-dependency to write
//! deterministic evaluation scenarios for it.
//!
//! ## Safety (T-28-05-01)
//!
//! A scenario file is untrusted structured input. It carries only deserialized
//! assertion and script parameters — no shell command, no path that gets executed, no
//! dynamically-loaded code. See [`scenario`]'s module docs for the closed-format
//! argument in full.
//!
//! ## Example
//!
//! ```rust
//! use paladin_eval::scenario::{Scenario, ScenarioTarget};
//!
//! let yaml = r#"
//! schema_version: "1"
//! target:
//! graph_doc: "fixtures/linear.json"
//! cases:
//! - name: happy_path
//! assertions:
//! - run_status: completed
//! "#;
//!
//! let scenario: Scenario = serde_yaml::from_str(yaml).unwrap();
//! assert_eq!(
//! scenario.target,
//! ScenarioTarget::GraphDoc {
//! graph_doc: "fixtures/linear.json".into(),
//! }
//! );
//! ```
pub use ;
pub use ;
pub use ;
pub use ;