eval-magic 0.5.0

One-stop CLI for running skill evals — measure whether an agent skill actually shifts behavior.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

//! The post-dispatch processing chain: stateless JSON-in/JSON-out stages.
//!
//! Chain order:
//! `record-runs` → `fill-transcripts` → `detect-stray-writes` → `grade` →
//! `aggregate`. Each stage reads JSON/JSONL artifacts from an iteration directory
//! and writes JSON back; no stage pipes to another in-memory, so any stage can
//! be run (and re-run) standalone.

pub mod aggregate;
pub mod detect_stray_writes;
pub mod error;
pub mod fill_transcripts;
pub mod grade;
pub mod io;
pub mod record_runs;
pub mod slots;

pub use aggregate::{Benchmark, aggregate};
pub use detect_stray_writes::{
    StrayFinding, StrayWritesReport, detect_live_source_reads, detect_stray_writes,
    detect_stray_writes_report,
};
pub use error::PipelineError;
pub use fill_transcripts::{FillTranscriptsResult, fill_transcripts};
pub use grade::{GradeContext, emit_judge_tasks, finalize};
pub use record_runs::{RecordRunsResult, record_runs};
pub use slots::{RunSlot, run_slots};