Struct Runner 

pub struct Runner { /* private fields */ }

Runs a corpus.

Implementations

impl Runner

pub fn new(config: EvalConfig) -> Self

A runner with this configuration and no judge, which is all a corpus of deterministic assertions needs.

pub fn with_judge(self, judge: Arc<Judge>) -> Self

Attaches a judge, used only for the criteria an item asks for.

pub const fn config(&self) -> &EvalConfig

The configuration in force.

pub async fn run(&self, suite: &Suite, harness: &dyn EvalHarness) -> EvalReport

Runs every selected item of suite.

pub async fn run_control( &self, suite: &Suite, harness: &dyn EvalHarness, ) -> ControlRun

Runs the same suite twice, against the same harness, and hands back both reports as a ControlRun.

This is how the noise floor stops being an assumption. Nothing changes between the two passes — same corpus, same harness, same code — so whatever difference ControlRun::noise_floor finds is the harness’s own variation, and a later comparison can say whether a difference exceeds it instead of leaving a reader to guess.

It costs exactly twice a run, which is the honest price of knowing whether the first one meant anything.

let control = Runner::new(EvalConfig::default().with_samples_per_item(10))
    .run_control(suite, harness)
    .await;
let floor = control.noise_floor();
println!("{}", floor.summary());

pub async fn run_item( &self, item: &EvalItem, harness: &dyn EvalHarness, ) -> ItemReport

Runs one item, samples_per_item times, at most sample_concurrency of them at once.

The default concurrency of one reproduces the strictly serial behaviour exactly. Above one the samples execute interleaved — which is the point, when each of them is a call to a real endpoint — but the report does not move: the samples come back in index order either way, and the same set of results comes back whatever the setting is.

pub async fn run_sample( &self, item: &EvalItem, harness: &dyn EvalHarness, sample: SampleIndex, ) -> SampleReport

Runs one sample of one item.

Trait Implementations

impl Debug for Runner

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.