Struct EvalConfig 

pub struct EvalConfig {
    pub execution: ExecutionConfig,
    pub judging: JudgingConfig,
    pub selection: SelectionConfig,
}

Everything one evaluation run needs to know.

use turnframe_eval::config::EvalConfig;

let config = EvalConfig::from_toml_str(
    r#"
    [execution]
    samples_per_item = 10

    [judging]
    votes_per_sample = 3
    "#,
)?;

assert_eq!(config.execution.samples_per_item, 10);
assert_eq!(config.judging.votes_per_sample, 3);
config.validate()?;

Fields

execution: ExecutionConfig

How the agent under test is exercised.

judging: JudgingConfig

How the judge is polled, when an item asks for one.

selection: SelectionConfig

Which subset of a suite runs.

Implementations

impl EvalConfig

pub fn single() -> Self

A run of one sample and one vote: the cheapest configuration that still exercises every stage. Use it for a smoke run; a release gate wants the numbers of spec §27.6 instead.

pub const fn with_samples_per_item(self, samples: u32) -> Self

Sets the number of executions of the agent under test per item.

pub const fn recording_answers(self) -> Self

Keeps the turn’s own words on each sample, for curating a corpus.

See ExecutionConfig::record_answers for why this is off by default.

pub const fn with_votes_per_sample(self, votes: u32) -> Self

Sets the number of judge opinions collected per sample.

pub const fn with_sample_concurrency(self, samples: u32) -> Self

Sets how many samples may be in flight at once.

Leave it at one for a reproducible in-memory run; raise it only for a corpus against a real endpoint. See ExecutionConfig::sample_concurrency.

pub const fn with_max_observed_events(self, events: usize) -> Self

Caps how many of a turn’s events one observation records, loudly.

See ExecutionConfig::max_observed_events: an observation that hits the cap fails every assertion about events rather than passing over a ledger it only half read.

pub fn including_tag(self, tag: impl Into<Tag>) -> Self

Restricts the run to items carrying tag.

pub fn excluding_tag(self, tag: impl Into<Tag>) -> Self

Excludes items carrying tag, whatever else selects them.

pub fn from_toml_str(source: &str) -> Result<Self, ConfigError>

Parses a configuration from TOML.

Errors

ConfigError::Parse when the document is malformed or carries a key this crate does not understand.

pub fn load(path: impl AsRef<Path>) -> Result<Self, ConfigError>

Reads a configuration from a TOML file.

Errors

• ConfigError::Read when the file cannot be read;
• ConfigError::Parse when its contents are not a configuration.

pub const fn validate(&self) -> Result<(), ConfigError>

Checks the configuration is one this crate will run.

Errors

• ConfigError::ZeroSamples — an item that never executes has no result, not an empty one;
• ConfigError::ZeroVotes — a judge with no votes has no verdict.

Trait Implementations

impl Clone for EvalConfig

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for EvalConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for EvalConfig

fn default() -> Self

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for EvalConfig

where EvalConfig: Default,

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Eq for EvalConfig

impl PartialEq for EvalConfig

fn eq(&self, other: &Self) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl Serialize for EvalConfig

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl StructuralPartialEq for EvalConfig