Struct EvalRunner 

pub struct EvalRunner { /* private fields */ }

Stateful evaluation engine that orchestrates scenario evaluation.

EvalRunner owns the scenario definitions and profiles (as Arcs), mirroring the ScouterQueue pattern. It provides:

• collect_scenario_data(): Populates scenario datasets and contexts
• evaluate(): Runs multi-level evaluation (sub-agent + scenario + aggregate), pulling captured spans from the global buffer automatically.

Implementations

impl EvalRunner

pub fn new( scenarios: EvalScenarios, profiles: HashMap<String, GenAIEvalProfile>, ) -> Self

pub fn scenarios(&self) -> EvalScenarios

pub fn evaluate( &mut self, config: Option<EvaluationConfig>, ) -> Result<ScenarioEvalResults, EvaluationError>

Run multi-level evaluation.

Spans are pulled automatically from the global capture buffer — no need to pass them explicitly.

LEVEL 1 — Sub-agent evaluation: flatten all records per alias → one EvalDataset → evaluate LEVEL 2 — Scenario evaluation: per scenario with tasks, evaluate against response + traces LEVEL 3 — Aggregate metrics

pub fn collect_scenario_data( &mut self, records: HashMap<String, Vec<EvalRecord>>, response: String, scenario: &EvalScenario, ) -> Result<(), EvaluationError>

Populate scenario data into the internal EvalScenarios container.

Arguments

• records - Map of alias → eval records for this scenario
• response - The agent’s final response for this scenario
• scenario - Reference to the scenario definition

Trait Implementations

impl Debug for EvalRunner

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'py> IntoPyObject<'py> for EvalRunner

type Target = EvalRunner

The Python output type

type Output = Bound<'py, <EvalRunner as IntoPyObject<'py>>::Target>

The smart pointer type to use.

type Error = PyErr

The type returned in the event of a conversion error.

fn into_pyobject( self, py: Python<'py>, ) -> Result<<Self as IntoPyObject<'_>>::Output, <Self as IntoPyObject<'_>>::Error>

Performs the conversion.

impl PyClass for EvalRunner

type Frozen = False

Whether the pyclass is frozen.

impl PyClassImpl for EvalRunner

const IS_BASETYPE: bool = false

#[pyclass(subclass)]

const IS_SUBCLASS: bool = false

#[pyclass(extends=…)]

const IS_MAPPING: bool = false

#[pyclass(mapping)]

const IS_SEQUENCE: bool = false

#[pyclass(sequence)]

const IS_IMMUTABLE_TYPE: bool = false

#[pyclass(immutable_type)]

const RAW_DOC: &'static CStr = /// Stateful evaluation engine that orchestrates scenario evaluation. /// /// `EvalRunner` owns the scenario definitions and profiles (as `Arc`s), /// mirroring the `ScouterQueue` pattern. It provides: /// - `collect_scenario_data()`: Populates scenario datasets and contexts /// - `evaluate()`: Runs multi-level evaluation (sub-agent + scenario + aggregate), /// pulling captured spans from the global buffer automatically.

Docstring for the class provided on the struct or enum.

const DOC: &'static CStr

Fully rendered class doc, including the text_signature if a constructor is defined.

type BaseType = PyAny

Base class

type ThreadChecker = SendablePyClass<EvalRunner>

This handles following two situations:

type PyClassMutability = <<PyAny as PyClassBaseType>::PyClassMutability as PyClassMutability>::MutableChild

Immutable or mutable

type Dict = PyClassDummySlot

Specify this class has #[pyclass(dict)] or not.

type WeakRef = PyClassDummySlot

Specify this class has #[pyclass(weakref)] or not.

type BaseNativeType = PyAny

The closest native ancestor. This is PyAny by default, and when you declare #[pyclass(extends=PyDict)], it’s PyDict.

fn items_iter() -> PyClassItemsIter

fn lazy_type_object() -> &'static LazyTypeObject<Self>

fn dict_offset() -> Option<isize>

fn weaklist_offset() -> Option<isize>

impl PyClassNewTextSignature for EvalRunner

const TEXT_SIGNATURE: &'static str = "(scenarios, profiles)"

impl PyMethods<EvalRunner> for PyClassImplCollector<EvalRunner>

fn py_methods(self) -> &'static PyClassItems

impl PyTypeInfo for EvalRunner

const NAME: &'static str = "EvalRunner"

Class name.

const MODULE: Option<&'static str> = ::core::option::Option::None

Module name, if any.

fn type_object_raw(py: Python<'_>) -> *mut PyTypeObject

Returns the PyTypeObject instance for this type.

fn type_object(py: Python<'_>) -> Bound<'_, PyType>

Returns the safe abstraction over the type object.

fn is_type_of(object: &Bound<'_, PyAny>) -> bool

Checks if object is an instance of this type or a subclass of this type.

fn is_exact_type_of(object: &Bound<'_, PyAny>) -> bool

Checks if object is an instance of this type.

impl DerefToPyAny for EvalRunner

impl ExtractPyClassWithClone for EvalRunner