Struct Evaluation 

pub struct Evaluation { /* private fields */ }

RubyLLM::Evaluation: a reusable evaluation. Clone it to reuse a definition; use Evaluation::subclass for Ruby’s Class.new(parent).

Implementations

impl Evaluation

pub fn new() -> Evaluation

An anonymous evaluation (Class.new(RubyLLM::Evaluation)): it needs an explicit dataset.

pub fn named(name: impl Into<String>) -> Evaluation

class SupportEvaluation < RubyLLM::Evaluation: the name reports carry and dataset discovery uses (SupportEvaluation reads app/evals/support_evaluation.yml).

pub fn subclass(&self) -> Evaluation

Class.new(parent): an anonymous child inheriting the dataset, evaluator, criteria, adapters, and hooks. The child may redeclare an inherited criterion; the parent is unchanged.

pub fn name(&self) -> Option<&str>

The evaluation’s name, if it has one.

pub fn with_config(&mut self, config: Arc<Config>) -> &mut Self

Use this configuration for events, the default evaluator model, and its requests.

pub fn dataset(&mut self, source: impl Into<Dataset>) -> &mut Self

dataset(source): a path, cases, data, or Dataset::block.

pub fn dataset_source(&self) -> Option<&Dataset>

dataset with no argument: the configured source, if any.

pub fn evaluator(&mut self, evaluator: Evaluator) -> &mut Self

evaluator target, **options: the default evaluator for every criterion.

pub fn without_evaluator(&mut self) -> &mut Self

evaluator false: no model grading; only assertions run.

pub fn current_evaluator(&self) -> Option<&Evaluator>

evaluator: the class evaluator, or None when grading is disabled.

pub fn evaluation( &mut self, name: impl Into<String>, instructions: Option<&str>, ) -> Result<&mut Self>

evaluation name, instructions: a semantic criterion. Declared criteria replace implicit correctness; evaluation(:correctness) without instructions includes the built-in comparison. None instructions set a policy on a question a Judge already defines.

pub fn evaluation_with( &mut self, name: impl Into<String>, instructions: Option<&str>, minimum: Option<f64>, evaluator: Option<Evaluator>, ) -> Result<&mut Self>

evaluation name, instructions, minimum:, evaluator:. A minimum applies to a native probability or score; evaluator overrides the class evaluator for this criterion.

pub fn definitions(&self) -> Vec<(String, Option<String>, Option<f64>)>

definitions: (name, instructions, minimum) for every declared criterion.

pub fn adapt<T: Any + Send + Sync>( &mut self, f: impl Fn(&T) -> Value + Send + Sync + 'static, ) -> &mut Self

adapt Type { |value| ... }: converts a custom result into evaluation evidence. The original stays available as the trial’s result.

pub fn adapter_count(&self) -> usize

adapters: how many are declared.

pub fn perform<F, Fut>(&mut self, f: F) -> &mut Self

where F: Fn(Instance) -> Fut + Send + Sync + 'static, Fut: Future<Output = Result<Outcome, Failure>> + Send + 'static,

def perform(input): runs the application under evaluation.

pub fn assertions( &mut self, f: impl Fn(&mut Assertions<'_>) -> Result<(), Failure> + Send + Sync + 'static, ) -> &mut Self

def assertions: runs after perform, with the Minitest-style assertion family.

pub fn clear_assertions(&mut self) -> &mut Self

Removes the assertions hook (def assertions; end).

pub fn setup<F, Fut>(&mut self, f: F) -> &mut Self

where F: Fn(Instance) -> Fut + Send + Sync + 'static, Fut: Future<Output = Result<(), Failure>> + Send + 'static,

def setup: runs before perform on each fresh case instance.

pub fn teardown<F, Fut>(&mut self, f: F) -> &mut Self

where F: Fn(Instance) -> Fut + Send + Sync + 'static, Fut: Future<Output = Result<(), Failure>> + Send + 'static,

def teardown: runs after each case, even when setup, perform, or an assertion fails.

pub fn cases( &self, dataset: Option<&Dataset>, only: Option<&[String]>, ) -> Result<Vec<Case>>

cases(dataset:, only:): the dataset cases, without running anything. only selects case names and fails when any is missing (or none is given).

pub async fn run(&self) -> Result<Report>

run: every case once, with the configured dataset.

pub async fn run_with(&self, options: RunOptions) -> Result<Report>

run(dataset:, only:, repetitions:, id:).

pub async fn run_each<F, Fut>( &self, options: RunOptions, on_trial: F, ) -> Result<Report>

where F: FnMut(&Trial) -> Fut + Send, Fut: Future<Output = Result<()>> + Send,

run(...) { |trial| ... }: on_trial receives each completed trial before the next case starts, and the future it returns runs to completion first; its error ends the run and is returned. Configuration errors return before any case runs; task, assertion, and evaluator failures are recorded per trial.

Trait Implementations

impl Clone for Evaluation

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Default for Evaluation

fn default() -> Evaluation

Returns the “default value” for a type.