Struct LlmJudgeMetric 

pub struct LlmJudgeMetric<M, T>
where
    M: CompletionModel,
    T: Judgment + Send + Sync + JsonSchema + Serialize + for<'a> Deserialize<'a>,
{ /* private fields */ }

Available on crate feature experimental only.

An LLM as a judge that judges an output by a given schema (and outputs the schema). The schema type uses the Judgment trait, which simply enforces a single function that checks whether it passes or not.

Trait Implementations

impl<M, T> Eval<T> for LlmJudgeMetric<M, T>

where M: CompletionModel + 'static, T: Judgment + Send + Sync + JsonSchema + Serialize + for<'a> Deserialize<'a> + Clone + 'static,

async fn eval(&self, input: String) -> EvalOutcome<T>

fn eval_batch( &self, input: Vec<String>, concurrency_limit: usize, ) -> impl Future<Output = Vec<EvalOutcome<Output>>> + Send

Send a bunch of inputs to be evaluated all in one call. You can set the concurrency limit to help alleviate issues with model provider API limits, as sending requests too quickly may result in throttling or temporary request refusal.