Crate lc_evaluation 

Evaluation module — LLM application evaluation

Provides the Evaluator / PairwiseEvaluator traits, built-in evaluators, dataset loading, and a batch runner, for quantifying the effect of prompt/model changes.

Core types:

• EvalError / Score / Example / Dataset / Evaluator / Predictor
• PairwiseEvaluator (pairwise comparison, a first-class citizen alongside pointwise, P1-1)
• EvalRunner and the Report (with original text + stddev + failure list)
• built-in evaluators: ExactMatch / StringDistance / EmbeddingSimilarity / LLMAsJudge
• other evaluators: Bleu / Faithfulness / PairwiseJudge / ContainsKeyword / RegexMatch

Example

use lc_evaluation::{EvalRunner, ExactMatch, StringDistance, Dataset, Example};
let dataset = Dataset::new(vec![Example::new("2+2?", "4")]);
let runner = EvalRunner::new(vec![Box::new(ExactMatch), Box::new(StringDistance)]);
// let report = runner.run(&dataset, &predictor).await?;

Re-exports

pub use price::OverallCost;

pub use price::Price;

pub use price::PriceBook;

pub use price::TokenUsage;

Modules

price

E1 — eval cost as a first-class citizen (v0.22.1 §S7).

Structs

Bleu

BLEU evaluator (BLEU-4 by default).

ContainsKeyword

Keyword-contains evaluator: checks whether the prediction contains the given keywords.

Dataset

Dataset

EmbeddingSimilarity

Embedding-similarity evaluator: scores by the cosine similarity of prediction and reference embeddings.

EvalRunner

Batch runner: holds both pointwise and pairwise evaluators.

ExactMatch

Exact-match evaluator: 1.0 when the prediction equals the reference after trimming, otherwise 0.0.

Example

Evaluation example

Faithfulness

Faithfulness evaluator (hallucination detection): how faithful an answer is to the reference context.

LLMAsJudge

LLM-judge evaluator: has the LLM score the prediction against a rubric (0 to max_score).

LengthCheck

Length-check evaluator: whether the prediction length (in chars) falls within [min, max].

PairwiseJudge

Pairwise-comparison evaluator (an LLM as the judge, picks one of two).

RegexMatch

Regex-match evaluator: checks whether the prediction matches the regex.

Report

Evaluation report (with original text, stddev, failure list; deserializable).

Score

Evaluation score (0.0–1.0, 1.0 is best)

StringDistance

String-distance evaluator: scores by Levenshtein edit distance, normalized.

Enums

EvalError

Evaluation error

Verdict

Pairwise comparison result

Traits

Evaluator

Evaluator trait

PairwiseEvaluator

Pairwise-comparison evaluator trait (arena mode): judges which of two answers (A/B) for the same input is better.

Predictor

Predictor trait (the object under evaluation: LLMChain / Agent, etc.)