Crate lc_evaluation 

Evaluation module — LLM application evaluation

Provides the Evaluator / PairwiseEvaluator traits, built-in evaluators, dataset loading, and a batch runner, for quantifying the effect of prompt/model changes.

Core types:

• EvalError / Score / Example / Dataset / Evaluator / Predictor
• PairwiseEvaluator (pairwise comparison, a first-class citizen alongside pointwise, P1-1)
• EvalRunner and the Report (with original text + stddev + failure list)
• built-in evaluators: ExactMatch / StringDistance / EmbeddingSimilarity / LLMAsJudge
• other evaluators: Bleu / Faithfulness / PairwiseJudge / ContainsKeyword / RegexMatch

Example

use lc_evaluation::{EvalRunner, ExactMatch, StringDistance, Dataset, Example};
let dataset = Dataset::new(vec![Example::new("2+2?", "4")]);
let runner = EvalRunner::new(vec![Box::new(ExactMatch), Box::new(StringDistance)]);
// let report = runner.run(&dataset, &predictor).await?;

Re-exports

pub use price::OverallCost;

pub use price::Price;

pub use price::PriceBook;

pub use price::TokenUsage;

Modules

price

E1 — eval cost as a first-class citizen (v0.22.1 §S7).

Structs

AnswerRelevancy

RAGAS answer relevancy: does the answer actually address the question?

Bleu

BLEU evaluator (BLEU-4 by default).

ContainsKeyword

Keyword-contains evaluator: checks whether the prediction contains the given keywords.

ContextPrecision

RAGAS context precision: are the retrieved chunks relevant, and are relevant chunks ranked high?

ContextRecall

RAGAS context recall: share of the reference answer’s claims that the retrieved contexts support.

Dataset

Dataset

EmbeddingSimilarity

Embedding-similarity evaluator: scores by the cosine similarity of prediction and reference embeddings.

EvalRunner

Batch runner: holds pointwise, pairwise, and RAG evaluators.

ExactMatch

Exact-match evaluator: 1.0 when the prediction equals the reference after trimming, otherwise 0.0.

Example

Evaluation example

ExampleReport

Complete evaluation record for one example (includes the original text, for tracing low scores).

FailureRecord

Failure record: a predict or an evaluator score failed for the example at a given index.

Faithfulness

Faithfulness evaluator (hallucination detection): how faithful an answer is to the reference context.

LLMAsJudge

LLM-judge evaluator: has the LLM score the prediction against a rubric (0 to max_score).

LengthCheck

Length-check evaluator: whether the prediction length (in chars) falls within [min, max].

PairwiseJudge

Pairwise-comparison evaluator (an LLM as the judge, picks one of two).

RegexMatch

Regex-match evaluator: checks whether the prediction matches the regex.

Report

Evaluation report (with original text, stddev, failure list; deserializable).

Score

Evaluation score (0.0–1.0, 1.0 is best)

ScoreSummary

Summary statistics for one evaluator (mean + population stddev + sample count).

StringDistance

String-distance evaluator: scores by Levenshtein edit distance, normalized.

Enums

EvalError

Evaluation error

Verdict

Pairwise comparison result

Traits

Evaluator

Evaluator trait

PairwiseEvaluator

Pairwise-comparison evaluator trait (arena mode): judges which of two answers (A/B) for the same input is better.

Predictor

Predictor trait (the object under evaluation: LLMChain / Agent, etc.)

RagEvaluator

RAG evaluator trait (RAGAS-style): scores a prediction together with the retrieved contexts, which a plain Evaluator has no slot for.