Module eval 

Evaluation harness: sweep a matrix of {chunk_size, overlap, retrieval mode} over a labelled dataset and rank the configurations by retrieval quality.

Each (chunk_size, overlap) builds a fresh in-memory index (chunk → embed → store) from the dataset’s Markdown documents; then every retrieval mode is run against every query and scored with metrics. Runs fully offline with the hashing embedder and no LLM (the LLM-backed modes are included only when a ChatModel is supplied).

Modules

metrics

Retrieval quality metrics: recall@k, MRR, and nDCG@k.

Structs

EvalDataset

A full evaluation dataset.

EvalDoc

A dataset document, supplied as already-converted Markdown for determinism.

EvalReport

The ranked results of a sweep.

EvalRow

One cell of the sweep: a chunking config paired with a retrieval mode.

Harness

Runs sweeps against a dataset using a fixed embedder (+ optional LLM).

QueryCase

A labelled query. A retrieved chunk is relevant if it contains any relevant substring (case-insensitive).

Question

A question as loaded from an external questions file. Accepts this crate’s {query, relevant} shape, the QA-benchmark {text, kind} shape, and the output of the answers subcommand ({question, answer, …} — extra fields are ignored, so answers --json output round-trips as a questions file). Only entries with non-empty relevant labels can score retrieval.

Functions

documents_from_md_dir

Build eval documents from a directory tree of Markdown files — typically the RAG_DOCUMENTS_OUTPUT mirror produced by ingest. Non-.md files are skipped; name is the path relative to dir.

load_questions

Load a questions file (a JSON array of Questions in either shape).