Module eval_harness 

Retrieval evaluation harness for lean-ctx hybrid search.

Runs a standardized query→expected_file benchmark to measure Recall@k, MRR (Mean Reciprocal Rank), and latency. Outputs NDJSON scorecards.

Usage: lean-ctx benchmark --eval [path]

Structs

AbReport

Full dense-vs-lean A/B scorecard for the default-flip decision (#686).

ArmScore

Aggregate score for one retrieval arm.

CategoryScore

EvalQuery

EvalResult

EvalScorecard

Enums

AbVerdict

Verdict of a dense-vs-lean retrieval A/B.

SearchArm

Which retrieval pipeline an eval arm exercises (#686 default-flip decision).

Functions

generate_self_eval

Generate self-eval queries from an indexed codebase. Picks random symbols/files and constructs retrieval queries.

load_suite

Loads a curated eval suite: one JSON EvalQuery per line; blank lines and # comments are ignored. Real labelled queries — no generation, no mocks.

run_ab

Run the dense-vs-lean retrieval A/B over queries and decide whether the default search path can be flipped to dense-off without losing quality.

run_eval

Run evaluation using the full hybrid search pipeline (BM25 + embeddings + SPLADE). Falls back to BM25-only if embeddings are not available.