Module eval 

Grounding retrieval benchmark — decided eval (PORT-CONTRACT.d/15).

Port of src/asdecided/services/eval.py. Deterministic by ADR-066: the scored path is a pure function of (corpus bytes, query set, retrieval code) — no network, no randomness, no clock. The only wall-clock/build values are metadata.generated_at and metadata.lore_version, both diagnostic and excluded from the gate (the parity harness masks them in --json).

The benchmark guards the REAL retrieval surface: a search_artifacts case consumes resolve::search_index order verbatim, and a get_related case consumes the incoming neighborhood ordering that the MCP get_related tool serializes (mirrored here from decided-mcp::graph:: incoming_references — decided-engine cannot depend on decided-mcp, and eval only needs the ordered id list).

Structs

EvalUsageError

EvalUsageError — the CLI maps this to exit 2 (decided eval: <msg>).

GateFailure

One fired gate rule.

QueryCase

One scored retrieval case (REQ-008).

Scorecard

A full benchmark run: gated metrics plus diagnostic context.

Constants

DEFAULT_BASELINE

DEFAULT_CONFIG

DEFAULT_CORPUS

DEFAULT_QUERIES

K_VALUES

The ranks the benchmark reports Precision@k / Recall@k at (REQ-003).

NEGATIVE_K

The hard-negative window: the widest k (REQ-003).

Functions

corpus_hash

corpus_hash(root) — sha256: over rel-path + NUL + bytes + NUL per walked Markdown file, in the corpus walk’s sorted order (REQ-005).

evaluate_gate

evaluate_gate(current, baseline, config) (REQ-006) — one failure per fired rule, deterministic order: negatives, then per gated pair (missing-metric floor / floor / regression).

load_baseline

load_baseline(path) — the committed baseline metrics object.

load_config

load_config(path) — floors and tolerance.

load_query_set

load_query_set(path) — parse and shape-check the committed query set.

query_set_hash

query_set_hash(path) — sha256: over the raw file bytes.

render_metrics_json

render_metrics_json(metrics) — what --update-baseline writes.

render_scorecard_human

render_scorecard_human(scorecard) — overall / by-category / by-tool / Violations, Python format-spec faithful.

render_scorecard_json

render_scorecard_json(scorecard) — pretty JSON, ensure_ascii=False.

run_eval

run_eval(root, queries_path) (REQ-001..REQ-005).