klieo-eval

Recall-quality metrics for klieo memory pipelines.

Status: stable at 2.x

What

Pure scoring — no I/O, no async, no memory-trait dependency. Hand the crate a slice of RecallSample { query, expected, returned_ids } and get back a RecallReport with:

• hit_rate — fraction of queries with any expected id in top-K
• mean_reciprocal_rank — mean of 1 / rank-of-first-hit
• mean_precision_at_k — |expected ∩ top-K| / K, averaged
• mean_recall_at_k — |expected ∩ top-K| / |expected|, averaged
• mean_ndcg_at_k — normalised discounted cumulative gain, binary relevance

The harness lets the caller run recall against any memory pipeline — Qdrant, sqlite-vec, hybrid GraphRAG — then scores the produced ranking.

Why

The same fixture-driven quality gate previously hard-coded inside the klieo-compliance regulation-recall integration test is reusable across every klieo memory pipeline. Centralised, deduped, JSON-serialisable for CI artefacts.

Example

use klieo_eval::{score_recall, RecallSample};

let samples = vec![
    RecallSample::new(
        "ICT risk management",
        vec!["dora-art-5".into(), "dora-art-6".into()],
        vec!["dora-art-5".into(), "ai-act-art-6".into(), "dora-art-6".into()],
    ),
];
let report = score_recall(&samples, 5);
assert!(report.hit_rate() >= 0.9);
assert!(report.mean_ndcg_at_k() > 0.95);

License

MIT.