Expand description
LoCoMo memory benchmark (#291).
Long-conversation memory, measured the way lean-ctx works: store every turn as a memory, then for each question recall the top-k memories and score the recalled context against the gold answers (token-F1 / exact-match / answer containment), plus the token cost of the recalled context versus dumping the whole transcript.
Pipeline: dataset (load) → runner (ingest + recall + score) →
report (aggregate to publishable numbers). Run via the locomo_bench
example: cargo run --example locomo_bench --features dev-tools.
Modules§
- dataset
- LoCoMo benchmark dataset schema + loader (#291).
- report
- Aggregate per-question results into publishable LoCoMo metrics (#291).
- runner
- Run a LoCoMo sample through lean-ctx memory: ingest every turn as a knowledge fact, then for each question recall the top-k memories and score the recalled context against the gold answers (#291).
Functions§
- run
- Run a suite end-to-end and aggregate a report.