# discovery ranking eval set
Fixtures for Part 2 of `docs/proposals/h2-builder-discovery-acceptance.md`
(acceptance spec v2). This directory is **fixtures + a pure shape validator**
only — the fixtures landed in a separate commit before the implementation,
per the spec's define-done-first mandate, enforced against self-grading.
**No scoring implementation lives here.**
The ranking harness that RUNS these fixtures ships with the Part 2
implementation: `car-server-core`'s `coder::rpc::tests::ranking_eval`
(`cargo test -p car-server-core ranking_eval -- --nocapture`). It ranks the
fleet through the real `rank_services` with shipped scoring defaults (config
dump in output), seeding/resetting a real `RoutingStore` per run keyed by
agentdns identifier (the `discovery.report` write path), with a deterministic
lexical embedder injected at the embedding seam (`rank_services` takes
pre-computed vectors; no live model).
## Files
- `fleet.json` — a synthetic fleet of 20 candidate services spanning all five
discovery provider kinds (`declarative`, `connector`, `registry`, `external`,
`a2a`), each with `capability_text` and seeded `successes`/`failures`.
- `discovery_ranking.jsonl` — 20 `need → expected_top` cases, split into
cold-start and post-feedback regimes.
- `../tests/discovery_ranking_fixtures.rs` — pure loader + validator (counts,
unique ids, every `expected_top`/`expected_below` references a fleet entry,
a non-declarative demotion case is present, the tie-break case names the
lexicographically-smaller identifier). No scoring.
## Case schema
```
{
"id": "...",
"mode": "cold_start" | "post_feedback", // ranked separately
"need": "...",
"expected_top": "agentdns://...", // must reference a fleet entry
"expected_below": "agentdns://...", // (ordering/demotion cases) top must outrank this
"demotion": true, // history-driven reordering case
"non_declarative": true, // the demoted candidate is NOT a declarative agent
"tie_break": true, // identical-capability twins; smaller identifier wins
"notes": "..."
}
```
- Cold-start (`cs-*`): routing store empty, all priors neutral — ranking is
capability similarity alone.
- Post-feedback (`pf-*`): the fleet's seeded `successes`/`failures` are applied
to the routing store, then ranked.
- Demotion proof (`pf-charge-demotion`): a NON-declarative MCP-connector service
with a failing history is demoted below its healthy sibling — the exact case
impossible today and the point of the H2 work.
- Tie-break (`cs-tie-break-echo`): two registry twins with identical capability
text and empty history tie exactly; the ranker breaks the tie on ascending
identifier.
## Pass targets (from the acceptance spec, verbatim)
> **Ranking eval** (fixture-first, separate PR): 20 need→expected-top cases over
> a synthetic fleet with seeded outcome histories, routing-store state
> seeded/reset per run, deterministic tie-breaks (the code ties on identifier —
> assert it). Targets: top-1 ≥ 85%, MRR ≥ 0.9 **with shipped defaults** (config
> dump in eval output). Cold-start and post-feedback rankings tested separately.
> **The demotion proof uses a NON-declarative candidate**: a seeded failure
> history on an MCP-connector-backed service demotes it below a healthy sibling
> in `discovery.resolve` output — the exact case that is impossible today and is
> the point of this work.
> **One scoring substrate.** Both `declagents.route` and `discovery.resolve`
> score success priors from the SAME posterior: the Beta(success+1, fail+1) UCB
> from `car-memgine::utility`, computed from the raw `successes`/`failures` that
> `RouteStats` already persists in `~/.car/routing.json`.
## Known miss under the test embedder
`cs-weather-current` ranks 2nd (not top-1) under the harness's
deterministic lexical embedder: "the weather right now" shares no lexical
mass with `get_current`'s "current conditions" beyond "weather" —
separating them needs the semantic embedder the shipped live path uses.
The case is deliberately left IN the scored set (cold-start 9/10 = 0.90
still clears the ≥ 0.85 target); tuning the test embedder or the fixture
to reach 10/10 would be fixture overfitting.