# gugen v0.2.0 large-scale blind benchmark report (Phase 11)
Generated by `cargo run --example large_scale_benchmark --features serde` (AGENTS.md §22/§23). Measured against `benchmarks/data/kononova_sample.jsonl`, a 1500-reaction holdout sample of the Kononova et al. 2019 corpus (CC BY 4.0) with every route already used by `tests/validation.rs`'s 5 fixtures or `src/literature_conditions.rs`'s Phase 10 curated records excluded -- see `benchmarks/data/ATTRIBUTION.md` for the exact filter counts and `benchmarks/fetch_kononova.py` for the exclusion mechanism. Re-run this example and replace this file's content after any change to `score.rs`, `planner.rs`, or the fetch script's filter criteria, rather than hand-editing numbers here.
- **Corpus loading:** 1500 rows loaded; 0 could not be represented by gugen's own types (`Element::new`/`Composition::new` failure) despite `fetch_kononova.py`'s own filtering -- expected near-zero, and is: a non-zero count here would mean that Python-side filter has a gap the Rust side has to defend against (AGENTS.md §25, never panic on ordinary input).
- **Valid reaction generation rate:** 1364/1500 holdout rows produced at least one plan.
- **Element-balance exactness:** 3730/3730 produced plans conserve every element exactly (re-verified against the plan's own reaction, not assumed from `balance()`'s design alone).
- **Known precursor-set exact recovery:** 1223/1500 holdout rows' cited route was recovered exactly by at least one produced plan (anywhere in the ranked list).
- **Partial precursor match (valid alternative beyond the cited route, found via the decoy pool):** 312/1500 holdout rows -- not errors, real chemically valid alternatives the decoy-augmented catalog happens to also support (same framing as `tests/validation.rs`'s own La2(CO3)3 precedent).
- **Route-family coverage:** 2/2 route families exercised across this corpus (Phase 12 added `Mechanochemical` alongside `ConventionalSolidState`; both are offered unconditionally for every accepted precursor set, AGENTS.md §13, so this is expected to be 100% for any corpus with at least one row that produces a plan, not a discriminator).
- **Process-step coverage:** 7/7 step kinds exercised across 3730 produced plans: {"Cool", "Form", "Grind", "Heat", "IntermediateCharacterization", "Mix", "Weigh"}.
- **Condition evidence coverage, split by Phase 10 target overlap:** 8/16 plans resolved a condition among the 6 holdout rows whose *target* matches one of Phase 10's 5 curated targets (via a *different* precursor route than the curated record -- `InMemoryLiteratureConditionProvider` matches on target composition alone, so these resolve `EvidenceScope::SimilarMaterial`, not `ExactTarget`; this is not evidence gugen predicts conditions for unseen targets, only that it correctly reuses a curated record across a different route to the *same* known target). 0/3714 resolved among the remaining 1494 rows -- expected near-zero, confirming Phase 10's coverage has not accidentally generalized beyond its 5 curated targets.
- **Unresolved condition rate:** 27043 total unresolved condition entries across 3730 plans (7.25 per plan on average).
- **False confident plan rate:** 5 distinct `confidence.overall` value(s) observed across 3730 plans (contrast `examples/benchmark_report.rs`'s small fixture set, which sees exactly 1 -- this corpus's mix of Phase-10-resolved and unresolved rows gives real variability, though still not yet a validated correctness signal).
- **Rejected-candidate reason stratification:** 40769 rejected candidates across the sample. `RejectionCode` counts: {"DuplicatePlan": 609, "NoStoichiometricBalance": 1312, "PrecursorCountExceeded": 1159, "SearchBudgetExhausted": 12, "UnsupportedByproductRequired": 37677}. `MissingTargetElement` dominates by a wide margin (0/40769, 0.0%) -- not a chemistry finding but a mechanical one: `search_precursor_sets` checks element coverage before any byproduct/balance check, and this corpus's decoy-augmented catalogs (mean ~10 candidates/row, see the search-budget line below) generate many small subsets that simply don't happen to cover one specific target's element set. `UnsupportedByproductRequired`'s share of *all* rejections is 37677/40769 (92.4%), but the more meaningful figure is its share among the 43328 combinations that *did* pass the element-coverage gate (byproduct + no-balance + duplicate + accepted-and-kept): 37677/43328 (87.0%). This isolates genuine byproduct-allow-list gaps from decoy-driven coverage noise -- chloride precursors (common in this corpus) still release byproduct species outside gugen's curated allow-list (CO2/H2O/O2/NO2/CO/acetone, `src/balance.rs::curated_byproducts`); nitrate, oxalate, and acetate precursors no longer do. NO2 (nitrate) was added on standard metal-nitrate thermal-decomposition grounds (`2 Ba(NO3)2 -> 2 BaO + 4 NO2 + O2`); CO (oxalate) on standard cross-metal oxalate thermal-decomposition grounds (`FeC2O4 -> FeO + CO2 + CO`); acetone (acetate) on the classic ketonic-decarboxylation pathway (`Ba(CH3COO)2 -> BaO + (CH3)2CO + CO2`, Friedel 1858) -- all three with explicit owner sign-off, each scoped to its own single class -- see `curated_byproducts`'s own doc comment for the citations and, for acetone specifically, the narrower (alkaline-earth-focused, not universal cross-metal) grounding it discloses. Chloride was investigated and explicitly declined, not merely deferred: the dedicated check this paragraph's earlier revisions called for found the chloride precursors in this corpus are overwhelmingly molten-salt-flux usage (an inert, unreacted medium), not the solid-state-metathesis reaction (`MCl2 + Na2CO3 -> MCO3 + 2 NaCl`) an alkali-chloride byproduct would model, and a direct algorithmic replication of `balance()`/`search_precursor_sets` found zero real recall gain from adding one against a pruning-relaxation footprint touching most of the corpus's decoy pools -- widening it anyway would be exactly the benchmark-driven overfitting AGENTS.md §27 forbids, now with no offsetting benefit to weigh against it.
- **Search-budget exhaustion rate:** 12/1500 rows hit `RejectionCode::SearchBudgetExhausted` against `SearchBudget::default()` (`max_precursor_sets: 10_000`), with up to 8 element-overlapping decoys added per row (mean catalog size 10.5 candidates per row, pool of the 60 globally most frequent precursor formulas in this sample). This decoy cap was chosen by measuring this exact rate at several candidate values and picking the largest that kept it negligible.
- **Deterministic reproducibility:** yes -- replanning the entire 1500-row sample a second time produced byte-for-byte identical reports.
- **Planning throughput:** well under 1 millisecond per `Planner::plan` call, averaged over 1500 calls against this sample's real (decoy-augmented) catalogs on the machine that generated this report. The raw microsecond figure is printed to stderr, not into this checked-in report, since it varies run to run and would make this file diff against itself on every regeneration.
## Skipped, not silently
- **§23 differential validation** against another synthesis-planning implementation: not attempted, same reasoning as `examples/benchmark_report.rs` -- §23 says 可能なら ("if possible"), no runnable reference implementation exists in this workspace.
- **§22 temperature-specific metrics** (predicted-range-contains-reference rate, unsupported-exact-value rate): still undefined, not zero. This corpus's own reported temperatures are deliberately *not* embedded in `kononova_sample.jsonl` (see `benchmarks/fetch_kononova.py`) precisely to avoid the temptation to score gugen's `None` predictions against them, which would not be a meaningful MAE.
- **Out-of-domain abstention rate / arithmetic overflow handling:** covered by `examples/benchmark_report.rs`'s dedicated adversarial cases, not duplicated here -- every row in this corpus is, by construction, a parseable in-domain composition (that's what `benchmarks/fetch_kononova.py`'s filter selects for), so this corpus cannot itself exercise either path.