Skip to main content

aggregate_benchmark

Function aggregate_benchmark 

Source
pub fn aggregate_benchmark(
    iteration_dir: &Path,
    skill_name: &str,
) -> Result<BenchmarkData>
Expand description

Aggregate pass-rate / time / token stats for the configurations found in an eval workspace iteration directory.

Layout expected (per skill-creator SKILL.md):

<iteration-dir>/
  <eval-name>/
    with_skill/{grading.json, timing.json, outputs/...}
    without_skill/{grading.json, timing.json, outputs/...}   # or old_skill/
    eval_metadata.json

with_skill is the treatment; the first of without_skill / old_skill that exists is the baseline. Missing files are skipped (the eval is still counted, but contributes no data to that configuration).