Skip to main content

run_variant_tasks

Function run_variant_tasks 

Source
pub fn run_variant_tasks<F>(
    variant: &str,
    tasks: &[VariantTask],
    cmd_for: F,
    min_files: f64,
) -> Result<AgentBenchSummary, String>
Expand description

The shared variant runner: for each task, copy the fixture repo, index it, ask cmd_for for the per-task shell command (this is where a variant generates its context artifact into the freshly indexed repo and returns the agent command that consumes it), then run the benchagent protocol. Aggregates the same metrics as bench agent plus first-plan accuracy and graph-query counts, and records variant.