Module bench 

Public benchmarks as evaluation cases.

speccoder bench import turns instances in the SWE-bench format (instance_id, repo, base_commit, problem_statement, test_patch, FAIL_TO_PASS, …: SWE-bench, SWE-bench Verified/Multilingual/Pro, and benchmarks that reuse the format) into cases of the evaluation suite:

• project/: the repository at base_commit as a Spec Kit project — the issue becomes the user story of spec.md, with one task to resolve it;
• oracle/: the files the benchmark’s test_patch adds or changes, which the agent never sees and the oracle runs after the attempt;
• case.toml: the task, expect = "complete", the test command.

Then speccoder eval <out> [--executor claude] runs them like any case. Downloading the datasets and cloning the repositories is the person’s (--repos holds one clone per repository, named owner__name), and so is the environment the tests need: SWE-bench runs each instance in a container image; here the tests run on this machine, in the sandbox.

Structs

ImportOptions

How to import.

Instance

One benchmark instance.

Issue

The issue a benchmark case is built around.

Functions

import

speccoder bench import.