Enum Verification 

pub enum Verification {
    FileContains(String),
    FileEquals(String),
    Command {
        argv: Vec<String>,
        expect_exit: i32,
    },
    None,
    CompilesRust,
    RustTestPasses {
        test_src: String,
    },
    WorkspaceFileContains {
        file: PathBuf,
        needle: String,
    },
    DocumentContains {
        file: PathBuf,
        needle: String,
    },
    EachCompilesRust(Vec<PathBuf>),
    WorkspaceTestPasses {
        files: Vec<PathBuf>,
        test_src: String,
    },
}

How the harness decides a task is done.

What a passing gate proves

A pass means the stated criterion was satisfied under the harness’s compile and run. It is not proof that the artifact is correct, and no execution gate can be. A criterion checks what it checks: a gate asserting hello() == 42 is silent about everything else the file does.

The execution gates (Verification::CompilesRust, Verification::RustTestPasses, and the workspace variants) are stronger than the content gates (Verification::FileContains, Verification::FileEquals) because a stub cannot compile and run — that is why 0.2.0 introduced them — but “stronger” is not “sufficient”.

What 0.8.1 changed

Until 0.8.1 the file under verification was compiled into the same crate as the caller’s criterion, so it could change how that criterion resolved. A file defining #[macro_export] macro_rules! assert made assert!(false, "this gate can never pass") — which no correct implementation can satisfy — report a pass; a file opening with #![cfg(any())] deleted the criterion outright and the empty test binary exited 0. An agent found the first of these unprompted during io-cli 0.1.0’s live runs (see iterations/US-IO-HARNESS-0.8.0-I01).

The subject and the criterion are still compiled as one crate — that is deliberate and is not the fix. Making the subject a separate crate was tried during 0.8.1 development and abandoned: privacy is a wall between crates, so an ordinary non-pub implementation began failing a gate it had always passed, and a passing implementation is allowed to be private. What changed is where the criterion sits inside that crate. It is appended in a child module that opens with use super::* — so test_src still calls the subject’s items unqualified, private ones included — and that re-imports the prelude macros a criterion is likely to invoke explicitly. A subject defining macro_rules! assert now makes the name ambiguous (rustc E0659) and the gate fails to compile, rather than capturing it and passing an impossible criterion. A macro the subject exports under any other name still reaches the criterion through the glob.

The deletion attack is caught elsewhere, because one crate cannot catch it: the subject is separately compiled to an rlib with a probe item appended, and a second tiny crate is type-checked against that rlib. A subject that strips its own contents strips the probe too, and the reference fails to resolve. That separate subject compile is not what the criterion compiles against — its purposes are classifying an ordinary “this file does not compile” failure and hosting the probe.

This is a boundary against the file under verification, not against a hostile author with other tools. Verification runs the produced code, so it remains governed by the exec Policy and the 0.6.0 sandbox.

Choosing one

A criterion is a field of the TaskContract, so the run has a definition of done before the model is asked anything:

use io_harness::{TaskContract, Verification};
use std::time::Duration;

// Execution-based, and the pair a repository task normally wants: the listed
// files are concatenated and compiled together, then `test_src` is compiled
// beside them and run. "Together" is the point — each file compiling on its
// own (`EachCompilesRust`) would not catch a caller updated out of step with
// the function it calls.
let contract = TaskContract::workspace(
    "make `parse` reject an empty input instead of panicking",
    "/path/to/repo",
    Verification::WorkspaceTestPasses {
        files: vec!["src/parse.rs".into(), "src/lib.rs".into()],
        test_src: r#"
            #[test]
            fn empty_input_is_an_error() {
                assert!(parse("").is_err());
            }
        "#
        .into(),
    },
)
.with_time_budget(Duration::from_secs(600));

// A pass proves this criterion was satisfied under the harness's compile and
// run — nothing wider. `parse` is silent about everything else in the repo,
// and a criterion is the only thing the gate can check.
// https://github.com/initorigin/io-harness/blob/main/docs/guide/verification.md

The content variants are the weak tier and exist for outcomes that genuinely are about text. They cannot lie about what a file says and cannot confirm it works — a model satisfied FileContains("fn hello") in the 0.1.0 live run by writing that literal string, which does not compile:

use io_harness::Verification;

let cheap = Verification::FileContains("fn hello".into());
// Both of these pass. Only one of them is a program.
assert!(cheap.passes("src/hello.rs".as_ref(), "pub fn hello() -> u32 { 42 }").await?);
assert!(cheap.passes("src/hello.rs".as_ref(), "fn hello").await?);

// `CompilesRust` fails the second: a substring stub does not type-check, and
// since 0.8.1 a file that deletes its own items with `#![cfg(any())]` fails
// too rather than compiling clean on nothing.

Variants

FileContains(String)

The file’s contents must contain this text. Cheap, but gameable — a model can satisfy it without producing working code.

FileEquals(String)

The file’s contents must equal this text exactly.

Command

Run a caller-supplied command in the workspace’s execution sandbox and require this exit status (0.17.0).

One variant that covers every language the machine has a toolchain for, and the reason the crate stopped being a Rust-shaped harness: cargo test, npm test, go test ./..., pytest, dotnet test, make check are all the same criterion with a different argv.

argv is an array, program first, and there is no shell — ;, &&, $( ) and a backtick are bytes inside one argument, not syntax. argv[0] is what the Policy is asked about, exactly as rustc is on the Rust-specific gates, and verification cannot prompt, so the spawn happens only when a rule explicitly allows it.

use io_harness::{TaskContract, Verification};

// A JavaScript project. Nothing on this path is Rust-aware.
let contract = TaskContract::workspace(
    "make the failing test in test/parse.test.js pass",
    "/path/to/js-repo",
    Verification::Command {
        argv: vec!["npm".into(), "test".into()],
        expect_exit: 0,
    },
);

expect_exit is a status rather than a bool because “the linter found nothing” and “the command ran” are different claims, and some tools say so with a number. A command killed by a signal or by a sandbox cap never satisfies the criterion, whatever expect_exit says: it did not exit.

Workspace mode only — the command runs with the workspace root as its working directory, and a single-file contract has no root to give it.

Fields

argv: Vec<String>

The command to run, program first. Passed to the OS as an array; no shell parses it.

expect_exit: i32

The exit status that means the criterion is satisfied. Usually 0.

None

No gate at all (0.17.0). The run ends when the agent stops calling tools.

This is what makes an open-ended task expressible. Until 0.17.0 TaskContract took a Verification by value and four of the five variants ran rustc, so “debug this production issue” or “work out why the deploy fails” could not be stated, let alone run — there was no criterion to name and no way to say there was none.

An assistant turn carrying no tool call ends the run with RunOutcome::Finished, which is a distinct outcome from a step cap, a stall and a budget stop — so an unattended run that simply finished is never later mistaken for one that ran out. No done tool is added: an unverified run gains no tool surface over a verified one, and a model that has nothing left to do says so by saying something.

use io_harness::{RunOutcome, TaskContract, Verification};

let contract = TaskContract::workspace(
    "work out why the nightly deploy has been failing and write up what you find",
    "/path/to/repo",
    Verification::None,
);

// What "it worked" means for a run with no criterion: it finished on its
// own terms rather than hitting a ceiling. Nothing here claims the work is
// *correct* — with no gate, nothing could.
fn done(outcome: &RunOutcome) -> bool {
    matches!(outcome, RunOutcome::Finished { .. })
}

What you give up is what a gate was ever worth: nothing checked the work. Reach for Verification::Command whenever the task has a checkable criterion — this variant is for the tasks that genuinely do not.

CompilesRust

👎Deprecated since 0.17.0:

use Verification::Command { argv: vec![“cargo”.into(), “build”.into()], expect_exit: 0 } — or any compiler invocation for the language in hand. Removed in 0.18.0.

The file must compile as a Rust library (rustc --crate-type lib), and its items must survive to be type-checked. Execution-based: a non-compiling stub fails.

Since 0.8.1 the second half of that is enforced. A crate-level attribute such as #![cfg(any())] strips every item before rustc examines it, so a file whose body does not type-check compiled clean and passed. A subject that deletes its own contents now fails. Legitimate crate-level attributes — #![allow(dead_code)], #![no_std] — are unaffected.

RustTestPasses

👎Deprecated since 0.17.0:

use Verification::Command { argv: vec![“cargo”.into(), “test”.into()], expect_exit: 0 } and put the test in the project’s own test suite, where the project’s own tooling can run it. Removed in 0.18.0.

The file must compile, and test_src must compile against it and pass. Execution-based.

Since 0.8.1 the file under verification can no longer shadow the names test_src uses, nor delete it. test_src is unchanged: it still refers to the file’s items unqualified, and still reaches private items — the two remain one crate, deliberately, so an implementation does not have to be pub to pass. See the type-level docs for what a pass does and does not prove.

Fields

test_src: String

👎Deprecated since 0.17.0:

use Verification::Command { argv: vec![“cargo”.into(), “test”.into()], expect_exit: 0 } and put the test in the project’s own test suite, where the project’s own tooling can run it. Removed in 0.18.0.

Rust source compiled against the file, e.g. a #[test] fn.

WorkspaceFileContains

(workspace/multi-file) A named file under the workspace root must contain this text. Deterministic and language-agnostic — no compilation — so a task whose success is “a file now holds X” can be verified directly. Like Verification::FileContains it is gameable; use it when the outcome is genuinely about content, or as a composed-tree checkpoint a parent reads.

Fields

file: PathBuf

File to read, relative to the workspace root.

needle: String

Text that must be present in it.

DocumentContains

(workspace/multi-file, 0.14.0) A document under the workspace root must contain this text once its text has been extracted — not in its raw bytes.

The distinction is the whole variant. A .docx, .xlsx and .pptx are zips and a .pdf is a compressed object graph, so none of them is UTF-8. Verification::WorkspaceFileContains reads with read_to_string(..).unwrap_or_default(), which on a document yields the empty string — so it reports “does not contain” for every document, including one whose visible text plainly does contain the needle. It does not fail loudly; it silently always fails. A criterion that can never pass is worse than no criterion, because a run that ends StepCapReached looks like an agent that could not do the work rather than a gate that was never able to say yes.

The reader is chosen by the file’s extension: .xlsx, .docx, .pptx, .pdf. Anything else is an error rather than a fallback to reading the bytes as text — a criterion that silently degrades into a weaker check is the failure mode this exists to remove.

The variant exists in every build; only its implementation is behind the document features. A build without them returns a typed Error::Config saying so, rather than the variant vanishing and every match arm growing a cfg. Loud beats absent, which is the same call single-file mode makes when handed a policy it cannot enforce.

Like the other content criteria this is gameable and does not prove the document is correct — see the type-level docs.

Fields

file: PathBuf

Document to read, relative to the workspace root.

needle: String

Text that must be present in its extracted text.

EachCompilesRust(Vec<PathBuf>)

(workspace/multi-file) Every listed file — relative to the workspace root — must compile on its own as a Rust library. The run only succeeds when all of them do, so one wrong file fails the whole set.

Hardened with Verification::CompilesRust in 0.8.1: each file goes through the same check, so no listed file can pass by deleting itself.

WorkspaceTestPasses

👎Deprecated since 0.17.0:

use Verification::Command { argv: vec![“cargo”.into(), “test”.into()], expect_exit: 0 }, which runs the repository’s own suite over the whole crate rather than a concatenation of the files you listed. Removed in 0.18.0.

(workspace/multi-file) All listed files, concatenated in order, must compile, and test_src must compile against them and pass. This proves the edited files work together, not merely each in isolation.

Hardened with Verification::RustTestPasses in 0.8.1: a shadowing definition in any one of the files cannot defeat the gate, and a private item in any of them still reaches test_src.

Fields

files: Vec<PathBuf>

👎Deprecated since 0.17.0:

use Verification::Command { argv: vec![“cargo”.into(), “test”.into()], expect_exit: 0 }, which runs the repository’s own suite over the whole crate rather than a concatenation of the files you listed. Removed in 0.18.0.

Files (relative to the workspace root) concatenated into the subject.

test_src: String

👎Deprecated since 0.17.0:

use Verification::Command { argv: vec![“cargo”.into(), “test”.into()], expect_exit: 0 }, which runs the repository’s own suite over the whole crate rather than a concatenation of the files you listed. Removed in 0.18.0.

Rust source compiled against the files, e.g. a #[test] fn.

Implementations

impl Verification

pub async fn passes(&self, path: &Path, contents: &str) -> Result<bool>

Check a single produced file against the criterion (0.1/0.2 single-file mode). The multi-file variants belong to Verification::passes_in and error here.

contents is the current file text (already read by the caller).

pub async fn passes_guarded( &self, _path: &Path, contents: &str, guard: &ExecGuard<'_>, ) -> Result<bool>

Verification::passes, with every spawn checked against a policy.

pub async fn passes_in(&self, root: &Path) -> Result<bool>

Check the criterion against a workspace root (0.3 multi-file mode). The multi-file variants read their own files relative to root.

pub async fn passes_in_guarded( &self, root: &Path, guard: &ExecGuard<'_>, ) -> Result<bool>

Verification::passes_in, with every spawn checked against a policy.

pub fn describe(&self) -> String

Human-readable description fed to the model as the success criterion.

Trait Implementations

impl Clone for Verification

fn clone(&self) -> Verification

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Verification

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.