ouro 0.1.0

A golden test runner for language hackers
Documentation

🥇 ouro 🥇

A golden test runner for language hackers. Embed test expectations directly in your source files as comments — no separate fixture files, no test harness boilerplate.

test tests/golden/simple.myc ... ok
test tests/golden/errors.myc ... ok
test tests/golden/multiline.myc ... ok

test result: ok. 3 passed; 0 failed

How it works

ouro runs a binary (your compiler, interpreter, or language tool) against each test file and compares the output to expectations written in the file's own comments.

// args: --optimize
// out: 42
// :out
let x = 42
console.log(x)

Directives live in comment lines starting with a configurable prefix (// by default). Everything else is source code passed to your binary.

Binary invocation

For each test file ouro calls:

<binary> [args...] <test-file-path>

The test file path is always the last argument. args come from args: directives in the file. If there are none, ouro calls <binary> <test-file-path> with no extra flags.

Directives

Directive Form Meaning
out: <text> inline Entire stdout must equal <text>
out: / :out block Multi-line stdout expectation
err: <text> inline Entire stderr must equal <text>
err: / :err block Multi-line stderr expectation
args: <flags> inline Append shell-split flags to args (can repeat)
args: / :args block Multi-line args, one arg per line
exit: <n> inline Expected exit code (default: 0)

Omitting out: or err: means that stream is not checked at all.

Comparison

Trailing newlines are trimmed from both sides before comparing. Everything else is an exact match — no whitespace normalization, no regex.

Inline shorthand

# args: --run
# out: hello world
# exit: 0

print("hello world")

Multi-line block

// out:
// ; optimized output
// mov rax, 42
// ret
// :out

Block content can contain anything — including }, //, or other tokens from your language — without ambiguity.

Setup

1. Add to Cargo.toml

[dev-dependencies]
ouro = "0.1"

2. Create ouro.toml in your project root

binary = "target/debug/myc"   # path to your binary
files  = "tests/**/*.myc"     # glob of test files
prefix = "// "                # comment prefix (default: "// ")

3. Write a test

// tests/golden.rs
#[test]
fn golden() {
    ouro::run_from_cwd().unwrap();
}

Or use the builder directly:

#[test]
fn golden() {
    ouro::Suite::new()
        .binary("target/debug/myc")
        .files("tests/**/*.myc")
        .prefix("// ")
        .run()    // Ok(()) = all passed; Err(()) = failures (already printed to stderr)
        .unwrap();
}

4. Run

cargo test golden

CLI

Install the ouro binary with:

cargo install ouro --features binary

ouro [OPTIONS]
ouro llm-context

  --binary <PATH>    Binary to test
  --files <GLOB>     Test file glob     [default: tests/**/*]
  --prefix <STR>     Comment prefix     [default: "// "]
  --update           Overwrite expected output with actual
  --jobs <N>         Parallel workers   [default: num CPUs]
  --config <PATH>    Path to ouro.toml  [default: search upward from CWD]

Exit 0 if all tests pass, 1 if any fail.

ouro llm-context prints a compact plain-text spec of directives, invocation contract, comparison rules, and the library API — suitable for pasting into an LLM context window.

Updating expectations

When your output intentionally changes, regenerate all expected values in one step:

ouro --update

This rewrites the directive lines in each test file with the actual output from your binary. Review the diff with git diff, then commit.

Changing the comment prefix

For languages with a different comment syntax, set prefix in ouro.toml or via --prefix:

# Python / Ruby / shell
prefix = "# "

-- Lua / Haskell
prefix = "-- "

; Assembly / .ini
prefix = "; "

Parallelism

Tests run in parallel by default using Rayon. Control the thread count:

# ouro.toml
jobs = 4

ouro --jobs 4

Disable parallelism entirely by depending on ouro without the parallel feature:

ouro = { version = "0.1", default-features = false }

Cargo features

Feature Default Description
parallel yes Parallel test execution via Rayon
binary no Build the ouro CLI binary (implies parallel)

Development

Prerequisites

  • Rust 1.70+ (stable)

Build

cargo build
cargo build --features binary   # includes the CLI

Test

cargo test

The test suite includes:

  • Unit tests for the parser state machine (src/parser.rs)
  • Unit tests for the output rewriter (src/runner.rs)
  • Integration tests that run the full suite against a small fake compiler (tests/integration.rs)

Project layout

src/
  lib.rs       public API: Suite builder, run(), run_from_cwd()
  config.rs    ouro.toml parsing, upward config search
  patterns.rs  PatternSet trait + DefaultPatterns
  parser.rs    line scanner / state machine → TestCase
  runner.rs    spawn binary, capture output, compare, --update rewriter
  diff.rs      colored unified diff output
  main.rs      CLI entry point (binary feature)
k
tests/
  integration.rs       end-to-end integration tests
  fixtures/myc         minimal fake compiler used by integration tests
  golden/              golden test files for the integration suite

Adding a new directive

  1. Add a method to PatternSet in src/patterns.rs
  2. Implement it in DefaultPatterns
  3. Handle the new state/transition in src/parser.rs
  4. Add a unit test in the parser::tests module

Releasing

Releases are managed by release-plz. No separate release branch is needed.

How it works

  1. When a PR is merged to main, release-plz opens a release PR that bumps the version in Cargo.toml and updates CHANGELOG.md.
  2. When that release PR is merged, release-plz:
    • Creates a GitHub release with the generated changelog
    • Publishes the crate to crates.io
  3. The release.yml workflow triggers on the published GitHub release and builds cross-platform binaries (linux-x86_64, macos-x86_64, macos-aarch64, windows-x86_64), attaching them to the release.

Required secrets

Add these in Settings → Secrets and variables → Actions:

Secret Where to get it
CARGO_REGISTRY_TOKEN crates.io → New token (scope: publish-new, publish-update)

GITHUB_TOKEN is provided automatically by GitHub Actions.

Prior art

jfecher/golden-tests

License

MIT