Crate mira 

Mira — a Rust-first, code-first evaluation framework for agents and tools.

Mira is a developer tool shaped like a test runner. You define evals in Rust (or any language that speaks the protocol), and a generic host CLI runs them across a target matrix, scores the results, and reports.

The model

Eval = Dataset(Sample…) + Subject + [Scorer…]  ×  target matrix

• Sample — one dataset row: input turns, an optional target, seeded files, tags, and free-form metadata.
• Subject — the thing under evaluation. One adapter per shape: an in-process closure (subject_fn), an external binary (CliSubject, the polyglot path), or a custom integration such as mira-everruns’s RuntimeSubject.
• Transcript — the normalized result every subject produces, so scorers and reporting are shared.
• Scorer — grades a Transcript into a Score. Deterministic built-ins, an arbitrary-closure escape hatch, and LLM-as-judge (model_graded) compose freely.
• Target — one case of the matrix. Provider-agnostic; missing API keys mark a case unavailable so it is skipped, not failed.

Two ways to run

• In process — build Evals and drive them with a Runner. Best for unit-style evals that live next to the code under test.
• Over the protocol — your program is a Study: it bundles evals and calls serve to expose them. The mira host CLI (Host) compiles/spawns it, plans the run, and owns selection, the matrix, run storage, and reporting. Provider keys never cross the wire — models are addressed by label. See protocol.

See the crate examples/ (greet, coding, cli_subject) for runnable studies.

Re-exports

pub use aggregate::TrialAggregate;

pub use aggregate::aggregate_trials;

pub use content::Message;

pub use content::Part;

pub use content::Role;

pub use content::Source;

pub use dataset::Dataset;

pub use dataset::Sample;

pub use eval::Eval;

pub use exec::Concurrency;

pub use exec::run_cases;

pub use glob::glob_match;

pub use host::Host;

pub use host::HostHandle;

pub use target::Target;

pub use registry::registered_evals;

pub use run::RunMeta;

pub use run::RunSummary;

pub use run::new_run_id;

pub use run::new_run_id_at;

pub use run::now_unix;

pub use runner::CaseOutcome;

pub use runner::RunReport;

pub use runner::Runner;

pub use scorer::Scorer;

pub use study::Study;

pub use subject::CliSubject;

pub use subject::Subject;

pub use subject::subject_fn;

pub use trajectory::ToolInvocation;

pub use trajectory::Trajectory;

Modules

aggregate

Trial aggregation — the contract for turning N repetitions of a case into pass@k, pass-rate, and score variance.

content

Multimodal content: a typed vocabulary for non-text inputs and outputs.

dataset

Samples and Datasets — the inputs to an eval.

eval

Eval: one evaluation = a dataset + a subject + scorers + a target matrix.

exec

Bounded, provider-aware, adaptive execution of a planned matrix.

glob

Tiny, dependency-free glob matcher for case selection.

host

Host side of the eval protocol. Spawns the study process and issues initialize / list / run requests, handling interleaved progress notifications. The mira CLI (mira-cli) is the user-facing driver built on top of this.

protocol

The Mira eval protocol: newline-delimited JSON over stdio, MCP-style.

registry

Automatic eval registration.

report

Reporting. Renders the host’s collected RunResults as a terminal grid, a canonical JSON record, a JUnit XML file (for CI test UIs), a Markdown summary (for PR job summaries), or a self-contained HTML transcript viewer. The JSON record is the machine-readable artifact the HTML viewer and trend aggregation consume.

run

Run identity, timing, and rolled-up summary for a single host invocation.

runner

Runner: the in-process run loop. Expands evals × targets × samples into cases, applies selective filtering, runs each case through its subject + scorers, and collects a RunReport.

scorer

Scorers grade a Transcript. They compose freely: an Eval holds a Vec<Box<dyn Scorer>> and every scorer runs against every transcript. This is one open, code-first vocabulary — deterministic built-ins, an arbitrary-closure escape hatch, and LLM-as-judge — rather than a closed enum.

study

The study side of the eval protocol. A Study is your eval program: it bundles the evals you’re investigating and, when you call serve, runs the stdio loop that answers the host’s initialize / list / run requests.

subject

Subject: the thing under evaluation. Implementors turn a Sample into a Transcript. Each run gets a fresh subject invocation (isolation), so state from one sample cannot leak into another.

target

Target — one case of the target matrix (the privileged comparison axis): the configured thing under evaluation. For an LLM eval a target is a model; for an agent eval it is a harness (e.g. yolop, codex), optionally wrapping a model.

trajectory

ATIF trajectories: the primary structured trajectory contract of the Mira eval protocol.

Macros

register_eval

Register a zero-argument fn() -> Eval so it is picked up by registered_evals / Study::registered.

Structs

RunCx

Per-run context handed to a Subject: which target to use for this matrix case, and the run limits.

Score

Outcome of a single Scorer on a Transcript.

Timing

Wall-clock timing for a run. Subjects that can measure it populate these; the rest leave them at their defaults.

Transcript

Normalized result of running a Subject on one Sample.

Trial

One trial’s reproducibility context: which repetition this case run is (index of count) and the seed handed to the subject, if any.

Usage

Token / cost accounting, summed across all turns of a run.

Enums

ErrorKind

Why a run failed, when it did — set alongside Transcript::error.

Functions

case_key

The canonical, stable identity of one matrix case: eval/sample@target, suffixed with [k=v,…] (axis params sorted by key) when extra axes vary. Used for selection, dedupe, checkpoint resume, and reporting — host and study compute it identically. target is the target label.

is_rate_limited

Heuristic: does this error message look like a provider rate-limit / quota / overload signal? The core is provider-agnostic, so detection is a substring match over the common phrasings (HTTP 429, “rate limit”, “overloaded”, “quota”, …). The host’s adaptive scheduler uses this to back off and retry a case instead of failing it (see exec).

metadata_display

Render an open-ended Metadata value for display (reports, CLI): a JSON string yields its raw contents (no surrounding quotes); anything else yields its compact JSON form (3, true, {"k":"v"}).

trial_suffix

The #index key suffix for a (trial, trials) pair: present only when the case is repeated (trials > 1), so a single-trial case keeps the plain eval/sample@target[…] key. Host and study compute it identically.

Type Aliases

Metadata

Free-form, open-ended metadata attached to evals, samples, targets, transcripts, and runs.

Params

Matrix-axis values for one case: axis name → chosen value.

Attribute Macros

eval

The #[eval] attribute: registers a fn() -> Eval factory for cargo test-style discovery (the ergonomic form of register_eval!).