mira-eval

The core of Mira — a Rust-first, code-first evaluation framework for agents and tools, built for multi-turn, tool-using, long-running agent trajectories. The library is imported as mira.

You define evals in Rust; the mira host CLI runs them across a model matrix, scores the results, and reports. This crate is provider-agnostic — it carries (provider, model) labels and no SDK types, so it has no heavy dependencies.

The model

Eval = Dataset(Sample…) + Subject + [Scorer…]  ×  model matrix × axes

• Sample — one dataset row: input turns, an optional target, seeded files, tags, and metadata. Write it inline, or load Dataset::{jsonl,json}.
• Subject — the thing under evaluation, one adapter per shape: subject_fn(closure) (in-process), CliSubject (an external binary — the polyglot path), or mira_everruns::RuntimeSubject (a live runtime session).
• Transcript — the normalized result every subject produces: final response, iteration/tool counts, token + cost usage, tool names, captured files, raw events, and any error. One shape, so scoring and reporting are shared.
• Scorer — score(&Sample, &Transcript) -> Score. Deterministic built-ins (contains, regex, tool_called, file_contains, …), operational budgets (tokens_within, cost_within, latency_within, ttft_within, tools_used_exactly, …), combinators (all_of/any_of/not), an arbitrary-closure escape hatch, and LLM-as-judge (model_graded) — one open vocabulary, freely composed.
• Target — one matrix case: a provider-agnostic (label, provider, model, available, metadata) tuple with no API keys. Add arbitrary axes (.axis("effort", ["low","high"])) and the runner takes the cross-product.

Example

use mira::scorer::{contains, succeeded, latency_within};
use mira::subject::subject_fn;
use mira::{Eval, Transcript, register_eval};

fn greet() -> Eval {
    Eval::new("greet")
        .sample("hi", "Say hi and tell me the answer to life.")
        .subject(subject_fn(|_s, _cx| async move {
            // A real subject calls a model; this one fakes a good answer.
            Transcript::response("Hi! The answer is 42.")
        }))
        .scorer(succeeded())
        .scorer(contains("42"))
        .scorer(latency_within(2_000))
        .build()
}
register_eval!(greet);

#[tokio::main]
async fn main() -> std::io::Result<()> {
    mira::Study::registered().serve().await
}

The #[eval] attribute (re-exported from mira-macros via the default macros feature) gives cargo test-style discovery as an alternative to register_eval!. Run the study with the mira-cli host:

mira --bin greet list   # what the study advertises
mira --bin greet run    # run the whole matrix, score, and report

Two processes, one protocol

Your eval program (the study) and the runner (the host) live on opposite sides of a process boundary, talking newline-delimited JSON over stdio (MCP-style). The study owns subjects and scoring — and provider API keys, which never cross the wire. The host owns selection, the matrix, saved runs, and rendering. The protocol is versioned and forward-compatible, which makes any program in any language a valid study.

Learn more

• Getting started
• How it works
• Authoring evals · Scorers · Subjects
• The eval protocol

Licensed under MIT — see LICENSE.