Struct Judge 

pub struct Judge { /* private fields */ }

Grades open-ended answers against a rubric, using a second model.

Kept separate from grade, which stays synchronous and pure. A judge is a model, so it is slow, costs money, and can be wrong — the deterministic checks should never have to wait behind it or inherit its uncertainty.

Implementations

impl Judge

pub fn new(provider: Box<dyn Provider>, model: Option<String>) -> Self

pub fn with_max_tokens(self, max_tokens: u32) -> Self

Override the verdict budget. Only worth touching for a judge that reasons at unusual length.

pub fn model(&self) -> &str

pub async fn preflight(&self) -> Result<()>

Prove the judge is reachable before a pass leans on it.

A validation pass whose judge is down must not look like one that found nothing. assess failing is handled per reflection — one bad response should not end a run — but that same path turns a judge that is entirely unreachable into a pass which skips every judged reflection, exits 0, and leaves a ledger that reads thin rather than broken. Retirement argues from that ledger, and under ungated learning it is the only brake there is; “nothing regressed” and “nothing was measured” would be indistinguishable.

Found on 2026-08-29: the nightly named a judge on a port nothing was serving, and had done for as long as the config had said so. It was invisible because validate had never had a rule to grade — the failure was waiting behind an earlier one.

One trivial call rather than a health endpoint, because what has to be true is that this provider answers this model, and only asking it establishes that.

pub async fn assess( &self, prompt: &str, rubric: &str, answer: &str, ) -> Result<Verdict>

Grade one answer. The judge gets no tools and no history.

pub async fn check(&self, case: &EvalCase, answer: &str) -> Option<Check>

Grade a case’s answer and return the check to append.

A judge that cannot be reached produces a failing check, never a skipped one. A case whose only real assertion silently evaporates is worse than a case that fails loudly.