Skip to main content

Module judge

Module judge 

Source
Expand description

Native LLM-judge gate (SPEC §8.3): a separate model grades the candidate output against a rubric. Three anti-gaming properties are enforced structurally here, not left to the prompt:

  1. maker ≠ checker — a model never grades its own output; if the judge model equals the candidate’s model, the gate abstains rather than returning a compromised verdict.
  2. candidate is data, not instructions — the candidate is embedded in a fenced data block under a pinned system prompt that tells the judge to ignore any instructions inside it; the judge is called with no tools.
  3. the operator’s threshold governs — the verdict is score ≥ threshold, not whatever the judge feels like passing.

Live judge quality (does a real model catch a bad output?) is verified only against a live provider; the wiring, injection-resistance, and verdict logic here are unit-tested offline with a mock judge.

Structs§

JudgeGate
A gate that asks a judge model to grade the candidate against a rubric.

Functions§

build_judge_request
Build the judge request: pinned system prompt + the candidate fenced as data (never as instructions), no tools.
parse_judgment
Parse the judge’s reply into a verdict. The operator’s threshold on the judge’s score is authoritative; an explicit pass is a fallback when no score is given; anything unparseable abstains (never a fabricated pass/fail).