Expand description
Evaluators: the domain rules attached to a case.
An evaluator is inlined on its case because it is only meaningful for that
case. Cross-snapshot comparability comes from (evaluatorId, digest), not
from a separate catalog resource nobody remembers to version.
Structs§
- Case
Evaluator - Judge
Allowlist Entry - Platform-maintained judge catalogue entry. Only models on this list may back a gating evaluator; a tenant may still judge with any visible model as long as the metric stays diagnostic.
- Judge
Inputs - What a judge is allowed to look at. Anything requiring a trace summary forces the case to demand a runtime-level trace.
- Judge
Model - LlmJudge
Spec - Model-graded evaluator. Everything that can move a verdict is pinned and digested, so judge drift can never be mistaken for subject regression.
- Metric
Spec - Rubric
- Rubric
Criterion - Rubric
Scale
Enums§
- Evaluator
Implementation - Judge
Aggregation - Judge
Kind - Metric
Class - How a metric participates in release decisions.
- Metric
Decision - Trace
Metric Spec - Deterministic metric computed from the case’s trace (plus evidence).
- Typed
Assertion - Deterministic assertion over bounded evidence.
Constants§
- JUDGE_
PROMOTION_ MIN_ AGREEMENT - JUDGE_
PROMOTION_ MIN_ LABELS - Minimum human-label evidence before a model-graded evaluator may gate. Applied to the aggregated ensemble output, not to an individual judge.