Expand description
Self-consistency uncertainty gate: measure a model’s own confidence by resampling.
Research basis
- Wang et al. 2022 (“Self-Consistency Improves Chain of Thought Reasoning in Language Models”): sampling multiple reasoning paths and taking the most-consistent answer significantly improves accuracy over greedy decoding. Answers a model reliably reproduces are far likelier correct.
- Farquhar et al., Nature 2024 (“Detecting Hallucinations in Large Language Models Using Semantic Entropy”): disagreement across samples — semantic entropy — flags hallucination. A continuous agreement score lets the conformal threshold machinery calibrate the serve cutoff against a target failure rate.
Mechanism: call the same model k times on the original request (concurrently — serial
would multiply added latency by k), then score the candidate’s agreement with each sample.
Per-sample agreement: (a) exact final-answer match (1.0 if the last number in the text, or
the whole normalized text, matches) or (b) token-set Jaccard similarity. Gate score is the
mean over all usable samples, clamped [0, 1]. Verdict = Pass iff score ≥ threshold.
maker == checker is ALLOWED — unlike crate::judge::JudgeGate, self-consistency is
definitionally self-referential: the model samples itself. This is not a bug; it is the
mechanism. The gate enforces no maker ≠ checker constraint.
Honest scope
- ponytail: lexical agreement (final-number extraction + token-set Jaccard) is a cheap proxy for semantic clustering. Strongest on short, factual, or structured outputs (numbers, labels, code identifiers). Upgrade to entailment-based semantic clustering via a judge model for long-form or heavily paraphrastic outputs.
- Cost: k extra model calls per request — that is the honest price of measured confidence. Pair with a cheap model (e.g. haiku) on the first rung so the sampling cost stays low; expensive frontier models only bear escalation cost when cheaper rungs fail.
Structs§
- Consistency
Gate - A gate that scores the candidate by agreement with
kfresh samples of the same model.