Skip to main content

Module calibration

Module calibration 

Source
Expand description

The calibration surface — the single documented home for the constants that shape ranking and confidence. Moving any value here (or an inline boost listed in the REGISTRY below) moves the frozen eval baselines, so every change is a reviewed diff against evals/baseline.json plus, in the ideal, a re-freeze justification.

Why this module exists (D4): the 2026-07-04 gate incident showed that scoring constants scattered as inline literals across a 1200-line retrieval.rs are the “constants nobody remembers choosing” — a change to one, hidden inside a refactor, silently drifted the baselines. The band thresholds and the ambiguity ratio live here as named constants; the remaining inline boosts are catalogued in the REGISTRY so a reviewer can see the whole surface in one place, with the test that pins each.

§REGISTRY — the full calibration surface (value · effect · pinning test)

Housed here (named constants):

  • BAND_STRONG_MIN = 45 — score ≥ this is Strong. Test: bands_are_calibrated_to_score_distribution.
  • BAND_WEAK_MIN = 25 — score ≥ this (and < strong) is Weak; below is Fallback. Same test.
  • AMBIGUITY_RATIO = 0.7 — runner-up ≥ ratio × top demotes a Strong top to Ambiguous. Test: calibrate_demotes_strong_top_when_runner_up_is_within_ambiguity_ratio.

Still inline in retrieval.rs / retrieval/scoring.rs (candidates for future extraction; each documented at its use site):

  • BM25F params (k1=1.2, b=0.75, weights [5,8,2,6,4,3]) — now the injected crate::retrieval::RankerConfig Default. Test: ranker_config_is_injected_not_read_from_env.
  • exact whole-query boost (+80) / partial substring (+6). Test: se-corpus exact-lookup golden.
  • intent boost (+20 × intents; generic-code gate → +6/0). Tests: generic_code_intent_does_not_make_body_only_helper_strong + code-intent golden.
  • name boost (+20, kind-gated for code). Covered via ground tests + the monotonicity pin.
  • relation-context (+12 × hits) / directional relation (+48 + 8 × endpoint) / code-action (+28 × hits + kind 8–16 + fn 18). Tests: directional_relation_*, code_action_*.
  • matched_identity additive tiebreak (+1/term). Covered via ground tests.
  • route penalty for body-only scatter (× coverage, floor 0.25). Test: route_score_penalizes_partial_body_only_mentions.
  • anchor arms: id_coverage ≥ 0.4 + name-grade + known_coverage ≥ 0.6. Tests: mostly_unknown_query_does_not_anchor_partial_name_match, anchored_typo_match_*.
  • canonical dominance (parent → best_child + 1). Test: canonical_dominance_lifts_a_parent_*.
  • apply_floor (drop trailing Fallback when a confident hit exists). Test: apply_floor_drops_trailing_fallback_only_when_a_confident_hit_exists.
  • band monotonicity invariant (a garbage term never raises a band). Test: adding_a_garbage_term_never_raises_a_hits_band.

Constants§

AMBIGUITY_RATIO
How close a runner-up must be, as a fraction of the top score, to demote a Strong top to Ambiguous — so a coin-flip query stops reporting false confidence.
BAND_STRONG_MIN
Score at or above which a hit bands Strong. Calibrated to the measured distribution: clean retrieval accuracy is 100% down to 45 and garbage tops out ~32, so 45 is discriminating yet keeps Strong high-accuracy and never labels garbage Strong.
BAND_WEAK_MIN
Score at or above which a hit bands Weak (below Strong); under this it is Fallback.
DISTINCTIVE_NAME_IDF
IDF at or above which an EXACT name-field match counts as a distinctive identity anchor — a query term rare enough that naming a node with it is unambiguous. Fixes band conservatism on common-vocabulary domains: “why did my CARBONARA turn into scrambled eggs” now anchors on carbonara (idf ≈ 1.54 here) instead of being capped to Weak by the common co-terms, while “egg” (idf ≈ 1.3, shared by many recipes) stays below the bar and cannot anchor. Empirically 1.5 is the separation point: a distinctive dish/symbol name in a small cross-referenced corpus reads as df≈3-4 (recipe + its own section + a technique that lists it), so its idf sits ~1.5, while genuinely common terms fall under it. Gated additionally by known_coverage ≥ 0.6, so an out-of-vocabulary garbage query cannot anchor on its one real term. Verified no-regression across all eval suites; only lifts bands that were honestly under-confident.