# Tuning a convention — its best range, and fixing its leaks
Three docs, three questions:
| Does this change ship? | [measurement.md](measurement.md) — the A/B playbook |
| Which whole *method* is best (natural vs DONT vs Woolsey…)? | [ai-bidder/gto-1nt-defense.md](ai-bidder/gto-1nt-defense.md) — the matrix-game tournament |
| **I've picked a method — what's its best range, and where does it leak?** | **this doc** |
Everything below is `1NT`-defense-flavored because that is where it was paid
for (the whole `project_natural-1nt-defense` saga), but the loop is
convention-agnostic — the last section maps it onto Stayman, Texas, transfers,
and minor keycard.
Prerequisite: read [measurement.md](measurement.md) first. This doc assumes its
checklist (knob it, complete both sides, paired seeds, dual scoring, reading
opponent) and only adds the parts specific to *tuning a parameter* and
*finding a leak*.
## Two axes
A convention that already exists gets improved along two axes, and they use
different tools:
1. **Range** — a strength/length knob: the overcall floor `(lo, hi)`, the
penalty-double floor, the one-suiter length minimum, the invite/game
boundary. You *sweep* it. This is where the obstruction wall bites hardest —
§"Classify the range before you sweep it".
2. **Leaks** — a specific call that bleeds IMPs. You *bucket and forensic* it:
which call, is it the action or the continuation, authored or floored.
§"Per-call forensics".
Most "improve the natural defense" work is axis 2 (find the bad call, fix its
continuation); most "best range of a new defense" work is axis 1.
## The knobs already exist
Every tunable already has a thread-local `set_*` read at book construction and
a harness flag. Sweeping is `for v in range { set_x(v); build; score }` on
paired seeds. The 1NT-defense range knobs (`src/bidding/american/defense.rs`):
| `set_natural_overcall_points(lo, hi)` | natural suit-overcall strength | `(8, 14)` |
| `set_natural_double_floor(f)` / `set_natural_double_shape(s)` | penalty-X strength / shape gate | `15` / `Any` |
| `set_woolsey_points(lo, hi)` | Woolsey suit-overcall band (= Landy's) | `(8, 19)` |
| `set_woolsey_double_floor(f)` | Woolsey takeout-X floor | `12` |
| `set_direct_dont_one_suiter_min(n)` | DONT one-suiter length minimum | `5` |
| `set_direct_landy_double_floor(f)` | both-majors-X floor | `15` |
If a new defense needs a range, add its knob the same way (default off / default
system byte-identical, per measurement.md ship rules) before you can sweep it.
## Classify the range before you sweep it
**This is the load-bearing rule.** Double-dummy cannot tune a *competitive*
range. It can tune a *constructive* one. Decide which you have before you read
a single sweep number.
### Constructive range → DD is trustworthy
A boundary between two contracts the hands *actually reach* — invite vs game,
game vs slam-try, which strain, how high to raise. Both arms of the sweep bid to
real contracts DD prices correctly, so an **interior optimum is real** and the
sweep is worth running on self-play or an analytic probe.
The recurring outcome here is a **null**: raw HCP usually wins the boundary and
fancier evaluators wash.
- 1NT-invite boundary: raw HCP beat points/fifths/bumrap/cccc/controls
(`project_nt-invite-evaluator-sweep`, `examples/probe-nt-invite-eval`).
- Texas slam boundary: responder `fit_value` beat raw HCP by +0.004,
CI touching 0 — not significant (`project_texas-slam-eval`).
Don't over-read a null: a wash means *keep the simpler evaluator*, which is a
result, not a failure. Report the CI.
### Competitive range → the obstruction wall
A floor/length on an *overcall, preempt, or takeout double* — a call whose value
is obstruction, lead-direction, and table pressure. DD prices all of that at
zero while counting the overbid cost in full (`project_preemption-dd-negative`,
measurement.md §"The obstruction wall"). So the DD-optimal setting is always
**"compete less"** (or, against a fallible opponent it can exploit, "compete
more") — never a bridge-real sweet spot.
The tell is a **monotonic sweep with no interior peak**:
> "the overcall floor wants garbage — perfectly linear no-downside gradient =
> cleanest obstruction-wall proof yet, strong-shapely wants Pass, X dominated,
> balancing wants the passive floor. Only the *constructive* non-vul overcall is
> DD-trustworthy." — `project_natural-1nt-defense`
Raising the natural-overcall floor was *monotonically worse* (8:14 base, 10:14
and 11:14 strictly worse) because the light 8–9-point overcalls are the winners
— against BBA, a fallible auction engine our obstruction pushes into DD-bad
contracts. That is **exploitation of a specific opponent**, not a real range;
discount it. In perfect-bidder self-play the same gradient points the other way
(compete less). Neither endpoint is the answer.
**How to actually price a competitive range:**
- **sd-lead** (`single_dummy_leads`, `--sd-worlds`) — the one bias DD gets
measurably wrong at the 1NT level (the blind opening lead). It gives
disclosure a price, so an overcall's lead-direction value finally registers.
This is the right scorer for competitive ranges; it is what flipped Woolsey to
the equilibrium at both vuls (`project_gto-1nt-defense`, gto-1nt-defense.md
§"The sd-lead bracket").
- **vs BBA, exploitation discounted** — informative about *this* opponent, not
GTO. State it as "beats BBA by tuning to its frozen range-model," not "the
best range."
- **A bridge-sound partition beats chasing the DD peak** — see next.
## The orphaned-points trap
A competitive-range sweep will often peak just above a clean threshold. Do **not**
chase that peak. Raising the Woolsey overcall floor to 16 "peaked" on DD only
because it stranded 15-point both-majors hands into Pass — and passing them is
the worst thing they can do, which the obstruction wall then rewards. The peak is
the wall flattering an unsound hole in the point-count.
Prefer the **clean partition** instead: 8–14 overcall / 15+ penalty-X, no orphaned
range. `set_direct_landy_double_floor` shipped at **15** (not its DD peak of 16)
for exactly this — the partition "8–14 overcall = a direct bid, 15+ X = too
strong to overcall" leaves no point-count homeless, and it matched the peak
within noise anyway (`project_natural-1nt-defense`, the direct-Landy floor
sweep).
Corollary (an architecture invariant, [bidding-architecture.md](bidding-architecture.md)):
**every rule table needs a finite catch-all.** A floor set so high the table
rejects a whole point-band drops those hands to the deterministic floor, which
usually competes wrong. A sweep that looks like it wants a high floor is often
really telling you the catch-all is missing.
## Reading a sweep
| Monotonic, no interior peak | Suspect. Competitive range hitting the wall, *or* an orphaned point-band. The endpoint is not an optimum — re-price on sd-lead / vs a reading opponent, or pick the sound partition. |
| Interior peak, bridge-sound partition | Trust it (constructive range). This is a real optimum. |
| Flat plateau within CI | The knob is inert on this measure. Pick the simplest/soundest value and move on — don't over-fit noise (the DONT `6:2` doubled-escape gate: the whole cluster tied, only the extreme was clearly worst). |
| Self-play and vs-BBA disagree on direction | Real, and expected. Self-play can't punish or exploit; vs-BBA can do both. State which opponent each verdict is *for*. |
Sub-0.1 IMPs/board is noise unless the sample is hundreds of thousands of boards
(measurement.md decision table). A monotonic gradient of ~0.001/board across the
whole range is second-order — the first-order facts (which method, does it beat
always-pass) rarely move with the knob.
## Per-call forensics — improving a method, not just picking one
To make natural or DONT *better* (not to pick between them), find the one call
that leaks and fix it. The loop:
1. **Bucket by the first defensive call.** `bba-score` re-buckets the we-defend
boards into `DIRECT (we bid over 1NT)` and `CONT <call>` per first action
(`examples/bba-score/main.rs:210`). Split conventions that share a syntactic
call — the defensive `X` bucket was hiding two different conventions until it
was split into `X (pen)` (direct 15+ penalty) vs `X (PH Landy)` (passed-hand
both-majors) on the doubler's lane; the combined bucket lied about which one
was leaking.
2. **Is it the action or the continuation?** A negative bucket is not
automatically the obstruction wall. Dump the worst boards. The penalty-X
bucket read as a "−1.2/board single-dummy wall" for weeks; worst-board
forensics showed most of it was the *doubler mis-pulling its own penalty X*
— the keyless floor's `we_have_not_bid()` overcall rules fire over the
opponents' escape (a Double isn't a bid), so a flat 15 doubles, they run, and
we bid 3NT into a busted partner. Gating those rules off after a penalty latch
(`set_penalty_no_pull`) moved the both-vul bucket **−2.086 → −0.608 IMPs/X-board**
— a fixable continuation bug, not a wall.
3. **Authored or floored?** `Trie::classify_with_provenance` (`book.rs:406`)
tells you: `depth:0, fallback:Some` = the call came from the deterministic
`instinct()` floor, not your authored node. A bad floored continuation is
fixed by improving the floor or authoring the node; a bad *authored* one is
your rule (`feedback_instinct_floor_over_node_authoring`).
4. **Fix the continuation, re-measure paired.** The dominant leak of a
convention is almost always a missing/ wrong continuation, not the action —
this is measurement.md's "a half-built convention measures as a loss" seen
from the tuning side. DONT's whole deficit was one missing escape node
(`1NT X XX 2♣ X P P P → 2♣x` on a misfit); authoring the doubler's real-suit
rebid over a doubled relay was worth +0.083/board.
**Iron rule, from measurement.md:** *before declaring a measured loss dead,
trace the worst divergent boards.* The usual culprit is an unauthored
continuation or an over-broad trigger, and it is fixable — the wall is only what
survives after the bugs are out.
## When a DD gain still doesn't ship
Tuning can find a real DD gain you still reject — the measure isn't the only
constraint:
- **Disclosure.** The both-majors takeout-X ("Landy X") beat penalty-X on DD
vs BBA by ~0.036/board, and was **dropped** — a both-majors double of 1NT is
not honestly disclosable as a penalty double, and disclosability overrides the
gain (`project_natural-1nt-defense`, "LANDY-X DROPPED"). Ship the honest card.
- **Soundness over the peak** — the orphaned-points trap above: the bridge-sound
partition wins even at a small DD cost.
- **jdh8's standing directive** — an artificial 1NT defense stays opt-in even
when it matches or beats the default on the honest measure ("at most opt-in; I
still have ideas for the natural defense"). Range tuning does not change a
default-off convention's default.
## This generalizes to any convention
Same two axes, same tools:
| Stayman | invite/game boundary via `fit_value` (`project_stayman-fit-value-raise`) — constructive, DD-trustworthy | missing continuation = missed slam (`project_stayman-slam-deficit`) |
| Texas / SAT slam | slam-try boundary evaluator sweep (`project_texas-slam-eval`) — constructive, mostly null | — |
| Jacoby transfers | super-accept strength — *competitive*, so opt-in (`project_transfer-competition`) | GF structure completion (`project_transfer-gf-structure`) |
| Minor keycard | — | every rule table needs a finite catch-all (`project_minor-keycard`) |
The rule of thumb transfers intact: **a boundary between two contracts the hands
reach is constructive (sweep it, trust the peak); a floor on a call whose value
is obstruction is competitive (the wall — sd-lead or a discounted vs-BBA read,
and prefer the sound partition).** Classify first, sweep second.
## The parameter-sweep checklist
1. Add the `set_*` knob + harness flag; default off / default system
byte-identical (measurement.md).
2. **Classify the range**: constructive (contract boundary) or competitive
(obstruction). This picks the scorer.
3. Constructive → sweep on self-play or an analytic probe; the interior peak is
real; expect a null (raw HCP wins). Competitive → **sd-lead**, or vs-BBA with
exploitation discounted; a monotonic gradient is the wall, not an optimum.
4. Paired seeds across the sweep (`SEED_BASE` once, reused), dual scoring.
5. Read the sweep with the table above. Prefer a bridge-sound partition to a
DD peak that strands a point-band.
6. Disclosability and soundness can veto a real DD gain — ship the honest card.