characterminator 0.1.2

find and eliminate characters outside an allowed set, per file type and per file, so text costs fewer tokens
Documentation
# SPEC

## §G GOAL

CLDR letter presets: vendored locale data, its generator, lazy per-name lookup. ⊥ set algebra & other presets (`src/charset`).

## §N NAV

rel|path|lens
up|.|-
up|src|the tool: charset presets, rule resolution, scan, fix, lint levels, token facade, render, CLI
up|src/charset|builtin presets, set membership, unions, custom sets, CLDR letters, preset data
self|src/charset/locale|CLDR letter presets: vendored locale data, generator, lazy lookup

## §R RESEARCH

id|topic|finding|src
R6|CLDR letters|766 locales in `cldr-misc-full`. `exemplarCharacters` = letters in normal use (pl: 32, 9 non-ASCII); `auxiliary` = loan letters; `punctuation` incl. locale quotes (pl `„”`). Unicode License v3|github.com/unicode-org/cldr-json
R15|CLDR full cost|`locales.ctrm-sets` 12,876 → 176,760 B: 1,563 lines = 481 sets + 1,082 alias lines (766 locales + default-content codes). release binary 7,212,864 → 7,364,064 B (+151,200, +2.1%). `ctrm check` on this repo, 5 rounds × 30 runs: before 19.7–23.8 ms/run, after 20.4–23.2 ∴ delta within noise: lazy parse (V64) costs a run naming ⊥ locale nothing measurable|2026-10-02, macOS arm64, release build, loop timer (hyperfine ∉ dev shell)

## §V INVARIANTS

V30: language letter presets ∀ CLDR locale, named by locale code (`pl`, `pt-BR`): main `exemplarCharacters` − ASCII, + uppercase forms. data vendored (Unicode License v3, R6), generated into `U+XXXX` data file (`src/charset:V22`), ⊥ fetched at runtime. `auxiliary` loan letters ship as `<code>-aux` from same vendoring ∴ 1 generator run, 2 presets.
V59: locale data (V30) = `locales.ctrm-sets`, generated by `cldr-letters.sh` from pinned cldr-json TAG + UCD `UnicodeData.txt` (simple uppercase); header records tag & ∀ input sha256; rerun over same inputs ≡ same bytes. `-aux` = aux + uppercase − main. multi-char exemplar (`{ch}`, base + combining mark) → its code points individually. ∀ locale set ∩ `hazard` = ∅ (subtracted at generation, test asserts). main empty after subtraction (all-ASCII, `en`) ⇒ alias line `<code> ascii` ∴ resolves, grants ASCII only; `-aux` empty ⇒ ⊥ set.
V61: FULL CLDR (T60) = ∀ locale in pinned `cldr-misc-full`, incl CJK, Indic, RTL, by the SAME generator (V59), extended: exemplar ranges (`[a-z]`, `[U+3041-U+3096]`) & `\uXXXX` escapes expanded; regional & script variants (`pt-BR`, `sr-Latn`, `zh-Hant`) follow the CLDR parent chain: emitted as a set ONLY when ≠ parent, else 1 ALIAS line naming the parent (`pt-BR pt`) ∴ ∀ CLDR code resolves, all-ASCII ones too (V59; test: `en` `en-US` `id` `ceb`) & ⊥ duplicate data. hazard ∩ = ∅ (V59). LAZY: locale file parsed only for the names a run's rules use ∴ a run naming ⊥ locale pays ⊥ parse; release binary size & `ctrm check` time on this repo, before vs after, recorded as an R row. per-locale `typography-<code>` (CLDR `punctuation` less ASCII & hazards, same generator & tag, T61 of `src/fix:V26`) rides the same file, alias rule & laziness.
V64: LAZY (V61) = `locales.ctrm-sets` ∉ eager builtin text (`SETS`), own compiled-in const. catalog MISS → line found by 1st token, ONLY it parsed, + names it composes in. alias's parent (`pt-BR pt`, `en ascii`) read from compiled-in data & inlined, ⊥ from catalog ∴ user set `pt` ⊥ stands in for `pt-BR`'s parent; ⊥ adds the parent's name. declared set of same name wins (only a miss looked up). rule-named locales adopted at catalog build ⟺ builtin sets on (`--no-builtin-sets` ⇒ ⊥ locale). `ctrm sets` lists ∀ locale. CLDR default-content codes (`pt-BR`, ⊥ file) = alias lines. test: run naming ⊥ locale adds ⊥ name.
V88: locale line lookup (V64) via name → line index built ONCE (`LazyLock`), on 1st lookup ∴ run naming ⊥ locale builds ⊥. first line per name wins ≡ old linear `find`. `adopt`, `adopt_all`, alias inlining share it ∴ ⊥ scan of `LOCALES` per name. runner: `lookup_test.rs` diffs index vs scan ∀ name + misses.

## §T TASKS

id|status|task|cites
T32|x|ARCHIVED to SPEC-ARCHIVE.md|V30,`src/charset:V22`,V59
T60|x|ARCHIVED to SPEC-ARCHIVE.md|V61,V59,V30

## §B BUGS

id|date|cause|fix
B32|2026-10-02|V59 dropped a locale w/ ⊥ non-ASCII letter ∴ 205 locales + 63 default-content codes (`en`, `en-US`, `id`, `ceb`) → "no set named", against V61|V59
B33|2026-10-02|alias line resolved its parent through the run's catalog ∴ `--set 'pt U+0161' --rule '* ascii+pt-BR'` granted U+0161 & ⊥ U+00E7 under `pt-BR`|V64