Skip to main content

Module folding

text_document_search

Module folding 

Source
Expand description

The fold: turning a source char into the chars a match compares.

A writer looking for Aurelien expects to find Aurélien; one looking for احمد expects أَحْمَد; one looking for strasse expects Straße. None of those are string equality, and none of them are to_lowercase.

§The pipeline, per source char

  1. Turkic tailoring, if the text’s language is Turkish or Azerbaijani (below).
  2. Canonical caseless matching, exactly as Unicode 3.13 defines it: NFD → case-fold → NFD. Case folding is not lowercasing — ß folds to ss, fi to fi, ς and Σ both to σ.
  3. Drop nonspacing marks (General_Category = Mn). One rule covers Latin accents, Arabic harakat and Hebrew niqqud. Deliberately not the Diacritic property (too broad) and deliberately not Mc — a Devanagari vowel sign is not decoration.
  4. The letter table, for what no normalization form can reach.
  5. Arabic orthographic normalization, and the tatweel.

Steps 3–5 are gated by diacritic_sensitive; step 2’s fold by case_sensitive.

§Why the marks are dropped rather than the Diacritic property tested

ø ł đ ħ ı ŧ æ œ þ ð have an empty canonical decomposition — the stroke is part of the letter, not a combining mark, so no normalization form touches them. They need the table in step 4, and the table is keyed on the already-case-folded char so that Ø and ø reach it alike. Get that wrong and Ørsted is not found by orsted, which is the kind of miss a writer never reports as a bug — they just conclude the search is broken.

ß is deliberately absent from the table: full case folding already owns it. Having it in both would mean the table always won, and the Straße/strasse test would prove nothing about the case fold it was written to verify.

§Why Turkic runs first, before NFD

İ (U+0130) canonically decomposes to I + COMBINING DOT ABOVE. Fold that under the Turkic rules and the I becomes ı — the dotless letter, which in Turkish is a different letter from i and means a different word. Turkish İstanbul would fold to ıstanbul and never be found. The tailoring is defined on the composed letter, so it must be applied to the composed letter — before anything decomposes it.

Locale is the only axis case folding has (there is no Greek or Lithuanian tailoring of Case_Folding, unlike of lower/upper/title), and tr/az is the whole of it: two code points, which is the entire T status in CaseFolding.txt.

Structs§

FoldSpec
What a fold is allowed to fold away.

Enums§

FoldLocale
The locale tailoring that changes how text folds.