Expand description
Unicode normalization for Urdu text.
کتاب typed on an Urdu keyboard and كتاب typed on an Arabic keyboard look
identical on screen. They are not the same bytes: the third letter is
U+06A9 in one and U+0643 in the other. So a == b is false, your
search box returns nothing, and your deduplication misses half the corpus.
This crate folds those shape-identical duplicates together, strips the marks that carry no meaning, and expands the pre-shaped glyphs that PDF extraction leaves behind.
use urdu_text::{eq_normalized, normalize};
// Arabic kaf vs Urdu keheh — same word, different bytes.
assert!(eq_normalized("كتاب", "کتاب"));
// Tatweel is decoration for line justification; it means nothing.
assert_eq!(normalize("کتـــاب"), "کتاب");
// A lam-alef ligature out of a PDF, expanded back into two letters.
assert_eq!(normalize("\u{FEFB}ہور"), "لاہور");§What this crate refuses to do
Urdu is written in a Perso-Arabic script, so it is tempting to reach for an Arabic normalizer. Every general-purpose Arabic normalizer gets at least one of the following wrong, and each mistake silently changes what the text says:
| Fold | Arabic normalizers | This crate | Why |
|---|---|---|---|
آ → ا | applied | never | آم “mango” ≠ ام |
ے → ی | applied | never | کے “of” ≠ کی “whose” |
ھ → ہ | applied | never | تھا “was” → تہا, not a word |
strip hamza U+0654 | applied | never | کوئی “someone” → کوی, not a word |
In Arabic, آ really is alef with a maddah stacked on it, and folding it is
defensible. In Urdu it is its own letter with its own place in the alphabet.
Hamza is worse: U+0654 sits inside the Unicode combining-marks range, so
“strip all combining marks” removes it — and in Urdu, hamza is spelling.
§Configuring
normalize applies a default preset. For anything else, build an
Options and call normalize_with:
use urdu_text::{normalize_with, Options};
// Everything on, plus ASCII digits (off in the default preset).
let indexing = Options { ascii_digits: true, ..Options::default() };
assert_eq!(normalize_with("۲۰۲۴ء", &indexing), "2024ء");
// Or start from nothing and switch on exactly one thing. Here the tatweel
// goes but the Arabic kaf U+0643 is deliberately left as it was.
let gentle = Options { strip_tatweel: true, ..Options::none() };
assert_eq!(normalize_with("\u{0643}ت\u{0640}اب", &gentle), "\u{0643}تاب");Options is deliberately not #[non_exhaustive], so ..Options::default()
works from outside the crate. The trade-off is that adding a field is a
breaking change — which is the honest signal, because adding a normalization
step changes everyone’s output anyway.
§Properties
- Idempotent.
normalize(normalize(s)) == normalize(s)for every preset. - Single pass, one allocation. No intermediate
Strings. - Zero dependencies,
#,#![forbid(unsafe_code)].
§Where the decisions live
Every character table is in src/tables.rs, one judgement call per comment.
The README lists the calls a native speaker should second-guess.
Structs§
- Options
- Which normalization steps to apply.
Constants§
- TATWEEL
- The tatweel (kashida) character,
U+0640.
Functions§
- eq_
normalized - Are these two strings the same once normalized?
- expand_
ligature - Expand a lam-alef presentation ligature into its two component letters.
- fold_
letter - Fold a single character to its canonical Urdu form.
- is_
aerab_ char - Is this character an aerab (harakat / diacritic) that carries no meaning once removed?
- normalize
- Normalize Urdu text with the default preset.
- normalize_
with - Normalize Urdu text with an explicit
Options. - presentation_
base - Map a single-character Arabic presentation form back to its base letter.
- strip_
aerab - Remove aerab (diacritics) and nothing else.
- strip_
tatweel - Remove tatweel (
U+0640) and nothing else. - to_
ascii_ digit - Map an Arabic-Indic or Urdu digit to the corresponding ASCII digit.