urdu-text
Unicode normalization for Urdu text. Zero dependencies, no_std, no unsafe.
The problem
These two words look identical:
کتاب کتاب
They are not the same bytes. One was typed on an Urdu keyboard, where the first letter is U+06A9 (keheh). The other was typed on an Arabic keyboard, where it is U+0643 (kaf). Every Urdu font draws them the same way.
So a == b is false. Your search box returns nothing. Your deduplication
misses half the corpus. Your database has two rows that should be one.
The same thing happens with he (U+0647 vs U+06C1), ye (U+064A vs U+06CC), teh marbuta (U+0629 vs U+06C3), and — worst of all — with anything pasted out of a PDF, which arrives as pre-shaped presentation forms that match nothing at all.
use ;
assert!; // Arabic kaf vs Urdu keheh
assert_eq!; // tatweel (justification kashida)
assert_eq!; // PDF ligature, expanded
Why not just use an Arabic normalizer?
Urdu is written in a Perso-Arabic script, so reaching for an Arabic normalization crate is the obvious move. It is also the wrong one. Every general-purpose Arabic normalizer applies at least one of these folds, and each one silently changes what the text says:
| Fold | Arabic normalizers | urdu-text |
What breaks |
|---|---|---|---|
آ U+0622 → ا U+0627 |
✅ applied | ❌ never | آم "mango" becomes ام |
ے U+06D2 → ی U+06CC |
✅ applied | ❌ never | کے "of" becomes کی "whose" |
ھ U+06BE → ہ U+06C1 |
✅ applied | ❌ never | تھا "was" becomes تہا, not a word |
| strip hamza U+0654/U+0655 | ✅ applied | ❌ never | کوئی "someone" becomes کوی, not a word |
ك U+0643 → ک U+06A9 |
❌ backwards | ✅ applied | Arabic normalizers fold toward U+0643 |
The first is a genuine difference of language, not a bug in those crates. In
Arabic, آ really is an alef with a maddah stacked on top, and folding it is
defensible. In Urdu it is its own letter with its own slot in the alphabet.
The fourth is the nastiest, because it happens by accident. U+0654 sits inside the Unicode combining-marks range, so any implementation that says "strip all combining marks" deletes it. In Urdu, hamza is spelling — not decoration.
And the last row means the fold runs the wrong direction even when it is the right fold: Arabic tooling normalizes Urdu keheh to Arabic kaf, which is exactly backwards for an Urdu corpus.
Install
or in Cargo.toml:
[]
= "0.1"
Use
use normalize;
let cleaned = normalize;
normalize applies the default preset: presentation forms expanded, letters
folded, aerab and tatweel and invisible marks stripped, whitespace collapsed,
digits left alone.
Configuring
use ;
// Building a search index: also fold ۲۰۲۴ to 2024.
let indexing = Options ;
// Preparing text for display: keep the diacritics a poet put there.
let display = Options ;
// Start from nothing and switch on exactly one step.
let just_ligatures = Options ;
let out = normalize_with;
| Option | Default | What it does |
|---|---|---|
expand_presentation_forms |
true |
U+FE80–U+FEFC → base letters; lam-alef ligatures → two chars |
fold_letters |
true |
Arabic/Persian letterforms → Urdu codepoints |
strip_aerab |
true |
Removes harakat; keeps hamza |
strip_tatweel |
true |
Removes U+0640 kashida |
strip_format_marks |
true |
ZWSP/ZWNJ/ZWJ, LRM/RLM, bidi controls, BOM |
ascii_digits |
false |
۲۰۲۴ → 2024 — opt-in, it changes how text reads |
collapse_whitespace |
true |
Runs of whitespace → one space, trimmed |
Options is a plain struct, deliberately not #[non_exhaustive], so
..Options::default() works from outside the crate.
Also available
eq_normalized(a, b) for comparison, plus the per-character primitives if you
are building your own pipeline: fold_letter, is_aerab_char,
presentation_base, expand_ligature, to_ascii_digit, and the standalone
strip_tatweel / strip_aerab.
Properties
- Idempotent.
normalize(normalize(s)) == normalize(s), for every preset. - Single pass, one allocation. No intermediate strings.
- Zero dependencies.
#,#![forbid(unsafe_code)],#![deny(missing_docs)].
Where the decisions live
All character tables are in src/tables.rs, one judgement call
per comment, each explaining why. src/lib.rs contains no tables — if you
disagree with a fold, it is one grep away and you never have to read the
normalizer.
Open questions
I am confident about the four never-fold rules above; they are not negotiable. The following are genuine judgement calls where reasonable Urdu speakers disagree, and I would rather flag them than pretend they are settled. If you work with Urdu text professionally, I would like to hear from you — open an issue.
Each is a candidate for becoming an Options field rather than a hard-coded
decision, which is the likely shape of 0.2.
1. ة U+0629 → ۃ U+06C3, or → ہ U+06C1?
Current behaviour: → ۃ U+06C3.
This crate treats teh marbuta as a letter whose Urdu codepoint is the "goal"
variant, and stops there. But Urdu pronounces the final sound of Arabic
loanwords as /h/, and a number of style guides therefore respell it as ہ — so
خلافۃ would become خلافہ. That is orthographic reform rather than a shape
fold, so it is out of scope by the rule this crate follows. Whether that rule is
the right one for real corpora is exactly the question.
2. ۀ U+06C0 → ۂ U+06C2
Current behaviour: folded.
Both encode heh-with-hamza for the izafat construction. U+06C2 is preferred here because its base is heh goal, consistent with folding plain heh to U+06C1. This looks safe, but izafat is orthographically fiddly and I would like it checked.
3. Removing ZWNJ U+200C
Current behaviour: removed, along with the other zero-width marks.
ZWNJ is invisible, and two strings differing only by an invisible character must compare equal — that is the whole premise of the crate. But in Urdu, ZWNJ is sometimes inserted deliberately to block a ligature that would otherwise form, which makes removing it a rendering change. It is never a spelling change, so for search and deduplication this is almost certainly right. For text you intend to typeset again, it may not be.
4. Quranic and recitation marks
Current behaviour: treated as aerab and stripped (U+06D6–U+06DC, U+06DF–U+06E8, U+06EA–U+06ED, U+0610–U+061A, U+08D3–U+08FF).
For search and deduplication over ordinary Urdu prose this is right. For a corpus of Quranic text it is destructive: these marks are recitation instructions, and stripping them loses real information. Note that U+06DD (end of ayah), U+06DE (start of rub el hizb) and U+06E9 (place of sajdah) are standalone symbols rather than marks and are already excluded — the ranges have holes at exactly those three points.
If you are processing religious text, use
Options { strip_aerab: false, ..Options::default() }.
5. MSRV
Not declared. rust-version is absent from Cargo.toml rather than guessed,
because a number nobody has tested is worse than no number. The crate uses
nothing exotic and should build on much older toolchains; it is verified on
stable. If you need a guaranteed MSRV, open an issue and it will be tested and
declared properly.
Licence
Dual licensed under either of
- Apache Licence, Version 2.0 (LICENSE-APACHE)
- MIT licence (LICENSE-MIT)
at your option.
Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in this crate by you, as defined in the Apache-2.0 licence, shall be dual licensed as above, without any additional terms or conditions.