Skip to main content

Crate urdu_text

Crate urdu_text 

Source
Expand description

Unicode normalization for Urdu text.

کتاب typed on an Urdu keyboard and كتاب typed on an Arabic keyboard look identical on screen. They are not the same bytes: the third letter is U+06A9 in one and U+0643 in the other. So a == b is false, your search box returns nothing, and your deduplication misses half the corpus.

This crate folds those shape-identical duplicates together, strips the marks that carry no meaning, and expands the pre-shaped glyphs that PDF extraction leaves behind.

use urdu_text::{eq_normalized, normalize};

// Arabic kaf vs Urdu keheh — same word, different bytes.
assert!(eq_normalized("كتاب", "کتاب"));

// Tatweel is decoration for line justification; it means nothing.
assert_eq!(normalize("کتـــاب"), "کتاب");

// A lam-alef ligature out of a PDF, expanded back into two letters.
assert_eq!(normalize("\u{FEFB}ہور"), "لاہور");

§What this crate refuses to do

Urdu is written in a Perso-Arabic script, so it is tempting to reach for an Arabic normalizer. Every general-purpose Arabic normalizer gets at least one of the following wrong, and each mistake silently changes what the text says:

FoldArabic normalizersThis crateWhy
آ → اappliedneverآم “mango” ≠ ام
ے → یappliedneverکے “of” ≠ کی “whose”
ھ → ہappliedneverتھا “was” → تہا, not a word
strip hamza U+0654appliedneverکوئی “someone” → کوی, not a word

In Arabic, آ really is alef with a maddah stacked on it, and folding it is defensible. In Urdu it is its own letter with its own place in the alphabet. Hamza is worse: U+0654 sits inside the Unicode combining-marks range, so “strip all combining marks” removes it — and in Urdu, hamza is spelling.

§Configuring

normalize applies a default preset. For anything else, build an Options and call normalize_with:

use urdu_text::{normalize_with, Options};

// Everything on, plus ASCII digits (off in the default preset).
let indexing = Options { ascii_digits: true, ..Options::default() };
assert_eq!(normalize_with("۲۰۲۴ء", &indexing), "2024ء");

// Or start from nothing and switch on exactly one thing. Here the tatweel
// goes but the Arabic kaf U+0643 is deliberately left as it was.
let gentle = Options { strip_tatweel: true, ..Options::none() };
assert_eq!(normalize_with("\u{0643}ت\u{0640}اب", &gentle), "\u{0643}تاب");

Options is deliberately not #[non_exhaustive], so ..Options::default() works from outside the crate. The trade-off is that adding a field is a breaking change — which is the honest signal, because adding a normalization step changes everyone’s output anyway.

§Properties

  • Idempotent. normalize(normalize(s)) == normalize(s) for every preset.
  • Single pass, one allocation. No intermediate Strings.
  • Zero dependencies, #![no_std] (needs alloc), #![forbid(unsafe_code)].

§Where the decisions live

Every character table is in src/tables.rs, one judgement call per comment. The README lists the calls a native speaker should second-guess.

Structs§

Options
Which normalization steps to apply.

Constants§

TATWEEL
The tatweel (kashida) character, U+0640.

Functions§

eq_normalized
Are these two strings the same once normalized?
expand_ligature
Expand a lam-alef presentation ligature into its two component letters.
fold_letter
Fold a single character to its canonical Urdu form.
is_aerab_char
Is this character an aerab (harakat / diacritic) that carries no meaning once removed?
normalize
Normalize Urdu text with the default preset.
normalize_with
Normalize Urdu text with an explicit Options.
presentation_base
Map a single-character Arabic presentation form back to its base letter.
strip_aerab
Remove aerab (diacritics) and nothing else.
strip_tatweel
Remove tatweel (U+0640) and nothing else.
to_ascii_digit
Map an Arabic-Indic or Urdu digit to the corresponding ASCII digit.