urdu-text 0.1.0

Unicode normalization for Urdu: folds Arabic and Persian letterforms to their Urdu codepoints, strips aerab and tatweel, and expands PDF presentation forms.
Documentation
  • Coverage
  • 100%
    21 out of 21 items documented13 out of 13 items with examples
  • Size
  • Source code size: 75.6 kB This is the summed size of all the files inside the crates.io package for this release.
  • Documentation size: 220.6 kB This is the summed size of all files generated by rustdoc for all configured targets
  • Ø build duration
  • this release: 4s Average build duration of successful builds.
  • all releases: 4s Average build duration of successful builds in releases after 2024-10-23.
  • Links
  • faizanali2k05/urdu-text
    16 0 0
  • crates.io
  • Dependencies
  • Versions
  • Owners
  • faizanali2k05

urdu-text

crates.io docs.rs CI

Unicode normalization for Urdu text. Zero dependencies, no_std, no unsafe.

The problem

These two words look identical:

کتاب      کتاب

They are not the same bytes. One was typed on an Urdu keyboard, where the first letter is U+06A9 (keheh). The other was typed on an Arabic keyboard, where it is U+0643 (kaf). Every Urdu font draws them the same way.

So a == b is false. Your search box returns nothing. Your deduplication misses half the corpus. Your database has two rows that should be one.

The same thing happens with he (U+0647 vs U+06C1), ye (U+064A vs U+06CC), teh marbuta (U+0629 vs U+06C3), and — worst of all — with anything pasted out of a PDF, which arrives as pre-shaped presentation forms that match nothing at all.

use urdu_text::{eq_normalized, normalize};

assert!(eq_normalized("كتاب", "کتاب"));        // Arabic kaf vs Urdu keheh
assert_eq!(normalize("کتـــاب"), "کتاب");       // tatweel (justification kashida)
assert_eq!(normalize("\u{FEFB}ہور"), "لاہور");  // PDF ligature, expanded

Why not just use an Arabic normalizer?

Urdu is written in a Perso-Arabic script, so reaching for an Arabic normalization crate is the obvious move. It is also the wrong one. Every general-purpose Arabic normalizer applies at least one of these folds, and each one silently changes what the text says:

Fold Arabic normalizers urdu-text What breaks
آ U+0622 → ا U+0627 ✅ applied ❌ never آم "mango" becomes ام
ے U+06D2 → ی U+06CC ✅ applied ❌ never کے "of" becomes کی "whose"
ھ U+06BE → ہ U+06C1 ✅ applied ❌ never تھا "was" becomes تہا, not a word
strip hamza U+0654/U+0655 ✅ applied ❌ never کوئی "someone" becomes کوی, not a word
ك U+0643 → ک U+06A9 ❌ backwards ✅ applied Arabic normalizers fold toward U+0643

The first is a genuine difference of language, not a bug in those crates. In Arabic, آ really is an alef with a maddah stacked on top, and folding it is defensible. In Urdu it is its own letter with its own slot in the alphabet.

The fourth is the nastiest, because it happens by accident. U+0654 sits inside the Unicode combining-marks range, so any implementation that says "strip all combining marks" deletes it. In Urdu, hamza is spelling — not decoration.

And the last row means the fold runs the wrong direction even when it is the right fold: Arabic tooling normalizes Urdu keheh to Arabic kaf, which is exactly backwards for an Urdu corpus.

Install

cargo add urdu-text

or in Cargo.toml:

[dependencies]
urdu-text = "0.1"

Use

use urdu_text::normalize;

let cleaned = normalize(messy_input);

normalize applies the default preset: presentation forms expanded, letters folded, aerab and tatweel and invisible marks stripped, whitespace collapsed, digits left alone.

Configuring

use urdu_text::{normalize_with, Options};

// Building a search index: also fold ۲۰۲۴ to 2024.
let indexing = Options { ascii_digits: true, ..Options::default() };

// Preparing text for display: keep the diacritics a poet put there.
let display = Options { strip_aerab: false, ..Options::default() };

// Start from nothing and switch on exactly one step.
let just_ligatures = Options {
    expand_presentation_forms: true,
    ..Options::none()
};

let out = normalize_with(input, &indexing);
Option Default What it does
expand_presentation_forms true U+FE80–U+FEFC → base letters; lam-alef ligatures → two chars
fold_letters true Arabic/Persian letterforms → Urdu codepoints
strip_aerab true Removes harakat; keeps hamza
strip_tatweel true Removes U+0640 kashida
strip_format_marks true ZWSP/ZWNJ/ZWJ, LRM/RLM, bidi controls, BOM
ascii_digits false ۲۰۲۴ → 2024 — opt-in, it changes how text reads
collapse_whitespace true Runs of whitespace → one space, trimmed

Options is a plain struct, deliberately not #[non_exhaustive], so ..Options::default() works from outside the crate.

Also available

eq_normalized(a, b) for comparison, plus the per-character primitives if you are building your own pipeline: fold_letter, is_aerab_char, presentation_base, expand_ligature, to_ascii_digit, and the standalone strip_tatweel / strip_aerab.

Properties

  • Idempotent. normalize(normalize(s)) == normalize(s), for every preset.
  • Single pass, one allocation. No intermediate strings.
  • Zero dependencies. #![no_std] (needs alloc), #![forbid(unsafe_code)], #![deny(missing_docs)].

Where the decisions live

All character tables are in src/tables.rs, one judgement call per comment, each explaining why. src/lib.rs contains no tables — if you disagree with a fold, it is one grep away and you never have to read the normalizer.

Open questions

I am confident about the four never-fold rules above; they are not negotiable. The following are genuine judgement calls where reasonable Urdu speakers disagree, and I would rather flag them than pretend they are settled. If you work with Urdu text professionally, I would like to hear from you — open an issue.

Each is a candidate for becoming an Options field rather than a hard-coded decision, which is the likely shape of 0.2.

1. ة U+0629 → ۃ U+06C3, or → ہ U+06C1?

Current behaviour: → ۃ U+06C3.

This crate treats teh marbuta as a letter whose Urdu codepoint is the "goal" variant, and stops there. But Urdu pronounces the final sound of Arabic loanwords as /h/, and a number of style guides therefore respell it as ہ — so خلافۃ would become خلافہ. That is orthographic reform rather than a shape fold, so it is out of scope by the rule this crate follows. Whether that rule is the right one for real corpora is exactly the question.

2. ۀ U+06C0 → ۂ U+06C2

Current behaviour: folded.

Both encode heh-with-hamza for the izafat construction. U+06C2 is preferred here because its base is heh goal, consistent with folding plain heh to U+06C1. This looks safe, but izafat is orthographically fiddly and I would like it checked.

3. Removing ZWNJ U+200C

Current behaviour: removed, along with the other zero-width marks.

ZWNJ is invisible, and two strings differing only by an invisible character must compare equal — that is the whole premise of the crate. But in Urdu, ZWNJ is sometimes inserted deliberately to block a ligature that would otherwise form, which makes removing it a rendering change. It is never a spelling change, so for search and deduplication this is almost certainly right. For text you intend to typeset again, it may not be.

4. Quranic and recitation marks

Current behaviour: treated as aerab and stripped (U+06D6–U+06DC, U+06DF–U+06E8, U+06EA–U+06ED, U+0610–U+061A, U+08D3–U+08FF).

For search and deduplication over ordinary Urdu prose this is right. For a corpus of Quranic text it is destructive: these marks are recitation instructions, and stripping them loses real information. Note that U+06DD (end of ayah), U+06DE (start of rub el hizb) and U+06E9 (place of sajdah) are standalone symbols rather than marks and are already excluded — the ranges have holes at exactly those three points.

If you are processing religious text, use Options { strip_aerab: false, ..Options::default() }.

5. MSRV

Not declared. rust-version is absent from Cargo.toml rather than guessed, because a number nobody has tested is worse than no number. The crate uses nothing exotic and should build on much older toolchains; it is verified on stable. If you need a guaranteed MSRV, open an issue and it will be tested and declared properly.

Licence

Dual licensed under either of

at your option.

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in this crate by you, as defined in the Apache-2.0 licence, shall be dual licensed as above, without any additional terms or conditions.