1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
//! Character tables and pure per-character lookups.
//!
//! Every linguistic judgement call this crate makes lives in this file, and
//! every one of them carries a comment saying *why*. If you disagree with a
//! decision, it is one `grep` away — you do not have to read the normalizer to
//! find it.
//!
//! The guiding rule: **fold only what is shape-identical and
//! semantically identical in Urdu.** When a fold would change meaning, it is
//! listed under "never folded" below and left alone, even though several
//! Arabic-oriented normalizers do apply it.
/// The tatweel (kashida) character, `U+0640`.
///
/// Tatweel is a typographic elongation mark. It carries no meaning at all —
/// it exists purely to stretch a word so a line of text can be justified.
/// `کتـــاب` and `کتاب` are the same word.
pub const TATWEEL: char = '\u{0640}';
// ---------------------------------------------------------------------------
// Letter folding
// ---------------------------------------------------------------------------
/// Fold a single character to its canonical Urdu form.
///
/// Returns `ch` unchanged if there is nothing to fold. This is the operation
/// that makes text typed on an Arabic keyboard compare equal to the same text
/// typed on an Urdu keyboard.
///
/// # What folds
///
/// | From | To | Reason |
/// |------|----|--------|
/// | `ك` U+0643, `ڪ` U+06AA | `ک` U+06A9 | Arabic kaf and swash kaf render identically to Urdu keheh |
/// | `ي` U+064A, `ى` U+0649, `ې` U+06D0, U+063D, U+063F | `ی` U+06CC | All render as Urdu choti ye |
/// | `ه` U+0647, `ە` U+06D5 | `ہ` U+06C1 | Arabic heh renders as Urdu gol he |
/// | `ۀ` U+06C0 | `ۂ` U+06C2 | Same izafat form, Urdu-preferred codepoint |
/// | `ة` U+0629 | `ۃ` U+06C3 | Teh marbuta, Urdu-preferred codepoint |
/// | `٠`–`٩` U+0660–U+0669 | `۰`–`۹` U+06F0–U+06F9 | Arabic-Indic digits render as Urdu digits |
///
/// # What never folds
///
/// These four are the reason this crate exists. Arabic normalizers apply them
/// and silently corrupt Urdu:
///
/// - `آ` U+0622 is **never** `ا` — it is a distinct letter, not a decorated
/// alef. `آم` ("mango") and `ام` are different words.
/// - `ے` U+06D2 (bari ye) is **never** `ی` — `کے` ("of") and `کی` ("whose")
/// are different words.
/// - `ھ` U+06BE (do-chashmi he) is **never** `ہ` — it forms aspirated
/// consonants. `تھا` ("was") becomes `تہا`, which is not a word.
/// - Hamza `U+0654` / `U+0655` are not diacritics; see [`is_aerab_char`].
///
/// # Examples
///
/// ```
/// use urdu_text::fold_letter;
///
/// assert_eq!(fold_letter('\u{0643}'), '\u{06A9}'); // ك -> ک
/// assert_eq!(fold_letter('\u{0622}'), '\u{0622}'); // آ stays آ
/// assert_eq!(fold_letter('\u{06D2}'), '\u{06D2}'); // ے stays ے
/// assert_eq!(fold_letter('a'), 'a');
/// ```
// ---------------------------------------------------------------------------
// Aerab (diacritics)
// ---------------------------------------------------------------------------
/// Is this character an aerab (harakat / diacritic) that carries no meaning
/// once removed?
///
/// Urdu is normally written without aerab. When they do appear — in textbooks,
/// poetry, or religious text — they are pronunciation aids layered on top of a
/// spelling that is already complete. `کِتاب` and `کتاب` are the same word.
///
/// # Hamza is excluded, on purpose
///
/// `U+0654` (hamza above) and `U+0655` (hamza below) sit in the middle of the
/// Unicode combining-marks range and are classified as combining marks, so a
/// naive "strip all combining marks" implementation removes them. In Urdu
/// hamza is **part of the spelling**: `کوئی` ("someone") becomes `کوی`, which
/// is not a word. This function returns `false` for both, which is why the
/// aerab ranges below have a hole at U+0654–U+0655.
///
/// # Examples
///
/// ```
/// use urdu_text::is_aerab_char;
///
/// assert!(is_aerab_char('\u{064E}')); // zabar
/// assert!(is_aerab_char('\u{0670}')); // khari zabar (superscript alef)
/// assert!(!is_aerab_char('\u{0654}')); // hamza above — NOT an aerab
/// assert!(!is_aerab_char('\u{0655}')); // hamza below — NOT an aerab
/// assert!(!is_aerab_char('ک'));
/// ```
// ---------------------------------------------------------------------------
// Format / invisible marks
// ---------------------------------------------------------------------------
/// Is this an invisible formatting or bidirectional-control character?
///
/// These are stripped because they are invisible: two strings that differ only
/// by a zero-width joiner look identical to a human and must compare equal.
/// They are extremely common in text copied out of PDFs and web pages.
pub
// ---------------------------------------------------------------------------
// Digits
// ---------------------------------------------------------------------------
/// Map an Arabic-Indic or Urdu digit to the corresponding ASCII digit.
///
/// Covers both `U+0660`–`U+0669` (Arabic-Indic) and `U+06F0`–`U+06F9`
/// (Extended Arabic-Indic, used by Urdu). Returns `None` for anything else,
/// including characters that are already ASCII digits.
///
/// This is off by default in [`crate::normalize`] because converting `۲۰۲۴` to
/// `2024` changes how the text *reads*, which is a decision only the caller
/// can make. It is the right choice when you are indexing for search; it is
/// the wrong choice when you are round-tripping text for display.
///
/// # Examples
///
/// ```
/// use urdu_text::to_ascii_digit;
///
/// assert_eq!(to_ascii_digit('\u{06F5}'), Some('5')); // ۵ Urdu five
/// assert_eq!(to_ascii_digit('\u{0665}'), Some('5')); // ٥ Arabic-Indic five
/// assert_eq!(to_ascii_digit('5'), None); // already ASCII
/// assert_eq!(to_ascii_digit('ک'), None);
/// ```
// ---------------------------------------------------------------------------
// Arabic Presentation Forms-B
// ---------------------------------------------------------------------------
/// Base letters for `U+FE80`–`U+FEF4`, indexed by `ch as u32 - 0xFE80`.
///
/// Presentation forms are the pre-shaped isolated/initial/medial/final glyphs
/// that PDF generators and older Windows software emit instead of the abstract
/// letter plus a shaping engine. `ﻛ` (U+FEDB, kaf initial form) *is* the letter
/// kaf — it just has the joining already baked in. Text extracted from a PDF is
/// routinely full of these, and it will not match anything typed on a keyboard
/// until they are mapped back.
///
/// This table maps each form to its **Arabic** base codepoint. Folding to Urdu
/// happens afterwards, by running the result through [`fold_letter`], so the
/// two tables stay independent: `U+FEDB -> U+0643 -> U+06A9`.
///
/// Ligatures (`U+FEF5`–`U+FEFC`) expand to two characters and so live in
/// [`expand_ligature`] instead.
const PRESENTATION_FORMS: = ;
/// Map a single-character Arabic presentation form back to its base letter.
///
/// Handles `U+FE80`–`U+FEF4`. Returns `None` for anything else, including the
/// lam-alef ligatures `U+FEF5`–`U+FEFC`, which expand to *two* characters and
/// are handled by [`expand_ligature`].
///
/// The value returned is the **Arabic** base codepoint, not the Urdu one — run
/// it through [`fold_letter`] to finish the job. [`crate::normalize`] does both.
///
/// # Examples
///
/// ```
/// use urdu_text::{fold_letter, presentation_base};
///
/// // ﻛ U+FEDB is kaf in initial form.
/// assert_eq!(presentation_base('\u{FEDB}'), Some('\u{0643}'));
/// // Folding then takes Arabic kaf to Urdu keheh.
/// assert_eq!(fold_letter('\u{0643}'), '\u{06A9}');
///
/// assert_eq!(presentation_base('ک'), None);
/// ```
/// Expand a lam-alef presentation ligature into its two component letters.
///
/// `U+FEF5`–`U+FEFC` are the eight pre-composed lam+alef forms. They are a
/// single codepoint that renders as two letters, so any text containing one
/// will fail to match the spelled-out equivalent and will also give the wrong
/// character count.
///
/// Note that `U+FEF5`/`U+FEF6` expand to lam + **`آ`** (U+0622), not lam +
/// alef — the madda is part of the letter and is preserved.
///
/// # Examples
///
/// ```
/// use urdu_text::expand_ligature;
///
/// // ﻻ U+FEFB is the lam-alef ligature.
/// assert_eq!(expand_ligature('\u{FEFB}'), Some(('\u{0644}', '\u{0627}')));
/// // ﻵ keeps its madda.
/// assert_eq!(expand_ligature('\u{FEF5}'), Some(('\u{0644}', '\u{0622}')));
///
/// assert_eq!(expand_ligature('ل'), None);
/// ```