pub struct Options {
pub expand_presentation_forms: bool,
pub fold_letters: bool,
pub strip_aerab: bool,
pub strip_tatweel: bool,
pub strip_format_marks: bool,
pub ascii_digits: bool,
pub collapse_whitespace: bool,
}Expand description
Which normalization steps to apply.
Every field defaults to true except ascii_digits.
Use ..Options::default() to change one thing, or Options::none() to
start from all-off.
use urdu_text::Options;
let o = Options { collapse_whitespace: false, ..Options::default() };
assert!(o.fold_letters);
assert!(!o.collapse_whitespace);Fields§
§expand_presentation_forms: boolMap Arabic Presentation Forms-B (U+FE80–U+FEFC) back to base
letters, expanding the lam-alef ligatures into two characters.
Turn this on for anything extracted from a PDF. Default: true.
fold_letters: boolFold shape-identical Arabic and Persian letterforms to their Urdu
codepoints — see fold_letter for the exact table, including the
four folds this crate refuses to make.
Default: true.
strip_aerab: boolRemove aerab (harakat / diacritics), leaving hamza in place.
See is_aerab_char for why hamza is excluded. Default: true.
strip_tatweel: boolRemove tatweel (U+0640), the kashida elongation used to justify
lines of text. Default: true.
strip_format_marks: boolRemove invisible formatting and bidi control characters: zero-width space/joiner/non-joiner, LRM/RLM, the embedding and isolate controls, and the BOM.
Default: true.
ascii_digits: boolConvert Urdu and Arabic-Indic digits to ASCII 0–9.
Defaults to false, unlike every other field. Rewriting ۲۰۲۴ as
2024 changes how the text reads, so it is opt-in: right for a search
index, wrong for text you intend to display back to a reader.
collapse_whitespace: boolCollapse each run of whitespace to a single ASCII space, and trim the leading and trailing runs.
Runs after format marks are stripped, so a zero-width space wedged
between two blanks does not defeat the collapse. Default: true.
Implementations§
Source§impl Options
impl Options
Sourcepub const fn none() -> Self
pub const fn none() -> Self
Every step switched off — a starting point for building up exactly the normalization you want.
normalize_with(s, &Options::none()) returns s unchanged.
use urdu_text::{normalize_with, Options};
let only_ligatures = Options {
expand_presentation_forms: true,
..Options::none()
};
// The ligature expands to lam + alef, but Arabic kaf is left alone.
assert_eq!(
normalize_with("\u{FEFB}\u{0643}", &only_ligatures),
"\u{0644}\u{0627}\u{0643}"
);