Expand description
§Multilingual Text Normalizer
Production-grade text normalization for multilingual content supporting:
- Unicode normalization (combining character removal)
- Script detection across Latin, Cyrillic, Arabic, CJK, and Devanagari scripts
- Language hint detection from script analysis
- Script-aware tokenization strategies
- Configurable normalization pipelines
Structs§
- Multilingual
Normalizer - Production-grade multilingual text normalizer.
- Normalization
Options - Configuration flags that control the normalization pipeline.
- Normalized
Text - The output produced by
MultilingualNormalizer::normalize. - Normalizer
Stats - A point-in-time snapshot of normalizer run statistics.
Enums§
- Language
Hint - High-level language hint inferred from script analysis.
- Script
- Unicode script classification for a run of characters.
- Tokenization
Strategy - Strategy that controls how normalized text is split into tokens.