Struct UnicodeNormalizer 

pub struct UnicodeNormalizer { /* private fields */ }

Unicode-aware text normalizer.

Supports NFC/NFD normalization, accent stripping, case folding, and language-agnostic CJK tokenization.

Example

use scirs2_text::tokenization::unicode_normalizer::{UnicodeNormalizer, UnicodeNormalizerConfig, NormForm};

let config = UnicodeNormalizerConfig {
    form: NormForm::Nfc,
    strip_accents: true,
    lowercase: true,
    tokenize_cjk: true,
};
let normalizer = UnicodeNormalizer::new(config);
let tokens = normalizer.tokenize_language_agnostic("Héllo 世界");
assert!(tokens.len() >= 3); // "hello", "世", "界"

Implementations

impl UnicodeNormalizer

pub fn new(config: UnicodeNormalizerConfig) -> Self

Create a new UnicodeNormalizer with the given configuration.

pub fn default_normalizer() -> Self

Create a normalizer with default settings.

pub fn normalize(&self, text: &str) -> String

Normalize text according to the configuration.

Steps applied in order:

1. Lowercase (if configured)
2. NFD decomposition + accent stripping (if configured)
3. NFC composition (if configured, after potential NFD strip)

pub fn tokenize_language_agnostic(&self, text: &str) -> Vec<String>

Tokenize text in a language-agnostic manner.

Algorithm:

1. Normalize the text.
2. Insert whitespace around CJK characters (when tokenize_cjk is set).
3. Split on Unicode whitespace.
4. Filter empty tokens.

This approach works across scripts without any language-specific logic.

pub fn config(&self) -> &UnicodeNormalizerConfig

Return the configuration.

Trait Implementations

impl Clone for UnicodeNormalizer

fn clone(&self) -> UnicodeNormalizer

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for UnicodeNormalizer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for UnicodeNormalizer

fn default() -> Self

Returns the “default value” for a type.