Struct MultilingualNormalizer 

pub struct MultilingualNormalizer {
    pub options: NormalizationOptions,
    pub tokenization: TokenizationStrategy,
    /* private fields */
}

Production-grade multilingual text normalizer.

MultilingualNormalizer executes a configurable normalization pipeline on arbitrary Unicode text and tokenizes the result according to the chosen TokenizationStrategy. It is internally thread-safe via atomic counters wrapped in an Arc, so a single instance may be shared across threads.

Example

use ipfrs_semantic::multilingual_normalizer::{
    MultilingualNormalizer, NormalizationOptions, TokenizationStrategy,
};

let opts  = NormalizationOptions { lowercase: true, ..Default::default() };
let norm  = MultilingualNormalizer::new(opts, TokenizationStrategy::Whitespace);
let result = norm.normalize("Hello World");
assert_eq!(result.normalized, "hello world");
assert_eq!(result.token_count, 2);

Fields

options: NormalizationOptions

Active normalization options.

tokenization: TokenizationStrategy

Active tokenization strategy.

Implementations

impl MultilingualNormalizer

pub fn new( options: NormalizationOptions, tokenization: TokenizationStrategy, ) -> Self

Create a new normalizer with the given options and tokenization strategy.

pub fn with_defaults() -> Self

Create a normalizer with default options and Whitespace tokenization.

pub fn normalize(&self, text: &str) -> NormalizedText

Normalize a single text string and tokenize it.

The pipeline is:

1. Detect dominant script.
2. Derive a language hint from the script.
3. Apply all enabled normalization options in order.
4. Tokenize according to the configured strategy.
5. Update internal statistics.

pub fn normalize_batch(&self, texts: &[&str]) -> Vec<NormalizedText>

Normalize a batch of text strings.

pub fn detect_script(&self, text: &str) -> Script

Detect the dominant script in text by counting characters per script.

Returns Script::Unknown when the text is empty or when two scripts tie for the highest count.

pub fn detect_language(&self, text: &str) -> LanguageHint

Derive a language hint from the dominant script in text.

pub fn apply_options(&self, text: &str) -> String

Apply all enabled normalization options to text and return the result.

Options are applied in this fixed order:

1. lowercase
2. strip_accents
3. normalize_whitespace
4. remove_punctuation
5. max_length
6. script_filter

pub fn tokenize(&self, text: &str) -> Vec<String>

Tokenize text according to the configured TokenizationStrategy.

pub fn strip_combining_chars(&self, text: &str) -> String

Remove Unicode combining diacritical marks (U+0300-U+036F) from text.

pub fn stats(&self) -> NormalizerStats

Return a point-in-time snapshot of accumulated statistics.

Trait Implementations

impl Debug for MultilingualNormalizer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.