Struct UnicodeTokenizer 

pub struct UnicodeTokenizer { /* private fields */ }

Language-agnostic Unicode tokenizer.

Works for any writing system:

• CJK ideographs become individual tokens (spaces inserted around them).
• Optional accent stripping via a pure-Rust NFD approximation.
• Optional lowercasing.
• Punctuation splitting (Unicode category Po/Ps/Pe/…).

No external Unicode library is required.

Implementations

impl UnicodeTokenizer

pub fn new(config: UnicodeTokenizerConfig) -> Self

Create a new tokenizer with the given configuration.

pub fn default_tokenizer() -> Self

Create a tokenizer with sensible defaults: lowercase=true, strip_accents=true, split on whitespace + punctuation.

pub fn tokenize(&self, text: &str) -> Vec<String>

Tokenize text into a list of tokens (Unicode-aware).

Processing order:

1. Insert spaces around CJK characters.
2. Optionally lowercase.
3. Optionally strip accents.
4. Split on whitespace (always) and optionally on punctuation.
5. Discard empty tokens and enforce max_token_length.

pub fn detect_script(&self, text: &str) -> ScriptFamily

Detect the dominant script family of text by majority vote over non-whitespace characters.

pub fn tokenize_cjk(&self, text: &str) -> Vec<String>

Tokenize a (potentially mixed) text where CJK characters each become their own token, while non-CJK sequences are tokenized by whitespace.

Trait Implementations

impl Default for UnicodeTokenizer

fn default() -> Self

Returns the “default value” for a type.