Struct UnicodeTokenizer 

pub struct UnicodeTokenizer { /* private fields */ }

Language-agnostic Unicode-aware tokenizer.

Handles:

• ASCII and multi-byte Unicode text
• CJK ideographs (each character becomes a separate token)
• Punctuation splitting
• Optional accent stripping (approximate, pure Rust — no ICU)
• Optional lowercasing

Implementations

impl UnicodeTokenizer

pub fn new(config: UnicodeTokenizerConfig) -> Self

Create a new tokenizer with the given configuration.

pub fn tokenize(&self, text: &str) -> Vec<String>

Tokenize text into a Vec<String> of Unicode-aware tokens.

pub fn encode(&self, text: &str, vocab: &HashMap<String, usize>) -> Vec<usize>

Tokenize text and convert tokens to vocabulary indices.

Unknown tokens are silently dropped.

pub fn is_cjk(c: char) -> bool

Returns true for CJK Unified Ideographs and CJK Extension A/B.

Covers:

• U+4E00–U+9FFF: CJK Unified Ideographs
• U+3400–U+4DBF: CJK Extension A
• U+20000–U+2A6DF: CJK Extension B

pub fn is_punctuation(c: char) -> bool

Returns true for ASCII punctuation and common Unicode punctuation.

pub fn strip_accents_approx(s: &str) -> String

Approximate accent stripping: remove characters in the Unicode combining diacritics range U+0300–U+036F.

This removes the combining marks but does not perform NFKD decomposition — for most Latin-script text the result is correct. Full NFKD would require the unicode-normalization crate.

Trait Implementations

impl Debug for UnicodeTokenizer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.