Struct Tokenizer 

pub struct Tokenizer { /* private fields */ }

Tokenizer for processing input text

Implementations

impl Tokenizer

pub fn new() -> Self

Create a new tokenizer with default configuration

pub fn tokenize_text(&self, text: &str) -> Vec<Token>

Tokenize input text into words and other tokens

pub fn tokenize_word(&self, word: &str) -> Vec<PhoneticUnit>

Tokenize a word into phonetic units for Bengali transliteration

pub fn tokenize_word_into(&self, word: &str, units: &mut Vec<PhoneticUnit>)

Tokenize a word into a caller-owned phonetic-unit buffer.

The buffer is cleared before use, but its existing allocation is reused. This is the preferred path for high-frequency callers such as editors or typing frontends that repeatedly analyze one changing word.

Trait Implementations

impl Default for Tokenizer

fn default() -> Self

Returns the “default value” for a type.