pub fn tokenize(
text: &str,
remove_stopwords: bool,
cjk_mode: CjkTokenizerMode,
) -> Vec<String>Expand description
General tokenization with optional stopword removal.
CJK tokenization mode is determined by cjk_mode:
Auto/Jieba: jieba word segmentationBigram: CJK character 2-gramsChar: single-character splitting
Equivalent to Python’s tokenize() in tokenizer.py.