Skip to main content

tokenize

Function tokenize 

Source
pub fn tokenize(
    text: &str,
    remove_stopwords: bool,
    cjk_mode: CjkTokenizerMode,
) -> Vec<String>
Expand description

General tokenization with optional stopword removal.

CJK tokenization mode is determined by cjk_mode:

  • Auto / Jieba: jieba word segmentation
  • Bigram: CJK character 2-grams
  • Char: single-character splitting

Equivalent to Python’s tokenize() in tokenizer.py.