Function wana_kana::tokenize::tokenize_with_opt

pub fn tokenize_with_opt(input: &str, compact: bool) -> Vec<String> 

Available on crate feature tokenize only.

Tokenizes the text. Splits input into array of strings separated by opinionated TokenType.

If compact is set, many same-language tokens are combined (spaces + text, kanji + kana, numeral + punctuation).