Skip to main content

segment_cjk

Function segment_cjk 

Source
pub fn segment_cjk(text: &str) -> String
Expand description

Insert a space after every CJK character so whitespace tokenization treats each CJK character as a separate token. Latin/digit/punctuation runs and existing whitespace are left untouched.

Examples: "知識グラフ""知 識 グ ラ フ ", "rust ownership""rust ownership", "Rustで非同期""Rust で 非 同 期 ".