Skip to main content

tokenize_whitespace

Function tokenize_whitespace 

Source
pub fn tokenize_whitespace(s: &str) -> Vec<(usize, usize)>
Expand description

Simple whitespace tokenizer. Returns token spans as (start, end) byte offsets.