pub fn tokenize(src: &[u8], ofs: usize, tokens: &mut Vec<HashedSpan>)Expand description
Tokenize data from src from the position ofs into tokens.
pub fn tokenize(src: &[u8], ofs: usize, tokens: &mut Vec<HashedSpan>)Tokenize data from src from the position ofs into tokens.