Skip to main content

tokenize

Function tokenize 

Source
pub fn tokenize(src: &[u8], ofs: usize, tokens: &mut Vec<HashedSpan>)
Expand description

Tokenize data from src from the position ofs into tokens.