[][src]Function diffr_lib::tokenize

pub fn tokenize(src: &[u8], ofs: usize, tokens: &mut Vec<HashedSpan>)

Tokenize data from src from the position ofs into tokens.