pub fn tokenize(src: &str) -> Vec<RawToken>
Lex src into a lossless RawToken stream. Never fails: an unlexable byte becomes a SyntaxKind::Error token, so the concatenation of token ranges always reproduces the source.
src
RawToken
SyntaxKind::Error