Skip to main content

tokenize

Function tokenize 

Source
pub fn tokenize(src: &str) -> Vec<RawToken>
Expand description

Lex src into a lossless RawToken stream. Never fails: an unlexable byte becomes a SyntaxKind::Error token, so the concatenation of token ranges always reproduces the source.