pub fn tokenize(source: &str) -> (Vec<Token>, Vec<Diagnostic>)Expand description
Splits source into tokens. Every byte of the source belongs to exactly one token, so the
concatenated token texts always equal the source. Malformed input produces
SyntaxKind::Unknown tokens and a diagnostic for each.