pub fn tokenize(source: &str) -> LexResultExpand description
Tokenize source.
Always succeeds. Malformed input produces SyntaxKind::Unknown tokens and
entries in LexResult::errors, never a hard failure.
pub fn tokenize(source: &str) -> LexResultTokenize source.
Always succeeds. Malformed input produces SyntaxKind::Unknown tokens and
entries in LexResult::errors, never a hard failure.