pub fn lex(source: &str) -> (Vec<Token>, Vec<LexError>)Expand description
Lex the given source text into a sequence of tokens.
Returns a vector of tokens and any errors encountered. Even if errors occur, tokens are still produced to enable error recovery in the parser.