pub fn tokenize(source: &str) -> Vec<(Token<'_>, Span)>Expand description
Tokenize source code into a vector of (Token, Span) pairs for the AST-style parser.
Filters out Token::Whitespace (mid-line horizontal whitespace)
but otherwise emits everything the lexer produces, with
post-processing for line-start # comments and indentation.
Callers that need a fully-lossless token stream (the CST builder)
use tokenize_lossless instead.