pub fn tokenize(source: &str) -> Vec<Token>Expand description
Tokenize a .cabal source string into a flat list of Tokens.
The returned token list always ends with a TokenKind::Eof token.
Every byte of the input is covered by either a token span or a trivia
piece on one of the tokens.