pub fn lex(source: &str) -> Vec<Token>Expand description
Tokenise source into a flat list of tokens, ending with Eof.
Whitespace and comments are skipped. The special comment -- allium: N
at the very start of the file is preserved as an Ident token followed
by normal tokens (version detection happens in the parser by inspecting
the raw source before tokenising).