Module lexer 

Structs

SourceMap

Token

Enums

TokenKind

Functions

lex

Tokenise source into a flat list of tokens, ending with Eof. Whitespace and comments are skipped. The special comment -- allium: N at the very start of the file is preserved as an Ident token followed by normal tokens (version detection happens in the parser by inspecting the raw source before tokenising).