This represents a lexer object.
A lexer takes a source file and lexes it into tokens.
Note that this struct should not be controlled manually,
lex function instead.
A static token. This must be sorted by length.
Simple function that lexes a source file into a token stream.
Exposes the functionality of the