Expand description
The Lexer module, which creates a vector of all of the tokens in the input. This input may come from either a file or a REPL. The Lexer struct tokenizes the input and returns a VecDeque of Tokens The lexer may prematurely return an error if it can not parse a specific character.
The lexer must be the first thing that is invoked because it generates the tokens necessary for the VM.
§Example
let contents = "push 1";
let tokens = Lexer::default().lex(contents)?;