Performs lexical analysis on string inputs
The Lexer module transforms strings into tokens using Strum.