Expand description
Lexical analyzer that transforms Rust source code into a stream of tokens.
Handles all Rust literal forms: strings (with escape sequences), characters,
integers (decimal, hex, binary, octal with _ separators), floats, and
keyword/identifier discrimination.