Given source strings, MiTeX Lexer provides a sequence of tokens
The core of the lexer is [Lexer<'a, S>] which receives a string &'a str
and a [TokenStream] trait object S, then it provides public methods to
peek and bump the token stream.
It has two main lexer implementations:
- [
Lexer<()>]: provides plain tokens- See [
TokenStream] for implementation
- See [
- [
Lexer<MacroEngine>]: provides tokens with macro expansion- See [
MacroEngine] for implementation
- See [