Crate mitex_lexer 

Given source strings, MiTeX Lexer provides a sequence of tokens

The core of the lexer is Lexer<'a, S> which receives a string &'a str and a TokenStream trait object S, then it provides public methods to peek and bump the token stream.

It has two main lexer implementations:

• Lexer<()>: provides plain tokens
  • See TokenStream for implementation
• Lexer<MacroEngine>: provides tokens with macro expansion
  • See MacroEngine for implementation

Modules

snapshot_map

Upstream rustc_data_structures::snapshot_map. Last checked commit: f4bb4500ddb4 Last checked time: 2023-12-28

Structs

Lexer

Small memory-efficient lexer for TeX

MacroEngine

MacroEngine has exact same interface as Lexer, but it expands macros.

Enums

BraceKind

Brace kinds in TeX, used by defining Token

CommandName

The command name used by parser

IfCommandName

The command name used by parser

Token

The token types defined in logos

Traits

MacroifyStream

Trait for querying macro state of a stream

TokenStream

A trait for bumping the token stream Its bumping is less frequently called than token peeking