Expand description
This module defines the Lexer struct and the LexerRule trait.
The Lexer is responsible for tokenizing the source code based on the provided rules.
The LexerRule trait defines the interface for lexer rules that can be used to generate
It also provides some utilities for common lexer rules.
Modules§
- utils
- This module provides utility functions and common lexer rules.
Structs§
- Lexer
- Tokenizes the source code.
Traits§
- Lexer
Rule - Defines the interface for lexer rules.