Pattern Lexer
My personal implementation of a lexer.
Principle
This lexer is making use of the Pattern
trait to find tokens.
The idea is to create Tokens
, explain how to match them with a Pattern
and build them from the matched String
value.
Pattern
A string Pattern
trait.
The type implementing it can be used as a pattern for &str
,
by default it is implemented for the following types.
Pattern type | Match condition |
---|---|
char |
is contained in string |
&str |
is substring |
String |
is substring |
&[char] |
any char match |
&[&str] |
any &str match |
F: Fn(&str) -> bool |
F returns true for substring (slow) |
Regex |
Regex match substring |
Usage
The lexer!
macro match the following syntax.
lexer!
It generates module gen
which contains Token
, LexerError
, LexerResult
and Lexer
.
You can now call Token::tokenize
to tokenize a &str
,
it should return a Lexer
instance that implements Iterator
.
Each iteration, the Lexer
tries to match one of the given Pattern
and returns a LexerResult<Token>
built from the best match.
Example
Here is an example for a simple math lexer.
lexer!;
That will expand to these enum and structs.
And you can use them afterwards.
use *;
let mut lex = tokenize;
assert_eq!;
assert_eq!;
// Our lexer doesn't handle parenthesis...
let mut err = tokenize;
assert!;