Installation
[dependencies]
lexer-lang = "1"
Usage
A lexer is a loop over a Cursor: peek at the next character, consume its run, and
emit a Token<K> of the language's own kind.
use intern_lang::Interner;
use token_lang::{Symbol, TokenKind};
use lexer_lang::Cursor;
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
enum Kind {
Ident(Symbol),
Number,
Whitespace,
Eof,
}
impl TokenKind for Kind {
fn is_trivia(&self) -> bool { matches!(self, Kind::Whitespace) }
fn is_eof(&self) -> bool { matches!(self, Kind::Eof) }
fn symbol(&self) -> Option<Symbol> {
match self { Kind::Ident(s) => Some(*s), _ => None }
}
}
let mut interner = Interner::new();
let mut cursor = Cursor::new("x42 7");
while let Some(c) = cursor.first() {
let kind = if c.is_whitespace() {
cursor.eat_while(char::is_whitespace);
Kind::Whitespace
} else if c.is_ascii_digit() {
cursor.eat_while(|c| c.is_ascii_digit());
Kind::Number
} else {
cursor.eat_while(|c| c.is_ascii_alphanumeric());
Kind::Ident(cursor.intern_lexeme(&mut interner))
};
let _token = cursor.emit(kind);
}
See docs/API.md for the full surface.
Status
Status: stable (1.0). v1.0.0 freezes the Cursor
scanner under Semantic Versioning: no breaking changes before 2.0. See
the ROADMAP and docs/API.md.
Contributing
See dev/DIRECTIVES.md for engineering standards and the definition of done. Before a PR: cargo fmt --all, cargo clippy --all-targets --all-features -- -D warnings, and cargo test --all-features must be clean.
