Expand description
Create ridiculously fast Lexers.
extern crate logos;
#[macro_use]
extern crate logos_derive;
use logos::Logos;
#[derive(Debug, PartialEq, Logos)]
enum Token {
#[end]
End,
#[error]
Error,
#[token = "."]
Period,
#[token = "fast"]
Fast,
#[regex = "[a-zA-Z]*"]
Text,
}
fn main() {
let mut lexer = Token::lexer("Create ridiculously fast Lexers.");
assert_eq!(lexer.token, Token::Text);
assert_eq!(lexer.slice(), "Create");
assert_eq!(lexer.range(), 0..6);
lexer.advance();
assert_eq!(lexer.token, Token::Text);
assert_eq!(lexer.slice(), "ridiculously");
assert_eq!(lexer.range(), 7..19);
lexer.advance();
assert_eq!(lexer.token, Token::Fast);
assert_eq!(lexer.slice(), "fast");
assert_eq!(lexer.range(), 20..24);
lexer.advance();
assert_eq!(lexer.token, Token::Text);
assert_eq!(lexer.slice(), "Lexers");
assert_eq!(lexer.range(), 25..31);
lexer.advance();
assert_eq!(lexer.token, Token::Period);
assert_eq!(lexer.slice(), ".");
assert_eq!(lexer.range(), 31..32);
lexer.advance();
assert_eq!(lexer.token, Token::End);
}
Structs
Lexer
is the main struct of the crate that allows you to read through a
Source
and produce tokens implementing the Logos
trait.Traits
Helper trait that can be injected into the
Lexer
to handle things that
aren’t necessarily tokens, such as comments or Automatic Semicolon Insertion
in JavaScript.Trait implemented for an enum representing all tokens. You should never have
to implement it manually, use the
#[derive(Logos)]
attribute on your enum.Trait for types the
Lexer
can read from.Type Definitions
A lookup table used internally. It maps indices for every ASCII
byte to a function that takes a mutable reference to the
Lexer
.