Crate logos

Logos λόγος

Create ridiculously fast Lexers.

extern crate logos;
#[macro_use]
extern crate logos_derive;

use logos::Logos;

#[derive(Debug, PartialEq, Logos)]
enum Token {
    #[end]
    End,

    #[error]
    Error,

    #[token = "."]
    Period,

    #[regex = "[a-zA-Z]*"]
    Text,
}

fn main() {
    let mut lexer = Token::lexer("Create ridiculously fast Lexers.");

    assert_eq!(lexer.token, Token::Text);
    assert_eq!(lexer.slice(), "Create");
    assert_eq!(lexer.range(), 0..6);

    lexer.advance();

    assert_eq!(lexer.token, Token::Text);
    assert_eq!(lexer.slice(), "ridiculously");
    assert_eq!(lexer.range(), 7..19);

    lexer.advance();

    assert_eq!(lexer.token, Token::Text);
    assert_eq!(lexer.slice(), "fast");
    assert_eq!(lexer.range(), 20..24);

    lexer.advance();

    assert_eq!(lexer.token, Token::Text);
    assert_eq!(lexer.slice(), "Lexers");
    assert_eq!(lexer.range(), 25..31);

    lexer.advance();

    assert_eq!(lexer.token, Token::Period);
    assert_eq!(lexer.slice(), ".");
    assert_eq!(lexer.range(), 31..32);

    lexer.advance();

    assert_eq!(lexer.token, Token::End);
}

Structs

Lexer

Lexer is the main struct of the crate that allows you to read through a Source and produce tokens implementing the Logos trait.

Traits

Extras

Helper trait that can be injected into the Lexer to handle things that aren’t necessarily tokens, such as comments or Automatic Semicolon Insertion in JavaScript.

Logos

Trait implemented for an enum representing all tokens. You should never have to implement it manually, use the #[derive(Logos)] attribute on your enum.

Source

Trait for types the Lexer can read from.

Type Definitions

Lexicon

A lookup table used internally. It maps indices for every ASCII byte to a function that takes a mutable reference to the Lexer.