Crate logos

Create ridiculously fast Lexers.

Logos works by:

• Resolving all logical branching of token definitions into a tree.
• Optimizing complex patterns into Lookup Tables.
• Always using a Lookup Table for the first byte of a token.
• Producing code that never backtracks, thus running at linear time or close to it.

In practice it means that for most grammars the lexing performance is virtually unaffected by the number of tokens defined in the grammar. Or, in other words, it is really fast.

Example

extern crate logos;

use logos::Logos;

#[derive(Logos, Debug, PartialEq)]
enum Token {
    // Logos requires that we define two default variants,
    // one for end of input source,
    #[end]
    End,

    // ...and one for errors. Those can be named anything
    // you wish as long as the attributes are there.
    #[error]
    Error,

    // Tokens can be literal strings, of any length.
    #[token = "fast"]
    Fast,

    #[token = "."]
    Period,

    // Or regular expressions.
    #[regex = "[a-zA-Z]+"]
    Text,
}

fn main() {
    let mut lexer = Token::lexer("Create ridiculously fast Lexers.");

    assert_eq!(lexer.token, Token::Text);
    assert_eq!(lexer.slice(), "Create");
    assert_eq!(lexer.range(), 0..6);

    lexer.advance();

    assert_eq!(lexer.token, Token::Text);
    assert_eq!(lexer.slice(), "ridiculously");
    assert_eq!(lexer.range(), 7..19);

    lexer.advance();

    assert_eq!(lexer.token, Token::Fast);
    assert_eq!(lexer.slice(), "fast");
    assert_eq!(lexer.range(), 20..24);

    lexer.advance();

    assert_eq!(lexer.token, Token::Text);
    assert_eq!(lexer.slice(), "Lexers");
    assert_eq!(lexer.range(), 25..31);

    lexer.advance();

    assert_eq!(lexer.token, Token::Period);
    assert_eq!(lexer.slice(), ".");
    assert_eq!(lexer.range(), 31..32);

    lexer.advance();

    assert_eq!(lexer.token, Token::End);
}

Re-exports

pub use logos_derive::Logos;

pub use self::source::Source;

pub use self::source::Slice;

Modules

source

This module contains a bunch of traits necessary for processing byte strings.

Macros

lookup

Macro for creating lookup tables where index matches the token variant as usize.

Structs

Lexer

Lexer is the main struct of the crate that allows you to read through a Source and produce tokens for enums implementing the Logos trait.

Traits

Extras	Helper trait that can be injected into the Lexer to handle things that aren't necessarily tokens, such as comments or Automatic Semicolon Insertion in JavaScript.
Logos	Trait implemented for an enum representing all tokens. You should never have to implement it manually, use the #[derive(Logos)] attribute on your enum.

Type Definitions

Lexicon

A Lookup Table used internally. It maps indices for every valid byte to a function that takes a mutable reference to the Lexer, reads the input and sets the correct token variant for it.