Ferrelex

Ferrelex is a crate that enables you to write powerful, Unicode-aware lexers entirely in Rust.

Ferrelex is a lexer generator: it leverages procedural macros to let you describe token patterns as readable regex expressions and match arms, then compiles them into an efficient DFA at compile time — no runtime regex engine, no heap allocation per token. You get the expressiveness of a traditional lexer generator tool (think ocamllex or flex) with full access to Rust's type system, pattern matching, and error handling.

Example

use ferrelex::{lexer::lex, lexbuf::{utf8::LexBuf, refiller::Utf8Refiller}};

// You can use any type you'd like as the return type of your lexer.
#[derive(Debug, PartialEq, Eq)]
enum Token {
    Ascii(String),
    Lambda,
    Eof,
    Invalid
}

// The lex! macro is where your regex definitions and match arms live.
lex! {
    // Regexes are constants of type `Regex`.
    const ASCII_LETTER: Regex = ('a'..='z') | ('A'..='Z');
    // Some characters don't fit in a Rust char literal — use a string instead.
    const LAMBDA: Regex = "λ";

    // Your lexer lives inside an ordinary Rust function. It must accept a
    // `&mut LexBuf` argument; the return type and any extra arguments are
    // entirely up to you.
    pub fn my_lexer(lexbuf: &mut LexBuf) -> Token {
        // Place `#[lexer]` before your match expression to activate the DFA.
        #[lexer]
        match lexbuf {
            ASCII_LETTER => Token::Ascii(lexbuf.lexeme()),
            LAMBDA => Token::Lambda,
            eof => Token::Eof,
            // The last arm must be a wildcard to handle unmatched input.
            _ => Token::Invalid,
        }
    }
}

fn main() {
    // Create a LexBuf from any input source — here, an owned String.
    let mut lexbuf = LexBuf::new(Utf8Refiller::new(String::from("λhello")));

    // Call your lexer function once per token. The LexBuf advances automatically.
    assert_eq!(my_lexer(&mut lexbuf), Token::Lambda);
    assert_eq!(my_lexer(&mut lexbuf), Token::Ascii(String::from("h")));
    assert_eq!(my_lexer(&mut lexbuf), Token::Ascii(String::from("e")));
    assert_eq!(my_lexer(&mut lexbuf), Token::Ascii(String::from("l")));
    assert_eq!(my_lexer(&mut lexbuf), Token::Ascii(String::from("l")));
    assert_eq!(my_lexer(&mut lexbuf), Token::Ascii(String::from("o")));

    // Once you reach the end of input, every call returns Token::Eof.
    assert_eq!(my_lexer(&mut lexbuf), Token::Eof);
    assert_eq!(my_lexer(&mut lexbuf), Token::Eof);
}

Regex syntax

Regex expressions inside lex! use Rust syntax and are evaluated at compile time:

¹ Compl, Sub, and Intersect require both operands to be single-character-class regexes. Prefer char literals (e.g. '"') over single-character string literals (e.g. "\"") as arguments — a string literal creates a sequence internally and will be rejected by these operators.

Built-in regex constants

The following names are always in scope inside lex!, without any const definition:

All built-in shorthands are ASCII-only — the _ascii suffix makes that explicit. For full Unicode coverage, use Unicode category and property names directly as identifiers — see the sections below.

Unicode General Categories

The two-letter Unicode General Category codes are available as identifiers directly inside lex!. Single-letter super-categories union all categories sharing that letter prefix.

Specific categories

Super-category aggregates

Unicode Derived Properties

The following Unicode derived core property names are available as identifiers inside lex! (Unicode 17.0.0, from DerivedCoreProperties.txt):

Skipping tokens with #[skip]

Annotate a match arm with #[skip] to consume the matched input and restart the DFA immediately, without returning to the caller. This is the idiomatic way to discard whitespace or comments:

#[lexer]
match lexbuf {
    #[skip] whitespace_ascii       => {}
    #[skip] ("//", Star(any))      => {}  // line comment
    IDENT => Token::Ident(lexbuf.lexeme()),
    eof   => Token::Eof,
    _     => Token::Error,
}

Specialized lexer functions

Some tokens (string literals, block comments, heredocs, ...) require different lexing rules mid-stream. The natural approach is to call a specialized lexer function defined in the same lex! block. To avoid stack overflows on long inputs, use loop { #[lexer] match … } inside the inner function instead of recursion:

lex! {
    const NOT_DQUOTE: Regex = Sub(any, '"');

    pub fn token(lexbuf: &mut LexBuf) -> Token {
        #[lexer]
        match lexbuf {
            '"'   => lex_string(lexbuf),
            IDENT => Token::Ident(lexbuf.lexeme()),
            eof   => Token::Eof,
            _     => Token::Error,
        }
    }

    // Inner lexer: iterative, no stack growth.
    pub fn lex_string(lexbuf: &mut LexBuf) -> Token {
        let mut acc = String::new();
        loop {
            #[lexer]
            match lexbuf {
                '"'        => return Token::Str(acc),
                NOT_DQUOTE => acc += lexbuf.lexeme_str(),
                eof | _    => return Token::Error,
            }
        }
    }
}

Note: each #[lexer] match resets the token start position. If you need the position of the opening ", save it before calling the inner function: let start = lexbuf.start_pos();

#[lexer] options

The #[lexer] attribute accepts optional settings:

#[lexer(no_line_tracking)]
match lexbuf {
    _ => ()
}

Extracting the matched text

Inside a match arm body, the lexbuf argument exposes the matched input:

Position tracking

Token positions are tracked automatically. These methods are accurate inside an arm body, after a successful match:

Position contains line (1-indexed), col (0-indexed character offset from the start of the line), and filename.

Invalid UTF-8

When the input contains an invalid UTF-8 byte, the DFA fires the wildcard (_) arm and sets lexbuf.invalid_byte to Some(byte). Check this field to distinguish invalid bytes from valid-but-unmatched characters:

_ => match lexbuf.invalid_byte {
    Some(b) => Token::InvalidByte(b),
    None    => Token::UnexpectedChar,
}

lexbuf.invalid_byte is cleared at the start of each new match. Calling lexbuf.lexeme() or lexbuf.lexeme_str() in the wildcard arm when an invalid byte triggered it produces a deprecation warning at compile time; use lexbuf.lexeme_bytes() instead.

Input sources

Four ready-to-use type aliases live in [lexbuf::utf8]:

utf8::LexBuf and utf8::CachingLexBuf are backed by a Refiller that feeds chunks of bytes into an internal buffer. Three Refiller implementations are provided:

Implement Refiller yourself to support any other streaming source. For complete control over the input and cache strategies, use the underlying LexBuf<I, C> directly with your own Input and CharCache implementations.