Macro declare_tokens 

declare_tokens!() { /* proc-macro */ }

Generates a token enum with Logos lexer integration.

This macro creates a token enum that implements the Logos trait for efficient lexical analysis. Each variant can specify a regex pattern or a literal string.

Syntax

declare_tokens! {
    error: ErrorType,
    tokens: {
        // Literal pattern
        Plus => "+",
        // Regex pattern
        Number => r"[0-9]+",
        // Skip pattern (not emitted as token)
        #[skip]
        Whitespace => r"[ \t\n]+",
    },
}

Generated Code

The macro generates:

• A Tok enum with variants for each token
• Logos derive implementation for lexing
• Display implementation for error messages

Example

use logos::Logos;

declare_tokens! {
    error: LexError,
    tokens: {
        #[token("+")]
        Plus,
        #[token("-")]
        Minus,
        #[regex(r"[0-9]+")]
        Number,
    },
}

let mut lexer = Tok::lexer("1 + 2");
assert_eq!(lexer.next(), Some(Ok(Tok::Number)));