-- Token and lexical rule definitions for a simple expression parser.
--
-- This file defines the lexical structure for an expression grammar.
-- It specifies tokens for identifiers, numbers, operators, parentheses,
-- whitespace, comments, etc.
--
-- The lexer supports multiple modes:
-- - <Expr>: main expression parsing state
-- - <Comment>: handles multi-line comment content
-- - <*>: matches any state (wildcard)
--
-- Comments in this lexer are **multi-lexeme tokens** — they can span multiple
-- regex matches (e.g., comment delimiters and inner text). To handle this,
-- the lexer uses an **accumulation mode** that automatically tracks the
-- source span and merges all fragments into a single `comment` token.
-- Basic token pattern definitions.
--
-- These are reusable regular expression patterns used to define
-- fundamental token types such as whitespace, newlines, identifiers,
-- and numbers. They serve as building blocks for later rules.
--
WS = [ \t]
NL = \r?\n
IDENT = [a-z_][a-z_A-Z0-9]*
NUMBER = [0-9]+
-- Lexer rules.
--
-- Each rule declares the mode(s) in which it is applicable.
-- These declarations do not cause mode changes; they only limit
-- where the rule may match.
--
Ident: <Expr> {{IDENT}}
Number: <Expr> {{NUMBER}}
Semicolon: <Expr> ;
Equals: <Expr> =
Plus: <Expr> \+
Minus: <Expr> -
Asterisk: <Expr> \*
Slash: <Expr> /
LeftParen: <Expr> \(
RightParen: <Expr> \)
CommentBegin: <Expr, Comment> /\*
CommentEnd: <Comment> \*/
CommentChar: <Comment> [^*/\r\n]+
NewLine: <*> {{NL}}
WhiteSpace: <Expr> {{WS}}+
Error: <*> .