Module lexer 

Lexer — logos-generated DFA producing a stream of LexToken.

Reference: spec 0001 §4.1. Every significant token carries its byte range; trivia (whitespace + comments) is surfaced as regular tokens so the parser can attach it to the enclosing node.

Case insensitivity is handled per-keyword via ignore(case) on the logos derive; the text of the token preserves the original casing so the formatter can honour user preference when requested.

Structs

LexError

A lexer-level diagnostic: a code (matching DiagCode discriminants in cyrs-diag) plus a message and byte range. Emitted by validate_tokens for errors that can only be detected after the DFA run — unterminated literals, invalid escape sequences, etc.

LexToken

A single lexed token: kind, original text, and byte range.

Functions

lex

Tokenise an entire source string. Unknown bytes become SyntaxKind::ERROR tokens that preserve their range; the lexer never panics on input.

validate_tokens

Scan a token stream for lex-level errors that the DFA cannot express directly: