Module lexer 

Structs

LexError

Self-contained lexical error including the offending span and a short snippet of the source text.

Lexer

Stateful Part 21 lexer that tracks line/column positions and wraps logos’s byte-level lexer.

Span

Position information for a lexed token.

Token

A lexed token carrying its kind and position.

Enums

LexErrorKind

Lexical error categories.

TokenKind

Part 21 token kinds.

Functions

tokenize

Tokenize a source string into a Vec<Token>, halting at the first lexical error. Callers that need streaming or custom recovery should construct a Lexer directly.