Module lexer 

Lexer for the PG-dialect subset that SPG accepts.

v0.2 token stream is value-only — no source spans yet. Errors do report the byte offset where the offending construct started. Identifiers are ASCII case-folded to lower-case (matches PG when un-quoted). Quoted identifiers ("...") preserve case; "" is an embedded quote. String literals ('...') follow PG single-quote convention with '' as the embedded quote. The lexer accepts but does not interpret E-strings or dollar-quoted strings — those land in a later milestone.

Structs

LexError

Enums

LexErrorKind

Token

Functions

tokenize

Tokenize input into a Vec<Token> ending in Token::Eof.