Crate sql_dialect_fmt_lexer 

Hand-written, lossless, error-resilient lexer for Snowflake SQL.

Design goals (mirroring Biome / rust-analyzer):

• Lossless — every byte of the input belongs to exactly one token, so the original source can always be reconstructed by concatenating token texts. This is what makes accurate formatting, comment preservation, and syntax highlighting possible.
• Error-resilient — malformed input (unterminated strings/comments) still produces tokens plus diagnostics, never a panic. Mid-edit SQL must still lex.
• Fast — single pass over the bytes, zero allocation per token (texts borrow the input), no regex.

The lexer is deliberately “dumb”: it emits SyntaxKind::IDENT for every keyword-like word. The parser reclassifies keywords contextually via sql_dialect_fmt_syntax::keyword_kind, because many SQL keywords (LEFT, ROW, VALUE, …) are contextual and may be identifiers.

Modules

• token — the Token / LexError / Lexed types.
• lexer — the single-pass tokenizer (tokenize).

Structs

BodyDelimiter

LexError

A lexical diagnostic (e.g. an unterminated literal), located at a byte span.

LexOptions

Lexed

The result of lexing: the token stream plus any diagnostics.

Token

A single lexed token. text borrows the source, so the lexer allocates nothing per token.

Enums

Dialect

The SQL dialect a lex/parse/format request targets.

SyntaxKind

Every lexical token kind and syntax node kind understood by sql-dialect-fmt.

Constants

DEFAULT_BODY_DELIMITERS

DOLLAR_QUOTED_BODY

Functions

tokenize

Tokenize Snowflake SQL into a lossless token stream.

tokenize_for_dialect

Tokenize input for dialect, using that dialect’s default lexing rules.

tokenize_with_options