Module tokens 

Token types and tokenization for SQL parsing

This module defines all SQL token types and the tokenizer that converts SQL strings into token streams.

Structs

Span

Represents a position in the source SQL

Token

A token in the SQL token stream

Tokenizer

SQL Tokenizer

TokenizerConfig

Tokenizer configuration for a dialect

Enums

TokenType

All possible token types in SQL

Functions

parse_dollar_string_token

Parse a DollarString token text into (tag, content). If the text contains ‘\x00’, the part before is the tag and after is content. Otherwise, the whole text is the content with no tag.