Struct Tokenizer 

pub struct Tokenizer {
    pub preserve_comments: bool,
    /* private fields */
}

SQL tokenizer that converts a SQL string into a stream of tokens.

Tracks line and column numbers for error reporting. Supports:

• Single-line comments (--)
• Block comments (/* ... */)
• Quoted identifiers ("..." and backtick)
• String literals with escape handling
• Multi-character operators (<=, >=, <>, !=, ||, ::, ->, ->>)

Fields

preserve_comments: bool

Whether to preserve comments as tokens.

Implementations

impl Tokenizer

pub fn new(input: &str) -> Self

Create a new tokenizer for the given SQL input.

pub fn with_comments(input: &str) -> Self

Create a tokenizer that preserves comment tokens.

pub fn tokenize(&mut self) -> Result<Vec<Token>>

Tokenize the entire input and return a vector of tokens.

Whitespace tokens are skipped. Comments are optionally preserved.