Struct Lexer 

pub struct Lexer<'a> { /* private fields */ }

SQL lexer that produces a stream of tokens from source text.

Implementations

impl<'a> Lexer<'a>

pub fn new(source: &'a str) -> Self

Create a new lexer for the given SQL source text.

pub fn tokenize(source: &'a str) -> Vec<Token>

Tokenize the entire input into a Vec of tokens.

pub fn tokenize_into(source: &'a str, tokens: &mut Vec<Token>)

Tokenize the entire input into a caller-owned buffer.

This preserves the buffer’s existing heap allocation across repeated parses so statement-level callers can treat token storage as lookaside scratch instead of rebuilding a fresh Vec<Token> on every miss.

pub fn metrics_snapshot() -> TokenizeMetricsSnapshot

Expose tokenize metrics as a snapshot.

pub fn reset_metrics()

Reset tokenize metrics.

pub fn next_token(&mut self) -> Token

Produce the next token.