Skip to main content

tokenize_with_offsets

spg_sql::lexer

Function tokenize_with_offsets 

Source 
pub fn tokenize_with_offsets(
    input: &str,
    backslash_escapes: bool,
) -> Result<(Vec<Token>, Vec<usize>), LexError>
Expand description

v7.39 (read01 round 95) — like tokenize_with but also returns, for each token, the byte offset in input where it started (the Eof token maps to input.len()). The parser uses this to translate a failing token index into PG’s 1-based character error position (the ErrorResponse P field that psql renders as LINE n: … ^).