fsqlite-parser

Hand-written SQL parser with direct statement/DDL routines and explicit heap-backed state machines: Pratt tasks for expressions and frames for SELECT trees. Converts SQL text into a typed AST defined in fsqlite-ast.

Overview

fsqlite-parser is the front-end of the FrankenSQLite query pipeline. It tokenizes raw SQL strings via a memchr-accelerated lexer, then parses them into a complete AST covering SELECT, INSERT, UPDATE, DELETE, CREATE TABLE, CREATE INDEX, CREATE VIEW, CREATE TRIGGER, ALTER TABLE, ATTACH, PRAGMA, transactions, and more.

A separate semantic analysis pass (Resolver) handles name resolution, scope tracking, and function arity validation.

Position in the dependency graph:

SQL text
  --> fsqlite-parser (this crate)
    --> fsqlite-ast (AST nodes)
    --> fsqlite-planner (query planning)
    --> fsqlite-vdbe (bytecode codegen)

Dependencies: fsqlite-types, fsqlite-error, fsqlite-ast, hashbrown, memchr, and tracing.

Key Types

• Lexer -- Tokenizer that converts SQL text into a stream of Token values. Uses memchr for fast string literal scanning and tracks line/column for error reporting.
• Token / TokenKind -- A single lexical token with span information and its variant (keyword, identifier, literal, operator, etc.).
• Parser -- SQL parser. Use Parser::from_sql() followed by Parser::parse_all() to produce Statement values and structured parse errors.
• ParseError -- Error type returned when parsing fails, with span and message.
• Resolver / Schema / Scope -- Semantic analysis layer for name resolution, column validation, and function arity checking after parsing.
• SemanticError -- Error type for semantic analysis failures (unknown column, ambiguous reference, wrong arity, etc.).
• ParseMetricsSnapshot / TokenizeMetricsSnapshot -- Point-in-time counters for parsed statements and tokenized tokens, useful for observability.

Usage

use fsqlite_parser::{Lexer, Parser};
use fsqlite_ast::Statement;

// Tokenize
let tokens = Lexer::tokenize("SELECT 1 + 2;");
assert!(!tokens.is_empty());

// Parse
let mut parser = Parser::from_sql(
    "SELECT name, age FROM users WHERE id = ?1;",
);
let (statements, errors) = parser.parse_all();
assert!(errors.is_empty(), "valid SQL: {errors:?}");
assert!(matches!(statements.as_slice(), [Statement::Select(_)]));

// Semantic analysis (optional, requires schema info)
use fsqlite_parser::{Resolver, Schema};
let schema = Schema::default();
let resolver = Resolver::new(&schema);
// resolver.resolve(&statements[0]) ...

License

MIT (with OpenAI/Anthropic Rider) -- see workspace root LICENSE file.