SQL Differential Testing Oracles

This crate provides oracle implementations for differential SQL testing, inspired by the Crucible framework and SQLancer research.

Architecture

The OracleRunner trait defines a common interface for executing SQL queries against different database engines. Implementations include:

• DuckDbOracle: Executes queries in DuckDB (ground truth oracle)
• KimberliteOracle: Executes queries in Kimberlite (system under test)

Usage

use kimberlite_oracle::{OracleRunner, DuckDbOracle, KimberliteOracle};

let duckdb = DuckDbOracle::new()?;
let kimberlite = KimberliteOracle::new(/* ... */)?;

let sql = "SELECT COUNT(*) FROM users WHERE age > 30";
let duckdb_result = duckdb.execute(sql)?;
let kimberlite_result = kimberlite.execute(sql)?;

// Compare results to find bugs
assert_eq!(duckdb_result, kimberlite_result);
# Ok::<(), Box<dyn std::error::Error>>(())

Differential Testing Strategy

Differential testing compares two implementations of the same specification:

1. Generate a valid SQL query
2. Execute in both DuckDB (reference) and Kimberlite (SUT)
3. Compare results byte-by-byte
4. Report any discrepancies as bugs

Why DuckDB?

• Battle-tested SQL engine with 99.9% TPC-H compliance
• Embedded (no network overhead)
• Fast (columnar execution)
• Well-documented semantics

References

• Crucible: "Detecting Logic Bugs in DBMS" (154+ bugs found)
• SQLancer: Differential testing framework (148+ bugs in nghttp2)
• Jepsen: Distributed systems testing methodology