sql-splitter

Split large SQL dump files into individual table files. Fast, memory-efficient, multi-dialect.

• 600+ MB/s throughput on modern hardware
• MySQL, PostgreSQL, SQLite support (including COPY FROM stdin)
• Compressed files — gzip, bzip2, xz, zstd auto-detected
• Streaming architecture — handles files larger than RAM
• 5x faster than shell-based alternatives

Installation

From crates.io

cargo install sql-splitter

From source

git clone https://github.com/helgesverre/sql-splitter
cd sql-splitter
make install  # Installs binary + shell completions + man pages

Or download pre-built binaries from GitHub Releases.

Man pages (optional)

After installation, view documentation with man sql-splitter or man sql-splitter-diff.

For cargo install users, install man pages manually:

git clone https://github.com/helgesverre/sql-splitter
cd sql-splitter
make install-man

Usage

# MySQL/MariaDB dump (default)
sql-splitter split dump.sql -o tables/

# PostgreSQL pg_dump
sql-splitter split pg_dump.sql -o tables/ --dialect=postgres

# SQLite dump
sql-splitter split sqlite.sql -o tables/ --dialect=sqlite

# Compressed files (auto-detected)
sql-splitter split backup.sql.gz -o tables/
sql-splitter split backup.sql.zst -o tables/

# Split specific tables only
sql-splitter split dump.sql --tables users,posts,orders

# Schema only (CREATE TABLE, indexes, etc.)
sql-splitter split dump.sql -o schema/ --schema-only

# Data only (INSERT/COPY statements)
sql-splitter split dump.sql -o data/ --data-only

# Merge split files back into single dump
sql-splitter merge tables/ -o restored.sql

# Merge specific tables only
sql-splitter merge tables/ -o partial.sql --tables users,orders

# Merge with transaction wrapper
sql-splitter merge tables/ -o restored.sql --transaction

# Analyze without splitting
sql-splitter analyze dump.sql

# Convert between SQL dialects
sql-splitter convert mysql_dump.sql -o postgres_dump.sql --to postgres
sql-splitter convert pg_dump.sql -o mysql_dump.sql --to mysql
sql-splitter convert dump.sql -o sqlite_dump.sql --to sqlite

# Convert with explicit source dialect
sql-splitter convert dump.sql --from postgres --to mysql -o output.sql

# Validate SQL dump integrity
sql-splitter validate dump.sql

# Validate with strict mode (warnings = errors)
sql-splitter validate dump.sql --strict

# Validate with JSON output for CI
sql-splitter validate dump.sql --json

# Batch operations with glob patterns
sql-splitter validate "dumps/*.sql" --fail-fast
sql-splitter analyze "**/*.sql"
sql-splitter split "*.sql" -o output/
sql-splitter convert "*.sql" --to postgres -o converted/

# Compare two SQL dumps for changes
sql-splitter diff old.sql new.sql

# Diff with schema-only or data-only
sql-splitter diff old.sql new.sql --schema-only
sql-splitter diff old.sql new.sql --data-only

# Diff with JSON or SQL migration output
sql-splitter diff old.sql new.sql --format json -o diff.json
sql-splitter diff old.sql new.sql --format sql -o migration.sql

# Diff with verbose PK samples and ignore timestamp columns
sql-splitter diff old.sql new.sql --verbose --ignore-columns "*.updated_at,*.created_at"

# Override primary key for tables without PK
sql-splitter diff old.sql new.sql --primary-key logs:timestamp+message

# Generate shell completions (auto-installed with make install)
sql-splitter completions bash >> ~/.bashrc
sql-splitter completions zsh >> ~/.zshrc
sql-splitter completions fish >> ~/.config/fish/completions/sql-splitter.fish

Shell Completions

Shell completions are automatically installed when using make install. For manual installation:

# Install for current shell only
make install-completions

# Install for all shells (bash, zsh, fish)
make install-completions-all

Why sql-splitter?

sql-splitter is a dump-first, CLI-first tool designed for automation and CI/CD pipelines.

What it's optimized for

When another tool might be better

• mydumper — Parallel snapshots from live MySQL/MariaDB databases
• Jailer — Rich GUI-based FK subsetting with JDBC across 12+ databases
• sqlglot — Query-level transpilation and AST manipulation (31 dialects)
• DuckDB — Complex analytical queries over SQL/CSV/JSON/Parquet

See docs/COMPETITIVE_ANALYSIS.md for detailed comparisons.

Options

Split Options

Input can be a file path or glob pattern (e.g., *.sql, dumps/**/*.sql).

Merge Options

Analyze Options

Input can be a file path or glob pattern (e.g., *.sql, dumps/**/*.sql).

Convert Options

Input can be a file path or glob pattern (e.g., *.sql, dumps/**/*.sql).

Supported conversions:

• MySQL ↔ PostgreSQL (including COPY → INSERT)
• MySQL ↔ SQLite
• PostgreSQL ↔ SQLite

Features:

• 30+ data type mappings
• AUTO_INCREMENT ↔ SERIAL ↔ INTEGER PRIMARY KEY
• PostgreSQL COPY → INSERT with NULL and escape handling
• Session command stripping
• Warnings for unsupported features (ENUM, arrays, triggers)

Validate Options

Input can be a file path or glob pattern (e.g., *.sql, dumps/**/*.sql).

Validation checks:

• SQL syntax validation (parser errors)
• DDL/DML consistency (INSERTs reference existing tables)
• Encoding validation (UTF-8)
• Duplicate primary key detection (all dialects)
• FK referential integrity (all dialects)

Sample Options

Shard Options

Diff Options

What diff detects:

• Tables added/removed/modified (columns, types, nullability)
• Primary key changes
• Foreign key changes
• Index changes (CREATE INDEX, inline INDEX/KEY)
• Rows added/removed/modified (via PK-based comparison)

Output formats:

• text: Human-readable summary with optional PK samples
• json: Structured data for automation (includes warnings)
• sql: Migration script with ALTER/CREATE INDEX/DROP INDEX statements

Performance

See BENCHMARKS.md for detailed comparisons.

Testing

# Unit tests
cargo test

# Verify against real-world SQL dumps (MySQL, PostgreSQL, WordPress, etc.)
make verify-realworld

AI Agent Integration

sql-splitter includes documentation optimized for AI agents:

• llms.txt - LLM-friendly documentation following the llmstxt.org specification
• Agent Skill - Claude Code / Amp skill for automatic tool discovery

Install the skill in Claude Code / Amp:

amp skill add helgesverre/sql-splitter

License

MIT — see LICENSE.md