sql-splitter

Split large SQL dump files into individual table files. Fast, memory-efficient, multi-dialect.

• 600+ MB/s throughput on modern hardware
• MySQL, PostgreSQL, SQLite support (including COPY FROM stdin)
• Compressed files — gzip, bzip2, xz, zstd auto-detected
• Streaming architecture — handles files larger than RAM
• 5x faster than shell-based alternatives

Installation

From crates.io

cargo install sql-splitter

From source

git clone https://github.com/helgesverre/sql-splitter
cd sql-splitter
make install  # Installs binary + shell completions

Or download pre-built binaries from GitHub Releases.

Usage

# MySQL/MariaDB dump (default)
sql-splitter split dump.sql -o tables/

# PostgreSQL pg_dump
sql-splitter split pg_dump.sql -o tables/ --dialect=postgres

# SQLite dump
sql-splitter split sqlite.sql -o tables/ --dialect=sqlite

# Compressed files (auto-detected)
sql-splitter split backup.sql.gz -o tables/
sql-splitter split backup.sql.zst -o tables/

# Split specific tables only
sql-splitter split dump.sql --tables users,posts,orders

# Schema only (CREATE TABLE, indexes, etc.)
sql-splitter split dump.sql -o schema/ --schema-only

# Data only (INSERT/COPY statements)
sql-splitter split dump.sql -o data/ --data-only

# Merge split files back into single dump
sql-splitter merge tables/ -o restored.sql

# Merge specific tables only
sql-splitter merge tables/ -o partial.sql --tables users,orders

# Merge with transaction wrapper
sql-splitter merge tables/ -o restored.sql --transaction

# Analyze without splitting
sql-splitter analyze dump.sql

# Convert between SQL dialects
sql-splitter convert mysql_dump.sql -o postgres_dump.sql --to postgres
sql-splitter convert pg_dump.sql -o mysql_dump.sql --to mysql
sql-splitter convert dump.sql -o sqlite_dump.sql --to sqlite

# Convert with explicit source dialect
sql-splitter convert dump.sql --from postgres --to mysql -o output.sql

# Generate shell completions (auto-installed with make install)
sql-splitter completions bash >> ~/.bashrc
sql-splitter completions zsh >> ~/.zshrc
sql-splitter completions fish >> ~/.config/fish/completions/sql-splitter.fish

Shell Completions

Shell completions are automatically installed when using make install. For manual installation:

# Install for current shell only
make install-completions

# Install for all shells (bash, zsh, fish)
make install-completions-all

Why sql-splitter?

sql-splitter is a dump-first, CLI-first tool designed for automation and CI/CD pipelines.

What it's optimized for

When another tool might be better

• mydumper — Parallel snapshots from live MySQL/MariaDB databases
• Jailer — Rich GUI-based FK subsetting with JDBC across 12+ databases
• sqlglot — Query-level transpilation and AST manipulation (31 dialects)
• DuckDB — Complex analytical queries over SQL/CSV/JSON/Parquet

See docs/COMPETITIVE_ANALYSIS.md for detailed comparisons.

Options

Split Options

Merge Options

Convert Options

Supported conversions:

• MySQL ↔ PostgreSQL (including COPY → INSERT)
• MySQL ↔ SQLite
• PostgreSQL ↔ SQLite

Features:

• 30+ data type mappings
• AUTO_INCREMENT ↔ SERIAL ↔ INTEGER PRIMARY KEY
• PostgreSQL COPY → INSERT with NULL and escape handling
• Session command stripping
• Warnings for unsupported features (ENUM, arrays, triggers)

Performance

See BENCHMARKS.md for detailed comparisons.

Testing

# Unit tests
cargo test

# Verify against real-world SQL dumps (MySQL, PostgreSQL, WordPress, etc.)
make verify-realworld

License

MIT — see LICENSE.md