DuckBridge

DuckDB ⇄ SQL Server / PostgreSQL / Parquet bridge. Keep your database of record where it is, do the heavy processing in DuckDB, ship the result back as a genuine, restorable SQL Server .bak, a live load, a PostgreSQL load, or Parquet files. Built to grow: each target is a self-contained module (src/mssql, src/pg, src/parquet.rs) plus a dialect rule set in src/transpile, so adding another engine means adding one of each.

One static binary (Linux, macOS, Windows). No ODBC driver, no Python, no bcp needed on the machine.

 SQL Server ──import──▶ work.duckdb ──run translated T-SQL──▶ processed ──export──▶ .bak / live load / script
                                                                                  ──▶ PostgreSQL / Parquet

Commands

Input files and outputs accept local paths or s3://bucket/key, az://container/blob, gs://bucket/key. Cloud clients are built in (SigV4 / Azure Shared Key or SAS), credentials come from the usual environment variables.

What changed in 0.1.1

0.1.1 makes loads into shared production servers safe (see Loading into production tables safely). New features are new flags or subcommands; the behaviour changes below are on by default.

Quick start

export MSSQL_PASSWORD='...'

# 1. SQL Server -> DuckDB (8 parallel streams, tables >= 20M rows split into key ranges)
duckbridge import work.duckdb --host sql01 --database Sales -j 8          # Windows auth by default; --user sa --password … for SQL logins

# 2. run your existing T-SQL on DuckDB
duckbridge run work.duckdb nightly_kpis.sql --from tsql --memory-limit 40GB --threads 8

# 3. ship the result back as a restorable .bak (only the tables that changed)
duckbridge export work.duckdb --to bak --tables customer_kpi \
    --host sql01 --user sa --database Sales_kpi --drop-existing \
    --backup-dir /var/opt/mssql/backup --fetch local --out s3://my-bucket/kpi/Sales_kpi.bak \
    --verify-restore --drop-staging

# other targets
duckbridge export work.duckdb --to script --out Sales.sql --create-database   # sqlcmd -i Sales.sql
duckbridge export work.duckdb --to server --host sql01 --database Sales_copy   # live load, no .bak
duckbridge export work.duckdb --to postgres --pg-dsn "host=pg user=postgres password=x dbname=sales"
duckbridge export work.duckdb --to parquet --out s3://lake/sales/

# translate a script without running it (prelude included), or target a newer SQL Server on the way back
duckbridge transpile job.sql --from tsql --to duckdb -o job.duckdb.sql
duckbridge transpile report.duckdb.sql --from duckdb --to tsql --target-version 2022

Connecting to SQL Server: Windows authentication by default

Connection flags (--host --port --user --password, env MSSQL_*) and the mssql:// URLs used by cp and the extension pick the login the same way:

duckbridge import work.duckdb --host sql01 --database Sales                      # Windows: your own identity
duckbridge import work.duckdb --host sql01 --database Sales --user 'CORP\etl' --password "$PW"   # Linux box, domain account
SELECT * FROM bridge_import('mssql://CORP%5Cetl:p%40ss@sql01/Sales');             # extension; \ is %5C

No system Kerberos libraries are needed: Windows uses the OS SSPI, other platforms use a pure-Rust NTLM implementation. --trust-cert (default on) accepts self-signed server certificates; TLS is always on unless --encrypt false.

Reading from a busy source

import protects the source server and the data it reads. Every import connection runs SET LOCK_TIMEOUT 1000; SET DEADLOCK_PRIORITY LOW; (DuckBridge loses deadlocks, production jobs win), and the source connection can only run reads. Before each table it checks for locks held by other sessions, and it compares a fingerprint (row count, last_user_update, modify_date) before and after the read; if the fingerprint changed, the table is read again.

The manifest has one row per table with the columns table, rows_read, rows_source_stats, fingerprint_before, fingerprint_after, started, finished, seconds, isolation, rereads. A skipped table has rows_read = -1. Downstream jobs can read it to confirm what they were given.

duckbridge import work.duckdb --host SRC --database SRCDB --tables t1,t2 \
    --isolation snapshot --lock-timeout 1000 --on-lock fail --manifest in.manifest.csv -j 8 \
    --wait-for "SELECT CAST(CASE WHEN EXISTS(SELECT 1 FROM ctl.flags WHERE step='inputs' AND done=1) THEN 1 ELSE 0 END AS bit)"

import also accepts --log <file> and --summary <file> (see Plan, log, summary).

Making DuckDB behave like SQL Server

run (and transpile --from tsql, unless --no-prelude) applies a session prelude before your T-SQL:

SET integer_division = true;                                   -- 7 / 2 = 3, like T-SQL
SET default_null_order = 'NULLS_FIRST_ON_ASC_LAST_ON_DESC';   -- SQL Server NULL ordering
SET TimeZone = 'UTC';
SET default_collation = 'nocase';                              -- SQL Server default collations are case-insensitive

Since 0.1.1 the default_collation line is only emitted with --nocase global; the default prelude (--nocase rewrite) is the first three statements, and case-insensitivity is handled per comparison (see Case-insensitive strings). T-SQL LIKE becomes DuckDB ILIKE (DuckDB LIKE is case-sensitive). Use --nocase off for a binary-collated source. When translating DuckDB SQL to T-SQL, a / b becomes CAST(a AS FLOAT) / b because DuckDB / is always floating point, and // becomes /.

Target version

--target-version 2019|2022|2025 (on transpile, run, export) controls which T-SQL is emitted and which types are used:

A .bak written by 2019 restores on 2022 and 2025 (the database keeps compat level 150 until you raise it); a newer version's backup never restores on an older one.

How the load is made fast

• Streaming everywhere. Rows flow DuckDB → TDS bulk-copy (or TDS → DuckDB appender) in batches; memory is bounded by batch_rows × parallel, not by table size. Nothing is written to temp files.
• Parallel streams. -j N tables at once (default: CPU count). A table above --split-threshold rows with an integer key is cut into key ranges so a single 1.5-billion-row table also uses every stream.
• Heap then key. By default the live load creates tables without the clustered primary key, bulk-copies into the heap with TABLOCK and KEEP_IDENTITY, then adds the clustered PK once at the end (a single parallel sort in SQL Server). Secondary indexes, unique/foreign keys and views are created after the data. --pk-inline keeps the old behaviour; --order-by-key adds ORDER BY key per range when the DuckDB data is already sorted.
• Identity preserved. Identity values are copied as-is and the seed is reset to the maximum afterwards. On import, SQL Server identities become DuckDB sequences that start after IDENT_CURRENT.
• Import of billions of rows. import --keys-after-load creates DuckDB tables without constraints, streams the data, then adds the primary key and unique indexes (DuckDB cannot add foreign keys afterwards, so those are skipped with a warning). Tables above --split-threshold rows with an integer key are read in key ranges on parallel connections.
• Staging database uses RECOVERY SIMPLE for minimal logging; the backup uses COMPRESSION and CHECKSUM, optionally striped (--stripes N).

The ceiling is SQL Server's own bulk-copy ingest speed; the tool runs that path in parallel, which is as fast as SQL Server goes.

Type mapping (DuckDB → SQL Server 2019)

Every lossy decision is reported per column by inspect and at the end of every export.

SQL Server → DuckDB covers every system type; hierarchyid/geography/geometry are converted with .ToString(), sql_variant and xml become VARCHAR, rowversion becomes an 8-byte BLOB, 2025 json becomes JSON and vector(n) becomes FLOAT[]. datetime2(7)/time(7) keep microseconds (digit 7 is dropped, warned) and datetimeoffset is stored as a UTC instant (offset not kept, warned). NVARCHAR sizes are measured in UTF-16 code units, so emoji and other supplementary characters are counted correctly.

SQL translation

transpile/run use a tokenizer and rule engine written for this tool. T-SQL → DuckDB covers TOP/OFFSET…FETCH, [brackets], N'…', table hints, ISNULL/IIF, GETDATE, DATEADD/DATEDIFF/DATEPART/DATENAME, CONVERT styles, CAST types, CHARINDEX, STUFF, FORMAT, EOMONTH, STRING_AGG … WITHIN GROUP, CROSS/OUTER APPLY, DECLARE/SET @var, table variables, #temp tables, SELECT … INTO, UPDATE … FROM JOIN, DELETE alias FROM JOIN, IF OBJECT_ID(…) DROP, IDENTITY → sequences, string + → ||, GO batches, and more. The reverse direction covers LIMIT/OFFSET, :: casts, ||, interval arithmetic → DATEADD, strftime → FORMAT, getvariable, temp tables, RETURNING → OUTPUT, and so on. Anything that cannot be translated faithfully (procedural IF/WHILE, cursors, regex on 2019, QUALIFY, nested types…) is left in place or commented out and reported as a warning, never silently changed.

Warnings and --strict

Constructs that DuckDB cannot reproduce exactly are reported with a code, line and column, for example W-DIV-DECIMAL (line 12:8): .... Free-text warnings from older rules carry W-GENERIC.

--strict (on transpile and run) turns any warning into an error: nothing is executed and the exit code is 1.

duckbridge run work.duckdb steps/nightly.sql --strict --nocase rewrite --mssql-math

Case-insensitive strings

--nocase rewrite|global|off (default rewrite):

To see whether you need any of this, run the audit. If a column has no values that differ only by case or trailing spaces, binary comparison gives the same result as SQL Server's CI_AS for that data:

duckbridge audit-ci work.duckdb --tables customers,orders --columns name,country --script nightly.sql

It lists, per column, the values that collide by case or trailing space, and every string literal in the script that matches a column value only case-insensitively (rewrite the literal). --json gives machine-readable output.

Exact SQL Server arithmetic

--mssql-math installs DuckDB macros that reproduce SQL Server results with integer/HUGEINT arithmetic instead of DOUBLE, and the translator uses them where it knows the column types:

Types come from CREATE TABLE / DECLARE in the same script, or from --types:

duckbridge transpile kpi.sql --mssql-math --types 'amount=decimal(18,4),qty=int' -o kpi.duckdb.sql
duckbridge verify-math --host sql01 --user sa --cases 20000      # random cases on both engines, reports mismatches

Loading into production tables safely

For a weekly batch into a shared SQL Server with fixed production DDL, use export --to server with the options below. The tool only touches objects it is allowed to, loads into the DDL you already have, checks the data before it moves, and can resume after a kill.

Safety guard

Database-level statements (CREATE/DROP DATABASE, ALTER DATABASE ... SET RECOVERY, SET COMPATIBILITY_LEVEL) are allowed only for databases whose name matches the disposable-database pattern: --disposable-db-pattern <regex>, env DUCKBRIDGE_DISPOSABLE_DB_PATTERN, default ^_ (names starting with an underscore).

Exit codes: 0 ok, 1 a check failed (types, lengths, PK, row counts, fingerprints), 2 could not run (connection, permissions), 3 stopped by the guard.

--into-existing

export --to server --into-existing treats the target table's DDL as the contract. No CREATE, ALTER or DROP touches the target table (its modify_date does not change). The tool reads the target's columns, indexes, partitioning and check constraints from the catalog and maps DuckDB columns to them by name, case-insensitively, in the target's column order.

• A DuckDB column with no target column is an error, unless --ignore-extra-columns.
• A target column with no DuckDB column is an error, unless it is nullable or has a default; it is then left out of the load.
• All checks run before any bulk session opens: one aggregate DuckDB query per table. Findings are reported together, with the longest value and sample keys. --check-only runs them and moves no data.

TYPE (incompatible pair, bad UUID or number text) and EXTRA_COLUMN / MISSING_COLUMN also appear.

duckbridge export work.duckdb --to server --host TGT --database _stage_db --into-existing \
    --check-only --check-pk hash

Splitting and strategies

A table is cut into jobs with --split <table>=<spec> (repeatable) and loaded with a strategy given by --strategy <table|*>=<strategy> (repeatable). All jobs of all tables share one queue, largest first, run by -j N bulk sessions.

On Standard Edition CREATE INDEX is single-threaded, so heap-then-index builds indexes one at a time; switch builds each partition's clustered index in its own session and is the only way to get parallel index builds there.

duckbridge export work.duckdb --to server --host TGT --database _stage_db --into-existing \
    --split big_table=column:Mkt --strategy big_table=switch --strategy '*'=serial -j 8 \
    --max-jobs-per-table 6 --state-dir state/ --replace-partitions --drop-old
duckbridge export work.duckdb --to server --host TGT --database _stage_db --into-existing \
    --source-parquet 'big_table=out/big_table/Mkt=*/' --strategy big_table=switch   # a Parquet source implies a files: split, one job per folder

Staging tables are dropped only if their name contains __stg_ or __old_; the target table is never dropped.

Verification

--verify counts|checksum|except (default counts, which always runs), --sample <n> (default 10).

After heap-then-index or switch, every index captured before the load must exist with the same definition (type, key columns and order, INCLUDE, filter, compression, fill factor, data space), and the PK must not be disabled.

Plan, log, summary

• --plan prints every statement that would be sent to each server and the list of bulk jobs with estimated rows, and executes nothing. It connects only for metadata reads.
• --log <file> appends one line per statement and event; a failed log write never stops the run.
• --summary <file> writes JSON: per table and job, rows, seconds, strategy and checks.
• --progress-seconds <n> (default 10) sets the progress line interval (rows/s per job and in total).

import accepts --log and --summary too.

Cut-over

duckbridge swap --host TGT --database PRODDB --pairs kpi_new=kpi,dim_new=dbo.dim --lock-timeout 5000
duckbridge drop-old --host TGT --database PRODDB --before 2026-11-01

swap renames in one transaction with SET LOCK_TIMEOUT (default 5000 ms): each live table becomes <live>__old_<yyyymmddhhmm> (UTC) and each new table takes the live name. If a lock cannot be obtained, everything rolls back and the blocking sessions are listed; nobody is killed. It refuses, before changing anything, when a schema-bound view or function references the tables, when other tables have foreign keys to them, or when a name collides. The old tables are kept (--keep-old is on by default; --keep-old false drops them after the commit). drop-old --before removes __old_ tables whose timestamp is earlier than the date (YYYY-MM-DD or "YYYY-MM-DD HH:MM", UTC); --schema limits it.

String types in create mode

Without --into-existing, DuckDB VARCHAR columns are created as NVARCHAR by default.

inspect takes the same two flags, so you can preview the DDL. Explicit varchar with non-ASCII data can lose characters in the target code page; use auto, or --into-existing, whose CODEPAGE check stops instead.

End to end

# 1. read the inputs safely
duckbridge import work.duckdb --host SRC --database SRCDB --tables t1,t2 \
    --isolation snapshot --lock-timeout 1000 --lock-check true --on-lock fail \
    --manifest in.manifest.csv -j 8

# 2. process; stop before running anything if a construct cannot be translated exactly
duckbridge run work.duckdb steps/nightly.sql --strict --nocase rewrite --mssql-math

# 3. pre-flight against the existing target DDL (no data moves)
duckbridge export work.duckdb --to server --host TGT --database _stage_db --into-existing \
    --check-only --check-pk hash

# 4. load and verify
duckbridge export work.duckdb --to server --host TGT --database _stage_db --into-existing \
    --split big_table=column:Mkt --strategy big_table=switch --strategy '*'=serial -j 8 \
    --state-dir state/ --verify checksum --log load.log --summary load.json

# 5. cut over (optional)
duckbridge swap --host TGT --database PRODDB --pairs kpi_new=kpi

Roadmap

DuckBridge is organised so that every engine is a pair: a target module (catalog reader, bulk loader, backup/restore) and a dialect rule set for the translator. SQL Server and PostgreSQL are done; the plan for the rest, in priority order:

DuckDB extension

The same engine ships as a loadable DuckDB extension (extension/), so everything above can be driven from inside DuckDB itself, with no CLI process:

Install: download duckbridge-<platform>.duckdb_extension.zip for your platform from the Releases page (linux_amd64, linux_arm64, osx_amd64, osx_arm64, windows_amd64), unzip it, and keep the file name duckbridge.duckdb_extension: DuckDB derives the extension's entry point from the file name, so a renamed file will not load. The build is pinned to DuckDB 1.5.6 and is not yet signed:

unzip duckbridge-linux_amd64.duckdb_extension.zip        # -> duckbridge.duckdb_extension
duckdb -unsigned work.duckdb                              # or SET allow_unsigned_extensions = true

LOAD '/path/to/duckbridge.duckdb_extension';

-- SQL Server -> this DuckDB database (parallel key-range streams, identity -> sequence)
SELECT * FROM bridge_import('mssql://sa:p%40ss@sql01:1433/Sales', parallel := 8, keys_after_load := true);

-- run the team's T-SQL against DuckDB; the SQL Server compatibility prelude is applied to
-- the extension's own session only, your session's settings are untouched
SELECT * FROM bridge_run('SELECT c.id, SUM(o.amount) AS total INTO dbo.kpi FROM dbo.customers c
                          JOIN dbo.orders o ON o.customer_id = c.id GROUP BY c.id;');

-- ship it back: live load into a heap, clustered PK after, then a verified, restorable .bak
SELECT * FROM bridge_export('mssql://sa:p%40ss@sql01:1433/Sales_kpi', tables := 'kpi',
                            bak := 's3://backups/Sales_kpi.bak', fetch_via := 'local', verify_restore := true);

SELECT * FROM bridge_export('postgres://u:p@pg/analytics', tables := 'kpi');
SELECT * FROM bridge_export('parquet:s3://lake/sales/', parquet_compression := 'zstd');
SELECT * FROM bridge_export('script:/tmp/Sales.sql', database := 'Sales', create_database := true);

-- text in, text out
SELECT bridge_transpile('tsql', 'duckdb', 'SELECT TOP 5 ISNULL(a, 0) FROM dbo.t');
SELECT bridge_transpile('duckdb', 'tsql', 'SELECT greatest(a, b) FROM t', 2022);   -- target version
SELECT bridge_prelude(true);                                                       -- the SET statements

Every table function returns (object, kind, rows, note) rows: one per table/view, a summary, and any warnings. Named parameters mirror the CLI flags (tables, exclude, schema_map, parallel, split_threshold, batch_rows, drop_existing, schema_only, overwrite, target_version, string_mode, method, pk_inline, order_by_key, bak, backup_dir, fetch_via, stripes, compression, verify_restore, drop_staging, backup_to_url, create_database, parquet_compression; bridge_run takes from, nocase, schema_map, continue_on_error). The 0.1.1 parameters:

split, strategy and source_parquet take the CLI form, for example split := 'big_table=column:Mkt'; several entries are comma-separated. bridge_prelude(true) now returns the 3-line rewrite-mode prelude (no default_collation).

SELECT * FROM bridge_export('mssql://sa:p%40ss@tgt/_stage_db', into_existing := true, check_only := true, check_pk := 'hash');
SELECT * FROM bridge_export('mssql://sa:p%40ss@tgt/_stage_db', into_existing := true,
                            split := 'big_table=column:Mkt', strategy := 'big_table=switch', state_dir := 'state/', verify := 'checksum');

URL-encode special characters in passwords (! → %21, # → %23, @ → %40).

Build and test:

cd extension && make release        # -> build/duckbridge.duckdb_extension
make test                           # scripts/e2e_extension.sh against the local SQL Server container

How it works: the extension opens a pool of 64 connections to the host database when loaded and runs every operation on those, so parallel workers get their own connections and nothing touches your transaction. Two consequences worth knowing: parallel is capped at 32 (each bulk-load worker uses two pooled connections), and the database stays open until the process exits, so a .wal file may remain next to the database after a session that used the bridge (DuckDB replays it on the next open; the bridge checkpoints after each operation to keep it small). The extension is built on DuckDB's C extension API via duckdb-rs, which currently requires the unstable struct, so each binary is pinned to one DuckDB release (v1.5.6 today, see extension/Makefile). CI builds Linux x86_64/arm64, macOS Intel/ARM and Windows. The community-extension submission (duckdb/community-extensions#3147, built from duckbridge-extension) will make it INSTALL duckbridge FROM community; until it is merged, load the file with duckdb -unsigned or SET allow_unsigned_extensions = true.

Build

Prebuilt binaries for Linux, macOS and Windows are on the Releases page. With a Rust toolchain, cargo install duckbridge builds the CLI from crates.io (the bundled DuckDB takes a few minutes to compile). From source:

cargo build --release            # ~3 min; DuckDB is compiled in
cargo test --release             # translator unit tests
scripts/e2e_mssql.sh             # full round trip against a SQL Server (see env vars in the script)

Cross-platform binaries are produced by .github/workflows/release.yml (Linux x86_64/aarch64, macOS Intel/ARM, Windows).

Notes and limits

• A .bak can only be written by SQL Server itself, so --to bak needs a SQL Server to stage into (the 2019 target, a Docker container, or the local 2025 instance). A 2019 .bak restores on 2019, 2022 and 2025; never the other way round.
• SQL Server 2019 cannot back up directly to S3 (2022+ can); the tool backs up locally and uploads.
• --fetch docker:<container> copies the file out of a container; --fetch local when the server runs on this machine; --fetch none leaves it on the server and prints the path.
• PostgreSQL loads use COPY … FROM STDIN without TLS (local/VPC use); Parquet is read by PostgreSQL via extensions, by SQL Server 2022+ via OPENROWSET, not by 2019.