Crate sqlite_diff_rs 

sqlite-diff-rs

A Rust library for building SQLite changeset and patchset binary formats programmatically.

Overview

SQLite’s session extension defines a binary format for tracking and applying database changes. This crate constructs that binary data without linking SQLite, which is useful for offline sync (build a changeset on a server and apply it on a SQLite client), CDC pipelines (produce the input expected by sqlite3_changeset_apply() from your own change events), cross-database sync (convert PostgreSQL change streams from wal2json or Maxwell into the SQLite format), and generating test fixtures for changeset processing code.

This crate is not the SQLite sqldiff tool, which compares two existing database files. This library constructs the changeset and patchset binary format programmatically from your own change data.

Installation

Add to your Cargo.toml:

[dependencies]
sqlite-diff-rs = "0.2"

Quick Start

use sqlite_diff_rs::{DiffOps, Insert, PatchSet, SimpleTable};

// Define a table schema: "users" with columns (id, name), PK at index 0
let users = SimpleTable::new("users", &["id", "name"], &[0]);

// Build a patchset with an INSERT
let patchset = PatchSet::<_, String, Vec<u8>>::new()
    .insert(
        Insert::from(users)
            .set(0, 1i64).unwrap()      // id = 1
            .set(1, "Alice").unwrap()   // name = "Alice"
    );

// Encode to binary format
let bytes: Vec<u8> = patchset.into();

// Apply with sqlite3_changeset_apply() or sqlite3session_patchset_apply()

Schema-aware CDC ingest (0.2.0+)

builder.digest(&event, &schema, &adapter) folds one wire event (from pg_walstream, wal2json, or maxwell) into a builder. Same call site for every source.

let patchset = PatchSet::<UsersTable, String, Vec<u8>>::new()
    .digest(&msg, &schema, &types)?;
let bytes: Vec<u8> = patchset.build();

The schema type implements WireSchema<Src> and its tables implement WireColumnTypes<Src>. TypeMap::defaults() ships with mappings for bool, integers, reals, text, bytea, decimals, temporals, and JSON.

Features

Feature	Description
testing	Enables rusqlite integration for differential testing
wal2json	Parse PostgreSQL wal2json output into changesets
pg-walstream	Integration with pg_walstream crate
maxwell	Parse Maxwell CDC JSON events
diesel	Execute patchsets as backend-generic Diesel queries via a downstream [Adapter]

Enable features in Cargo.toml:

[dependencies]
sqlite-diff-rs = { version = "0.2", features = ["wal2json"] }

Binary Format Reference

Both formats share the same container structure: one or more table sections, each with a table header followed by change records. The key difference is how much old-row data each operation carries.

Changesets ('T' / 0x54) store the complete old state of every column. This makes them reversible, so INSERTs can be turned into DELETEs and vice versa.

Patchsets ('P' / 0x50) omit old values for non-PK columns, producing a smaller, forward-only encoding that cannot be reversed.

Aspect	Changeset ('T')	Patchset ('P')
INSERT	All column values	All column values
DELETE	All old column values	PK values only
UPDATE old	All old column values	PK values + Undefined for non-PK
UPDATE new	All new column values	All new column values
Reversible	Yes	No
Wire size	Larger (carries full old state)	Smaller (omits non-PK old values)

See the SQLite session extension docs for the full specification.

Apply patchsets with Diesel

The diesel feature turns each PatchsetOp into a backend-generic Diesel query. Downstream implements one Adapter per schema (the set of tables), and it maps (table_name, column_index) pairs to column identifiers and to per-column Binders. Each Binder calls push_bind_param with the target SqlType the column expects, so values travel as native binary binds and the emitted SQL contains no CAST wrappers regardless of backend. The ApplyOps extension trait wraps the batch-execute and conn.transaction shapes so a full apply reads as one call.

use diesel::pg::Pg;
use diesel::prelude::*;
use diesel::query_builder::AstPass;
use diesel::result::QueryResult;
use diesel::sql_types::Bool;
use sqlite_diff_rs::{
    Adapter, ApplyOps, Binder, DefaultBinder, DiffOps, Insert, PatchSet, SimpleTable, Value,
};

// A binder for target BOOLEAN columns. Native SQL type, no CAST.
struct BoolBinder(bool);
impl Binder<Pg> for BoolBinder {
    fn walk<'b>(&'b self, out: &mut AstPass<'_, 'b, Pg>) -> QueryResult<()> {
        out.push_bind_param::<Bool, bool>(&self.0)
    }
}

// One adapter per schema, dispatching on (table_name, column_index).
struct MyAdapter;
impl<S: AsRef<str> + Sync, B: AsRef<[u8]> + Sync> Adapter<Pg, S, B> for MyAdapter {
    fn column_name(&self, _table: &str, index: usize) -> &str {
        ["id", "active"][index]
    }
    fn bind<'a>(
        &self,
        table: &str,
        column_index: usize,
        value: &'a Value<S, B>,
    ) -> diesel::result::QueryResult<Box<dyn Binder<Pg> + Send + 'a>> {
        match (table, column_index, value) {
            ("users", 1, Value::Integer(v)) => Ok(Box::new(BoolBinder(*v != 0))),
            _ => Ok(Box::new(DefaultBinder::from(value))),
        }
    }
}

let schema = SimpleTable::new("users", &["id", "active"], &[0]);
let patchset = PatchSet::<SimpleTable, String, Vec<u8>>::new().insert(
    Insert::from(schema.clone())
        .set(0, 1_i64)
        .unwrap()
        .set(1, 1_i64)
        .unwrap(),
);

let mut conn = PgConnection::establish("postgres://...")?;
patchset
    .iter()
    .map(|op| op.with_adapter::<Pg, _>(&MyAdapter))
    .apply_transactional(&mut conn)?;

Enable via Cargo.toml:

[dependencies]
sqlite-diff-rs = { version = "0.1", features = ["diesel"] }
diesel = { version = "2", features = ["postgres"] }

End-to-end tests against real SQLite, Postgres, and MySQL containers live under integration-tests/diesel-e2e/. The unit test file tests/diesel_patchset.rs has the fully runnable version of the example above.

no_std Support

This crate is no_std compatible (requires alloc). It can be used in embedded or WebAssembly environments.

License

MIT License, see LICENSE for details.

Re-exports

pub use builders::ChangeDelete;

pub use builders::ChangeSet;

pub use builders::ChangesetFormat;

pub use builders::ChangesetOp;

pub use builders::ChangesetUpdatePair;

pub use builders::ColumnNames;

pub use builders::DiffOps;

pub use builders::DiffSet;

pub use builders::DiffSetBuilder;

pub use builders::Indirect;

pub use builders::Insert;

pub use builders::PatchDelete;

pub use builders::PatchSet;

pub use builders::PatchsetFormat;

pub use builders::PatchsetOp;

pub use builders::PatchsetUpdateEntry;

pub use builders::Reverse;

pub use builders::Update;

pub use parser::FormatMarker;

pub use parser::ParseError;

pub use parser::ParsedDiffSet;

pub use parser::TableSchema;

pub use schema::DynTable;

pub use schema::IndexableValues;

pub use schema::NamedColumns;

pub use schema::SchemaWithPK;

pub use schema::SimpleTable;

pub use wire::BoolDecoder;

pub use wire::DateVerbatimDecoder;

pub use wire::DecimalTextDecoder;

pub use wire::DecodeError;

pub use wire::Decoder;

pub use wire::Digestable;

pub use wire::Int64OverflowToTextDecoder;

pub use wire::IntDecoder;

pub use wire::IntervalVerbatimDecoder;

pub use wire::JsonCanonicalDecoder;

pub use wire::JsonVerbatimDecoder;

pub use wire::MySqlBinaryDecoder;

pub use wire::NullDecoder;

pub use wire::PgByteaBinaryDecoder;

pub use wire::PgByteaTextModeDecoder;

pub use wire::RealDecoder;

pub use wire::TextDecoder;

pub use wire::TimeVerbatimDecoder;

pub use wire::TimestampTzVerbatimDecoder;

pub use wire::TimestampVerbatimDecoder;

pub use wire::TypeMap;

pub use wire::TypeMapDefaults;

pub use wire::UuidBlob16Decoder;

pub use wire::UuidText36Decoder;

pub use wire::WireAdapter;

pub use wire::WireColumnTypes;

pub use wire::WireSchema;

pub use wire::WireSource;

pub use errors::Error;

Modules

builders

Builder for constructing changesets and patchsets.

errors

Submodule defining the errors used across the crate.

parser

Parser for SQLite changeset/patchset binary format.

schema

Schema traits for compile-time and runtime table definitions.

wire

Schema-aware forward conversion from CDC wire formats into Value.

Enums

Value

A value that can be encoded in SQLite changeset format.

Type Aliases

ChangeUpdate

Type alias for Update<T, ChangesetFormat, S, B>.

PatchUpdate

Type alias for Update<T, PatchsetFormat, S, B>.