sqlite-diff-rs 0.10.0

Build SQLite changeset and patchset binary formats programmatically, without SQLite
Documentation
# Changelog

## 0.10.0

### Fixed

Patchset UPDATE records now write every column in column order. The encoder previously wrote all primary key columns first and the remaining columns after, a layout that agrees with SQLite only when the primary key is the first column, so a table whose key sits anywhere else produced records SQLite reads with the values landing on the wrong columns. A table header now writes its column count as a varint instead of a raw byte, which panicked outright at 256 columns and silently emitted a malformed header between 128 and 255. Both defects are now verified byte for byte against SQLite's own session extension in both directions, the encoder against SQLite's bytes and the parser against them, so a pair of cancelling errors cannot hide. Anything 0.9.0 wrote for an affected table was already malformed and cannot be read back.

The base64 decoder rejects malformed input it used to accept: an effective length one above a multiple of four, an all-padding payload, and a trailing character carrying bits that no decoded byte can hold (RFC 4648 section 3.5).

### Changed

The Maxwell and wal2json decoders now come from the extracted `maxwell-cdc` and `wal2json-events` crates, which moves four public shapes. `maxwell::Message` is `#[non_exhaustive]`, so an exhaustive match no longer compiles. `maxwell::RowChange::data` is a `serde_json::Map` rather than a `BTreeMap`. `wal2json::MessageV2` is an enum rather than a struct. `wal2json::Column` is `#[non_exhaustive]`, so a struct literal no longer compiles.

`parser::ParseError`, `errors::Error` and `builders::sql::ParseError` are now `#[non_exhaustive]`, matching their `DecodeError` and `ConversionError` siblings, so a match on any of them needs a wildcard arm.

`digest_sql` rejects three inputs it used to accept quietly: an INSERT whose column list is shorter than its VALUES list (the missing columns became NULL), a WHERE naming only part of a composite primary key (the result was an operation that could never match a row), and an OR in a WHERE (one branch was kept and the rest discarded). A failure anywhere in a multi-statement input now leaves the builder untouched rather than applying the statements that happened to parse first.

The minimum supported Rust version is 1.87, which is what the `pg-walstream` feature has required since 0.5.0. The manifest claimed 1.85.

`testing::compare_db_states` is removed. It had no caller and mis-parsed any table name containing a space.

## 0.9.0

### Added

A new `PgBinary` wire source decodes PostgreSQL binary result fields (the form diesel's Postgres backend hands back) straight into `Value` through the same `WireType` and `Decoder` machinery the CDC sources use, so a query result and a CDC event reach a byte-identical representation by construction. The payload `PgBinaryColumn` carries the caller's catalog `WireType` explicitly (binary `int4` and `float4` are both four opaque bytes), a `None` `raw` short-circuits to `Value::Null`, and `TypeMap::<PgBinary, _, _>::defaults()` covers the v1 set `bool`, `int`, `real`, `text`, `bytea`, and `uuid` (as a 16-byte blob). The remaining types stay `NotYetImplemented` until their numeric binary layouts render back to text byte-identically to the verbatim decoders. A cross-source parity test guards that `PgBinary` never diverges from `PgWalstream`.

### Changed

The three CDC wire sources now share one internal decoding core instead of carrying near-identical copies. `wal2json::ConversionError`, `maxwell::ConversionError`, and `pg_walstream::ConversionError` are now a single re-exported `#[non_exhaustive]` type with the union of the previous variants, so matching on it must account for the shared set. The per-source `build_insert`/`build_patchset_update`/`build_changeset_delete`/`build_patch_delete` helpers, the float and PostgreSQL-binary scalar decoders, and the JSON-source (`Maxwell`, `Wal2Json`) `text`/`decimal`/`uuid`/`json` decoders are now shared, and the duplicated `pk_indices` computation in the SQL and diesel renderers now calls the existing `SchemaWithPK::primary_key_columns`. No wire-format or decoded-value behavior changes.

## 0.8.0

### Added

`ParsedDiffSet::rename_tables` renames table sections in place. A callback maps each section name to a new one (or `None` to keep it), and the count of renamed sections is returned. Only the name changes, so a diffset captured against one schema can apply where the same tables carry different physical names (for example an RLS translation that renames the storage table), which `sqlite3changeset_apply` cannot remap on its own.

## 0.6.1

### Fixed

Changeset UPDATE digests now always capture the old primary key, so a changeset produced from a wire event applies through the diesel changeset path even when the update does not change the key. The wal2json digest previously dropped the old-row image entirely (it now reads `identity` in the v2 format and `oldkeys` in the v1 format). The Maxwell digest treats a column absent from `old` as unchanged, taking its old value from the new value, since Maxwell lists only changed columns in `old`. The pgoutput digest keeps a primary-key column's old value from the new tuple when `old_data` is absent under `REPLICA IDENTITY DEFAULT`. The patchset builders are unchanged, since they use only the new image.

## 0.6.0

### Added

The `diesel` feature now renders `ChangeSet` operations as backend-generic Diesel queries, alongside the existing `PatchSet` support. `ChangesetOp` implements `QueryFragment`, `QueryId`, and `RunQueryDsl`, and `ChangesetOp::with_adapter` builds a `BoundChangesetOp` just as `PatchsetOp::with_adapter` builds a `BoundPatchsetOp`. Both bound types are aliases of the new generic `BoundOp`, and `ApplyOps` drives either one. Because a changeset carries the old and new value of every column, it renders primary-key changes (including composite keys) as `UPDATE ... SET <changed columns> WHERE <old key>`, which a patchset cannot represent. A changeset `UPDATE` writes only the columns whose value actually changed, so applying it never rewrites (or spuriously triggers on) a column that did not move.

## 0.5.1

### Fixed

The wal2json `bytea` decoder (`PgByteaTextModeDecoder`) now accepts wal2json's bare lowercase hex form (for example `0001deadff`) in addition to the Postgres-style `\x`-prefixed hex form, so `bytea` columns round-trip on the wal2json vehicle. The pgoutput and Maxwell paths are unchanged.

## 0.5.0

The optional `pg_walstream` dependency moves from 0.7 to 0.8.

### Breaking

The `pg_walstream` and `pg_walstream_reverse` module re-exports (`EventType`, `RowData`, `ColumnValue`, `ChangeEvent`, `ReplicaIdentity`, `ColumnData`, `ColumnInfo`, `LogicalReplicationMessage`, `TupleData`, `Oid`, and the rest) now resolve to `pg_walstream` 0.8 types. Consumers of the `pg-walstream` feature that also depend on `pg_walstream` directly must move to 0.8. Enabling the feature raises the minimum supported Rust version to 1.87, which `pg_walstream` 0.8 requires.

## 0.4.0

Source-independent semantic type key for `digest`. A catalog carrying semantic column types now drives `DiffSetBuilder::digest` for every wire source without translating to a source-native key.

### Added

`WireType` enum (`Bool`, `Int`, `Real`, `Text`, `Bytes`, `Uuid`, `Decimal`, `Timestamp`, `TimestampTz`, `Date`, `Time`, `Interval`, `Json`, `Jsonb`), re-exported from the crate root. It is the single decoder-dispatch key shared by `pg_walstream`, `wal2json`, and `maxwell`.

`TypeMap::defaults()` now registers a `WireType::Uuid` decoder for every source (`UuidText36Decoder`), which fixes the previously missing wal2json `uuid` mapping.

### Breaking

`WireSource` drops the associated `TypeKey` and its `type_key` method in favor of `fn wire_type(payload) -> WireType`.

`WireColumnTypes` and `WireSchema` are no longer generic over the source. `WireColumnTypes::column_type_key(idx) -> Src::TypeKey` becomes `WireColumnTypes::column_type(idx) -> WireType`, and `WireSchema<Src>` becomes `WireSchema`. `Digestable` and `DiffSetBuilder::digest` drop `Src` from their schema and column-type bounds.

`TypeMap` is keyed by `WireType` instead of `Src::TypeKey`, so `register` and `with` take a `WireType`.

The per-column payload structs replace their native type field with `wire_type: WireType`: `PgWalstreamColumn` drops `oid` and `type_modifier`, `Wal2JsonColumn` drops `pg_type_name`, and `MaxwellColumn` drops `mysql_type`. Binary integer and float widths on `pg_walstream` are now inferred from the payload byte length. The wal2json paren-stripping and maxwell `tinyint(1)` type-name normalizations are gone because the schema declares the semantic type directly.

Downstreams migrate by implementing the source-independent `WireColumnTypes` and `WireSchema` and registering decoders under `WireType`.

## 0.3.0

`wal2json::MessageV2` now carries the optional wal2json LSN.

### Added

`lsn: Option<String>` field on `wal2json::MessageV2`, populated from the message's `lsn` when wal2json runs with `include-lsn=true` and `None` otherwise. The value stays a raw `hi/lo` hex string (for example `0/16B2270`) so the module carries no Postgres-specific numeric LSN type, and the consumer decides how to parse it. The `Digestable` wire output is byte-identical whether or not the field is present.

### Breaking

Struct-literal construction of `MessageV2` must now supply `lsn` (for example `lsn: None`). Deserialization from wal2json JSON is unaffected because the field is `#[serde(default)]`.

## 0.2.0

Schema-aware forward conversion for CDC wire formats. Downstream users register a type-to-decoder mapping once and consume `pg_walstream`, `wal2json`, or `maxwell` interchangeably.

### Removed

`debezium` module and feature. Users on Postgres have `pg_walstream` and `wal2json`. Users on MySQL have `maxwell`.

The 0.1.x `TryFrom<..>` impls on `Insert`, `Update`, `ChangeDelete`, and `PatchDelete` for pg_walstream, wal2json, and maxwell events. Use `builder.digest(&event, &schema, &adapter)` instead.

`SnifferDecoder` and `SnifferAdapter` (migration bridge for the deleted `TryFrom` impls).

Legacy `digest_pg_walstream`, `digest_wal2json_v2`, `digest_wal2json_v1_change`, and `digest_maxwell` methods on `DiffSetBuilder`.

Per-source `maxwell`, `pg_walstream`, and `wal2json` Criterion benches. They benched the deleted `TryFrom` path.

### Added

`sqlite_diff_rs::wire` module with:

- `WireSource` sealed trait implemented by `PgWalstream`, `Wal2Json`, `Maxwell`. Each carries an associated `Payload` struct and `TypeKey`.
- `Decoder<Src, S, B>` trait. Zero-sized unit types for the built-ins, state-carrying structs for custom user decoders.
- `WireAdapter<Src, S, B>` trait. Single-method dispatcher. `TypeMap` is the primary implementer.
- `TypeMap<Src, S, B>` generic hashmap-backed registry keyed by `Src::TypeKey`.
- `TypeMapDefaults<S, B>` companion trait giving `TypeMap::<Src>::defaults()` a pre-populated registry.
- `DecodeError` shared enum wrapped by each format's existing `ConversionError` via a new `Decode(_)` variant.

Built-in decoders shipping in 0.2.0:

- `BoolDecoder`
- `IntDecoder`, `Int64OverflowToTextDecoder`
- `RealDecoder` (NaN normalizes to `Null`, `-0.0` to `0.0`)
- `TextDecoder` (strict UTF-8)
- `PgByteaBinaryDecoder`, `PgByteaTextModeDecoder`, `MySqlBinaryDecoder` (vendored base64 and PG `\xHEX` decoders, no external deps)
- `UuidBlob16Decoder`, `UuidText36Decoder` (vendored parser, no `uuid` dep)
- `DecimalTextDecoder`
- `TimestampVerbatimDecoder`, `TimestampTzVerbatimDecoder`, `DateVerbatimDecoder`, `TimeVerbatimDecoder`, `IntervalVerbatimDecoder`
- `JsonVerbatimDecoder`, `JsonCanonicalDecoder` (canonical does recursive key sort)
- `NullDecoder`

New traits `WireColumnTypes<Src>` (schema-side per-column type key), `WireSchema<Src>` (table-name lookup), and `Digestable<F, T, S, B>` (event dispatch, implemented in-crate for `pg_walstream::EventType`, `wal2json::MessageV2`, `wal2json::ChangeV1`, `maxwell::Message` times both formats).

One unified digest entry point: `builder.digest(&event, &schema, &adapter)`. Replaces the 0.1.x `digest_pg_walstream` / `digest_wal2json_v2` / `digest_wal2json_v1_change` / `digest_maxwell` methods. `RelationInfo` is no longer a digest argument (OIDs come from the schema).

`ConversionError::TableNotFound(String)` added to each format's error type.

`IndexableValues` promoted from `pub(crate)` to `pub` so external code can implement `SchemaWithPK`.


### MSRV

Rust 1.85 (edition 2024). Unchanged from 0.1.x.

## 0.1.4 and earlier

See git history.