datum-cdc

datum-cdc exposes PostgreSQL logical replication as Datum streams.

The MVP supports PostgreSQL pgoutput protocol v1 with streaming transactions off, two-phase commit off, and text-mode tuple values. A CdcSource builder materializes a single-owner Tokio replication carrier that feeds a bounded Datum Source<ChangeEvent, CdcHandle>.

use datum_cdc::{CdcSource, PostgresCdcConfig, SlotLifecycle};

# fn build() -> datum_cdc::CdcResult<()> {
let source = CdcSource::postgres()
    .connect(PostgresCdcConfig::from_url(
        "postgresql://datum_cdc@127.0.0.1:55433/datum_cdc",
    )?)
    .slot("datum_orders")
    .publication("orders_pub")
    .slot_lifecycle(SlotLifecycle::Existing)
    .buffer_capacity(8192)
    .build()?;
# let _ = source;
# Ok(())
# }

Delivery And Feedback

The source is at-least-once. PostgreSQL standby feedback advances only after downstream code calls CdcCheckpointHandle::checkpoint for every event in the next contiguous committed transaction. If a process crashes after delivery but before checkpoint feedback, PostgreSQL may replay those rows. Downstream can deduplicate with (slot, tx_end_lsn, xid, event_index).

build_with_context() returns a SourceWithContext<ChangeEvent, CdcOffset, _> so the offset stays attached through context-preserving operators.

WAL Retention

Backpressure is the bounded Datum channel. When downstream stalls, the carrier stops reading ahead and PostgreSQL retains WAL until feedback advances. Operators should monitor CdcHandle::lag() and server-side pg_replication_slots (restart_lsn, confirmed_flush_lsn, wal_status, safe_wal_size where available). Use max_slot_wal_keep_size on PostgreSQL to bound blast radius; if PostgreSQL marks a slot lost, create a new slot after a fresh snapshot.

Slot Lifecycle

SlotLifecycle::Existing is the production default. CreateIfMissing validates or creates a pgoutput slot but never drops it. CreateOwned and Temporary may be dropped through CdcHandle::drop_slot; all other modes require force_drop_slot.

Deferred Scope

Initial snapshots, pgoutput protocol v2-v4 streaming/parallel/two-phase messages, binary decoding, schema-DDL capture, multi-slot merge ordering, and a provider-compatible rustls replication build are intentionally deferred. The current workspace disables pgwire-replication's TLS feature because it forces rustls/ring while Datum's network stack uses rustls/aws-lc-rs under the workspace --all-features gate.