Expand description
Change Data Capture: durable record of every committed write, ordered, addressable, and consumable by external systems. The transaction layer hands committed deltas here at commit time; this crate persists them, exposes the consumer-side cursor APIs that downstream replication, subscriptions, and external sinks read from, and runs the background compactor that prunes records past retention.
Producers and consumers are decoupled - a write only needs to be persisted before the transaction returns; the consumer side can fall arbitrarily far behind and catch up later. Storage is pluggable so the same protocol can be backed by SQLite for embedded deployments and by a horizontally scaled log for production.
Invariant: a CDC record published for a transaction reflects exactly the deltas that were committed under that transaction id, in the order the engine produced them. Reordering or dropping deltas inside CDC desynchronises replicas and subscriptions from the source of truth.
Modules§
- compact
- consume
- Consumer side of the CDC stream. A consumer registers with the host actor, polls for new records past its checkpoint, and advances a watermark so the producer side knows what is safe to compact. Each subscriber holds its own checkpoint independently; a slow consumer never blocks a fast one.
- error
- produce
- Producer side of the CDC stream. The transaction layer hands committed deltas here at commit time; the producer encodes them into CDC records, persists them to storage, and advances the publisher watermark so consumers can observe the new commit boundary. Decoding lives next to producing so the round-trip is symmetric.
- storage
- Pluggable backing store for the CDC log. The in-memory implementation is the testing default; SQLite is the durable default for production deployments. Both implement the same trait surface so the producer and consumer sides are agnostic to which is configured.
- testing