rhei_cdc_rocksdb/
lib.rs

1//! RocksDB-backed CDC event log for the Rhei HTAP engine.
2//!
3//! This crate provides [`RocksDbCdcLog`], a durable, high-throughput alternative
4//! to the trigger-based SQLite CDC log used in standard HTAP mode. Benchmarks show
5//! **5–7× higher write throughput** compared to SQLite triggers, because RocksDB's
6//! LSM-tree write path has lower per-write overhead than SQLite's WAL trigger
7//! execution.
8//!
9//! # When to use this crate
10//!
11//! | Scenario | Recommended CDC backend |
12//! |---|---|
13//! | Development / low-volume workloads | SQLite trigger-based CDC (default) |
14//! | High-throughput OLTP with many small writes | `RocksDbCdcLog` (this crate) |
15//! | Sidecar mode (external source, no local OLTP) | N/A — sidecar polls the external DB directly |
16//!
17//! This crate is compiled only when the parent workspace enables the
18//! `rocksdb-cdc` feature. The RocksDB C++ library is bundled via
19//! `rust-rocksdb` so no external installation is required.
20//!
21//! # Storage layout and key encoding
22//!
23//! All data lives in RocksDB's **default column family** (no custom CFs).
24//! Two key namespaces co-exist:
25//!
26//! | Key range | Content |
27//! |---|---|
28//! | `[0x00, 0xFF)` (8-byte big-endian `i64`) | CDC event payloads serialised as JSON |
29//! | `\xff__meta__/*` | Metadata: latest sequence number, bridge watermark |
30//!
31//! Big-endian encoding is chosen so that RocksDB's default byte-wise comparator
32//! iterates events in sequence order without a custom comparator. Metadata keys
33//! start with `0xFF`, which sorts *after* all valid sequence keys (which are
34//! non-negative `i64` values encoded big-endian and therefore never start with
35//! `0xFF`), so a forward iterator stops naturally before reaching metadata.
36//!
37//! # Durability and crash recovery
38//!
39//! Every write goes through a [`WriteBatch`](rust_rocksdb::WriteBatch), which
40//! RocksDB commits atomically. On crash recovery, [`RocksDbCdcLog::open`]
41//! reads the persisted `META_LATEST_SEQ` key to restore the in-memory sequence
42//! counter, so appends resume from the correct position without scanning all
43//! events.
44//!
45//! The bridge watermark ([`RocksDbCdcLog::append_and_set_bridge_watermark`])
46//! is written in the **same `WriteBatch`** as the events it covers, making the
47//! bridge operation idempotent: after a mid-write crash, either both the events
48//! and the watermark are present, or neither is.
49//!
50//! # Quick start
51//!
52//! ```rust,no_run
53//! use rhei_cdc_rocksdb::{RocksDbCdcLog, RocksDbCdcConfig};
54//!
55//! let config = RocksDbCdcConfig {
56//!     path: "/var/lib/rhei/cdc_rocksdb".to_string(),
57//!     create_if_missing: true,
58//! };
59//! let log = RocksDbCdcLog::open(&config).unwrap();
60//! ```
61
62/// Configuration for opening a [`RocksDbCdcLog`].
63pub mod config;
64
65/// Error types returned by [`RocksDbCdcLog`] operations.
66pub mod error;
67
68/// Key encoding and metadata-key constants for the RocksDB storage layout.
69pub mod keys;
70
71/// [`RocksDbCdcLog`] implementation and its [`rhei_core::CdcConsumer`] trait impl.
72pub mod log;
73
74pub use config::RocksDbCdcConfig;
75pub use error::RocksDbCdcError;
76pub use log::RocksDbCdcLog;
rhei_cdc_rocksdb/lib.rs

rhei_cdc_rocksdb/
lib.rs
