Module sync_engine 

Core CDC-to-OLAP sync engine.

CdcSyncEngine polls the CDC consumer, groups consecutive INSERT events into SyncOp::BatchInsert groups, and applies them to the OLAP engine using either the Arrow-native path or a SQL fallback.

Transaction semantics

When rhei_core::OlapEngine::supports_transactions returns true (e.g. DuckDB), each sync cycle is wrapped in BEGIN TRANSACTION … COMMIT. A failure triggers ROLLBACK and the watermark is not advanced, so the next cycle retries from the same position.

When the backend does not support transactions (e.g. DataFusion), statements are committed one at a time. On failure the watermark is advanced to the last successfully applied event so that the next cycle does not re-apply already-committed changes — partial-failure recovery.

Stale-schema handling

If the OLAP engine rejects a statement with a message that matches known column-not-found patterns, the event (or batch) is skipped with a warning and the watermark advances past it. This prevents a single stale-schema event from blocking the entire sync pipeline.

Structs

CdcSyncEngine

CDC-based sync engine that replicates changes from OLTP to OLAP.