surreal-sync-runtime 0.6.0

Shared runtime: apply pipeline, init, SurrealDB config, and transform loading for surreal-sync
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54

//! Watermark-based interleaved snapshot full-sync framework.
//!
//! This crate implements a DBLog-style interleaved snapshot: tables are copied
//! in resumable, primary-key-ordered chunks *concurrently with* consuming the
//! source change stream, using low/high watermarks to deduplicate snapshot
//! reads against live changes (the log event always wins).
//!
//! The generic algorithm lives in [`run_interleaved_snapshot`]; each source
//! backend implements [`WatermarkSource`] to supply chunk reads, watermark
//! writes, stream consumption, position reporting, and consumed-log freeing.
//! Results are written through the version-independent
//! [`surreal_sync_core::SurrealSink`].
//!
//! # Guarantees
//!
//! - **Bounded memory**: the only buffered state is a single chunk (at most
//!   `chunk_size` rows), so memory is `O(chunk_size)` regardless of table size.
//!   [`InterleavedSnapshotResult::peak_buffered_rows`] exposes the exact peak.
//! - **Resumable**: progress is checkpointed per chunk via
//!   [`SnapshotCheckpointer`], so a crash resumes at the last copied primary
//!   key rather than restarting the table.
//! - **Bounded retention**: [`WatermarkSource::commit_reconciled`] is called as
//!   the stream is applied, so the source can free change-log data continuously
//!   instead of pinning it for the whole snapshot.
//! - **Overlapping transforms and R∩W**: reconciliation events and surviving
//!   chunk rows share one long-lived [`crate::pipeline::run_source_runtime`]
//!   apply window across chunks, so `max_in_flight > 1` can hide slow
//!   transforms and keep polling while ordered sink applies run. Progress /
//!   [`WatermarkSource::commit_reconciled`] are saved only after each chunk's
//!   events are sink-safe.

mod checkpointer;
mod runner;
mod source;
mod types;

#[cfg(test)]
#[path = "tests.rs"]
mod tests;

pub use checkpointer::{ManagerCheckpointer, NoopCheckpointer, SnapshotCheckpointer};
pub use runner::{
    run_adhoc_snapshot_tables, run_adhoc_snapshot_tables_with_transforms, run_interleaved_snapshot,
    run_interleaved_snapshot_with_resume, run_interleaved_snapshot_with_resume_and_transforms,
    run_interleaved_snapshot_with_transforms, InterleavedSnapshotConfig, InterleavedSnapshotResult,
    SnapshotTransforms, DEFAULT_CHUNK_SIZE,
};
pub use source::WatermarkSource;
pub use types::{
    PkTuple, ReconciliationEvent, ReconciliationPos, SnapshotSignal, TableSpec, WatermarkKind,
};

// Re-export the resumable checkpoint types the framework produces.
pub use surreal_sync_core::{InterleavedSnapshotCheckpoint, SnapshotTableProgress};