Skip to main content

Module backfill

Module backfill 

Source
Expand description

faucet backfill — replay a bounded historical window of a pipeline (#282).

A CLI-level orchestration layer (like replication/ and sla/) over expand + executor::run_expanded: the requested range is chunked into independent window units, each unit re-runs the pipeline with its source config scoped to that window (${backfill.start} / ${backfill.end} token substitution + a per-unit ${now.*} clock), and a durable progress marker in the pipeline’s state store makes the whole backfill resumable (--resume). Unit state keys are namespaced ({name}::backfill::{unit}) so the forward-sync bookmark is never touched.

Re-exports§

pub use orchestrator::BackfillOptions;
pub use orchestrator::BackfillOutcome;
pub use orchestrator::BackfillRange;
pub use orchestrator::run_backfill;
pub use spec::BackfillSpec;

Modules§

orchestrator
Backfill orchestration: plan units → gate → run each unit through executor::run_expanded under bounded concurrency → record every unit’s terminal outcome in the durable progress marker.
plan
Backfill-specific planning: ${backfill.*} token substitution and the stable range hash the progress marker is keyed by.
spec
Serde config types for the optional top-level backfill: block.
state
Durable progress marker for faucet backfill — one JSON document per backfill range at {name}::__backfill__::{range_hash} in the pipeline’s state store, recording each unit’s terminal outcome so --resume re-runs only failed/pending units. Kept separate from every live bookmark key ({name}::{row}) and every unit’s scoped key ({name}::backfill::{unit}).