Expand description
faucet backfill — replay a bounded historical window of a pipeline
(#282).
A CLI-level orchestration layer (like replication/ and sla/) over
expand + executor::run_expanded: the requested range is chunked into
independent window units, each unit re-runs the pipeline with its
source config scoped to that window (${backfill.start} /
${backfill.end} token substitution + a per-unit ${now.*} clock), and a
durable progress marker in the pipeline’s state store makes the whole
backfill resumable (--resume). Unit state keys are namespaced
({name}::backfill::{unit}) so the forward-sync bookmark is never
touched.
Re-exports§
pub use orchestrator::BackfillOptions;pub use orchestrator::BackfillOutcome;pub use orchestrator::BackfillRange;pub use orchestrator::run_backfill;pub use spec::BackfillSpec;
Modules§
- orchestrator
- Backfill orchestration: plan units → gate → run each unit through
executor::run_expandedunder bounded concurrency → record every unit’s terminal outcome in the durable progress marker. - plan
- Backfill-specific planning:
${backfill.*}token substitution and the stable range hash the progress marker is keyed by. - spec
- Serde config types for the optional top-level
backfill:block. - state
- Durable progress marker for
faucet backfill— one JSON document per backfill range at{name}::__backfill__::{range_hash}in the pipeline’s state store, recording each unit’s terminal outcome so--resumere-runs only failed/pending units. Kept separate from every live bookmark key ({name}::{row}) and every unit’s scoped key ({name}::backfill::{unit}).