Expand description
In-run datetime window slicing for forward incremental (#527).
ReplicationBind (#513) binds only a single lower
bound (?since=<bookmark>); faucet backfill (#282) windows only
a bounded historical range. Neither bounds each request of the ordinary
forward-incremental run.
Many APIs — analytics / ads / reporting feeds especially — require both a
lower and an upper bound and cap the span (e.g. reject a range over 30 or
90 days). Against those, an unbounded ?since=<bookmark> either errors or,
worse, silently truncates. Window slicing bounds each request to a rolling
[start, end) interval between the persisted bookmark and now, iterating the
windows within a single run and persisting the window boundary as the bookmark
at each step so the run is resumable mid-sweep. This is parity with Airbyte’s
DatetimeBasedCursor (start_datetime / end_datetime / step /
cursor_granularity / lookback_window).
The enumeration is a pure function (enumerate_windows); a source injects
the rendered boundaries into its requests via the WindowBinds (the window
analogue of ReplicationBind).
Structs§
- Window
- One half-open
[start, end)slice of the replication timeline. - Window
Bind - Injects a rendered window boundary into the outgoing request — the window
analogue of
ReplicationBind. Reuses the sameBindTargetplacement andBindFormatformatting. - Window
Spec - Declarative in-run datetime window slicing (#527).
Constants§
- DEFAULT_
MAX_ WINDOWS - Default
WindowSpec::max_windows: a runaway backstop, far above any real sweep. On a first run against years of history at a smallstep, the sweep is truncated here and the next run resumes from the last window. - WINDOW_
PLACEHOLDER - The placeholder replaced by the formatted window boundary inside a
WindowBind::template. Thelowerbind renders the window start, theupperbind renders the window end.
Functions§
- enumerate_
windows - Enumerate contiguous half-open
[start, end)windows fromstart(minuslookback) up tonow, eachstepwide (the last clamped tonow). - parse_
step - Parse a
WindowSpecduration string (step/granularity/lookback) into an absolutechrono::Duration:45s/30m/6h/30d(d= 24h), or a bare integer (= seconds). Must be positive.