Skip to main content

Module chunking

Module chunking 

Source
Expand description

Pure range chunking shared by faucet backfill and the partition: block (#479). Boundary parsing, timezone/DST-correct time windows, integer-range splitting, and offset/limit splitting. No I/O.

Both consumers are CLI-layer — the substitution that uses these chunks walks config strings, which no connector crate participates in — so this lives here rather than in faucet-core. That also avoids making chrono non-optional and adding chrono-tz to core, neither of which a connector author needs.

The time half was moved here verbatim from backfill::plan, which now re-exports it, so faucet backfill keeps byte-identical planning (and its existing tests keep passing against the moved code).

§Bounds are the correctness hazard

An integer range can be split two ways, and picking wrong is silent data loss, not an error. With chunk_size: 10000 from 0:

Boundschunk 1chunk 2emitted end
Inclusive[0, 9999][10000, 19999]9999
HalfOpen[0, 10000)[10000, 20000)10000

Half-open chunks against an API whose upper bound is inclusive fetch record 10000 twice; inclusive chunks against an exclusive API never fetch record 9999. Neither surfaces as a failure, which is why the config field has no default — the user has to state which their source is.

Structs§

IntChunk
One independent slice of an integer range.
OffsetChunk
One offset/limit slice of a countable result set.
TimeChunk
One independent, resumable slice of a time range.

Enums§

Bounds
Whether a chunk’s upper edge is included. See the module docs — this has no default on purpose.
WindowStep
How a window advances the cursor.

Constants§

MAX_UNITS
Hard ceiling on planned chunks — a tiny window over a huge range is a config error, not a workload.
WARN_UNITS
Above this many chunks a loud warning is emitted (but planning proceeds).

Functions§

parse_boundary
Parse a range boundary: RFC3339 (2026-06-01T00:00:00Z) or a bare date (2026-06-01, interpreted as midnight in tz). A date that falls in a DST gap resolves to the earliest valid instant.
parse_window
Parse a window duration: 45s, 30m, 6h, 1d, 1w (or a bare integer = seconds). Must be positive.
plan_int_chunks
Split [from, to] (or [from, to) per bounds) into contiguous chunks of at most chunk_size values.
plan_offset_chunks
Split a result set of total rows into offset/limit chunks.
plan_windows
Chunk [from, to) into contiguous half-open windows of window (the last window truncated at to). window: None = the whole range as one unit. Window arithmetic is absolute (instants), so units never gap or overlap — including across DST transitions; boundaries are re-rendered in tz so ${now.*} tokens see local wall-clock time.