spate-s3

Coordinated object-storage backfill source for the Spate framework: read a prefix of an S3 (or S3-compatible) bucket as a pipeline source, across as many instances as you care to run.

An elected leader plans the prefix into splits; workers lease them with fenced per-split progress in a coordination store; resume is drift-checked against ETag pins, so an object rewritten under a running job is detected rather than silently half-read; and the job self-terminates when the plan completes. It is always coordinated — a single instance is just the one-worker case, which means there is no separate uncoordinated path to diverge.

Applications should depend on the spate facade with the s3 feature.

Sharp edges worth knowing

• The framing is yours to choose and is not optional. This crate is format-agnostic; it hands out byte ranges and something has to say where one record ends and the next begins. with_framer is required, and spate-json supplies an NDJSON framer.
• Backpressure runs through bounded ranged GETs. For an object with an ETag, the source reads GETs sized by prefetch_bytes (8 MiB by default) and buffers each GET in full before handing its chunks on, so a slow sink stalls each read in progress with no request in flight. An object of at most prefetch_bytes arrives in one GET and is held in memory whole, unless split_target_bytes is set below its size and the object is cut into byte ranges. An object with no ETag is read in one streamed GET, and that response stays open while the sink is slow.
• Credentials come from the standard environment chain. Nothing is logged that could carry one — the config's Debug redacts secrets, and there is a test that fails if it stops doing so.
• Enumeration is a single listing pass today. Very large prefixes on plans without an inventory feed are the case to watch.