spate-s3

Object-storage source for the Spate framework: read a prefix of an S3 (or S3-compatible) bucket as a pipeline source, across as many instances as you care to run.

The pipeline's feed: section names the feed the source reads, and a pipeline with none fails to start. The source reads work units from the feed, each naming one object, one byte range of one object, or several objects read whole in turn on one lane, and storage notifications, one part per non-empty created object. It reads a large object in byte ranges. It acknowledges each finished unit to the feed, and every read is pinned to the object's ETag, so an object rewritten under a running job is detected rather than silently half-read. A bounded job, one whose section names a location, ends once the lister's end marker holds over a settling window in which the feed reads empty, and ends with an error when the feed's dead-letter destination then holds units. A solo run on the in-process feed the runtime built ends once its listing is read, with no window; any other job runs until shut down. On a durable feed the source lists nothing: a bounded job's units come from a separate lister run, S3Source::list_once, once per job, and a continuous job's from storage notifications or code. On the in-process feed the run lists its own prefix.

Applications should depend on the spate facade with the s3 feature.

Sharp edges worth knowing

• The framing is yours to choose and is not optional. This crate is format-agnostic; it hands out byte ranges and something has to say where one record ends and the next begins. with_framer is required, and spate-json supplies an NDJSON framer.
• Backpressure runs through bounded ranged GETs. For an object with an ETag, the source reads GETs sized by prefetch_bytes (8 MiB by default) and buffers each GET in full before handing its chunks on, so a slow sink stalls each read in progress with no request in flight. While a lane hands on one window, the source reads the next, so a lane holds three windows when each spans at least four chunks, as at the defaults, and more when windows are shorter. An object of at most prefetch_bytes arrives in one GET and is held in memory whole, unless split_target_bytes is set below its size and the object is cut into byte ranges. An object with no ETag is read in one streamed GET, and that response stays open while the sink is slow.
• Credentials come from the standard environment chain. Nothing is logged that could carry one — the config's Debug redacts secrets, and there is a test that fails if it stops doing so.
• Enumeration is a single listing pass today. Very large prefixes on plans without an inventory feed are the case to watch.