Crate spate_s3 

Coordinated object-storage (S3) backfill source for Spate.

Point the source at a bucket/prefix and it streams every object’s records through the pipeline: a bounded job that self-terminates (the pipeline exits Completed) once the prefix is exhausted. Delivery is at-least-once: a resume replays from the last committed position, so duplicates are possible and loss is not.

Shape

• Work is planned as splits. The fleet’s elected leader lists the prefix once and packs the sorted listing into splits of ~64 MiB by default: small batches of whole objects, or one byte range of a large uncompressed object. Each has a deterministic identity (split_id_for, split_id_for_range) and a self-contained SplitDescriptor carrying its member keys, sizes, and ETags, and its range. Workers lease splits through the coordination store and read them straight from the descriptors: workers never list.
• One lane per in-flight split. A gained split materializes one data lane (one framework partition, one monotonic offset stream); a record’s i64 offset packs (member ordinal within the split, record index within the object, or within the range for a split over a byte range). coordination.max_in_flight bounds the working set and therefore read parallelism.
• Progress lives in the coordination store, nowhere else. Commits are fenced per-split writes; a lost or stolen split resumes on its next owner from the acked watermark, drift-checked against the descriptor’s ETag pins. Point the pipeline’s coordination: section at a shared store (or hand in a coordinator built in code via S3Source::with_coordinator) and every replica of the pipeline shares the backfill, takes a leader-assigned share, and takes over from the dead. Without one the source runs solo over an in-process store with default tuning, and store: { memory: {} } runs that store with the section’s tuning: correct, but a restart replays the prefix (a startup WARN says so). This crate names no concrete backend and has no store feature.
• Bad objects poison their split, not the pipeline. An object deleted after planning, overwritten under its ETag pin, corrupt, or unreadable past the retry budget hands its split back (surfaced in spate_s3_source_objects_failed_total{reason}); at the attempt cap the split is quarantined. A bounded job with quarantined splits ends failed, never silently incomplete. A rejected object read and a client-configuration error are immediately fatal.
• Records are framed, not decoded, here. The source is format-agnostic: it streams object bytes (after gzip/zstd decompression) through a RecordFramer you supply for the objects’ format via S3Source::with_framer (e.g. spate-json’s NdjsonFramer for NDJSON), emitting one raw payload per framed record. Deserialization then stays in the operator chain (spate-json etc.), as with the Kafka source.

Split identity and drift

A split’s id digests its sorted member keys and ETags (plus the packing-algorithm version, and a byte-range split’s range), and every GET is pinned If-Match to the descriptor’s ETag. The consequences:

• Replanning an unchanged prefix reproduces identical ids, so replans are create-if-absent no-ops and completed splits stay completed.
• An overwritten object shows up as a new split (new id) on the next plan; under an in-flight split it trips the If-Match pin and poisons the split. Either way it can never silently splice into committed progress.
• A deleted object poisons only the splits that still needed it.

Prefer an append-only prefix for the lifetime of a backfill; by default the listing is taken once (refresh_listing: false), so late-arriving keys are not part of the job.

No object_store types appear in this crate’s public API (the same dependency policy that keeps rdkafka out of spate-kafka’s). The one seam that would break it, S3Source::with_store, is gated behind the off-by-default testing feature so tests can inject wrapped or fault-injecting stores without that type reaching a real consumer.

Structs

DescriptorObject

One member object inside a SplitDescriptor.

S3Batch

One poll’s records, borrowed from the lane’s held buffer.

S3Lane

Data-plane pollable unit of the S3 source: owns one slice’s chunk stream and framing state.

S3Source

Coordinated object-storage backfill source. See the crate docs for the delivery model and the split identity/drift contract.

S3SourceConfig

Configuration of an S3Source, deserialized from the pipeline’s opaque source: { s3: ... } section.

SplitDescriptor

The opaque payload carried in a SplitSpec::descriptor: the split’s member objects, in listing order.

SplitRange

A byte range [start, end) of one object, read as a split of its own.

Enums

Compression

Compression codec of the objects under the prefix.

Constants

DESCRIPTOR_VERSION

Version of the SplitDescriptor wire encoding. Bumped on any change to the descriptor’s schema; a worker refuses a descriptor written by an incompatible release instead of misreading it.

Functions

split_id_for

Mint the deterministic split id for a member set.

split_id_for_range

Mint the deterministic split id for the byte range of one object.