Struct S3SourceConfig 

#[non_exhaustive]
pub struct S3SourceConfig {
    pub url: String,
    pub compression: Compression,
    pub split_target_bytes: ByteSize,
    pub refresh_listing: bool,
    pub prefetch_bytes: ByteSize,
    pub chunk_bytes: ByteSize,
    pub store: BTreeMap<String, String>,
}

Configuration of an S3Source, deserialized from the pipeline’s opaque source: { s3: ... } section.

Fields (Non-exhaustive)

Non-exhaustive structs could have additional fields added in future. Therefore, non-exhaustive structs cannot be constructed in external crates using the traditional Struct { .. } syntax; cannot be matched against without a wildcard ..; and struct update syntax will not work.

url: String

Bucket and prefix to backfill, e.g. s3://my-bucket/exports/2026/. Every object under the prefix is read; the planned key set is pinned per split (see the crate docs). file:// URLs work for infrastructure-free runs and tests.

compression: Compression

Compression codec of the objects.

split_target_bytes: ByteSize

Target size of one split — the unit of work distribution, and the unit the leader balances. Objects are packed toward it in listing order with a per-object cost floor of a sixteenth of the target, so a split holds at most ~16 objects; an object at or above the target gets a split of its own. Part of the job fingerprint: every instance of a coordinated job must agree on it. Default 64 MiB.

refresh_listing: bool

Re-list the prefix on the coordinator’s replan interval, planning newly arrived objects as additional splits (an open plan: the job never self-terminates). Default false: one listing, a bounded backfill that completes.

prefetch_bytes: ByteSize

Per-lane read-ahead budget: the size of each lane’s in-memory read window. A lane fetches one window at a time as a bounded ranged GET and drains it before fetching the next, so peak per-lane read-ahead is ~one window and total source read-ahead is in-flight splits × prefetch_bytes (the decoded records a lane holds while framing are a separate, smaller budget). An object at or below this size is read in a single GET.

Keep this modest. A large window is read as one long-held GET — which works against the connection release this source relies on — and a single window must complete inside the client request timeout (the store timeout option, 30s by default), so an oversized window on a slow link degrades into repeated whole-window retries. The 8 MiB default suits typical objects.

chunk_bytes: ByteSize

Target size of a single buffered chunk.

store: BTreeMap<String, String>

Raw object_store options, applied when building the store from url (credentials, region, endpoint, timeouts, …). Bucket keys are rejected — the bucket comes from url.

Implementations

impl S3SourceConfig

pub fn from_component_config( section: &ComponentConfig, ) -> Result<Self, ConfigError>

Deserialize and validate from the pipeline’s opaque component section.

pub fn validate(&self) -> Result<(), ConfigError>

Cross-field validation, including the passthrough denylist.

Trait Implementations

impl Clone for S3SourceConfig

fn clone(&self) -> S3SourceConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for S3SourceConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for S3SourceConfig

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl PartialEq for S3SourceConfig

fn eq(&self, other: &S3SourceConfig) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl StructuralPartialEq for S3SourceConfig