Struct S3SourceConfig 

pub struct S3SourceConfig {
    pub bucket: String,
    pub prefix: Option<String>,
    pub region: Option<String>,
    pub endpoint_url: Option<String>,
    pub file_format: S3FileFormat,
    pub max_objects: Option<usize>,
    pub concurrency: usize,
    pub batch_size: usize,
    pub compression: CompressionConfig,
}

Available on crate features source-rest and source-s3 only.

Configuration for the S3 source connector.

Fields

bucket: String

S3 bucket name.

prefix: Option<String>

Object key prefix filter.

region: Option<String>

AWS region. None uses the SDK default.

endpoint_url: Option<String>

Custom endpoint URL for S3-compatible services (e.g. MinIO).

file_format: S3FileFormat

Format of the files to read.

max_objects: Option<usize>

Maximum number of objects to read.

concurrency: usize

Maximum number of concurrent object reads (default: 10).

batch_size: usize

Records per emitted StreamPage. For JsonLines and RawText formats, the object body is decoded line-by-line via tokio::io::AsyncBufReadExt and a page is yielded whenever the buffer reaches this size; multi-object scans flatten so a single page may contain lines from any object. For JsonArray, each object is buffered fully before its records are chunked into pages of this size (see the README “Streaming and batching” section for the caveat). Defaults to DEFAULT_BATCH_SIZE.

batch_size = 0 is the “no batching” sentinel: every page is one complete object — no within-object chunking. Useful for small lookup files, or for sinks (e.g. SQL COPY, BigQuery load jobs) that prefer one large request per file to many small ones.

compression: CompressionConfig

Available on crate feature compression only.

Compression codec applied to each downloaded object. Defaults to CompressionConfig::Auto — the codec is resolved per-object-key, so a single source can read a mix of compressed and uncompressed objects. Requires the crate-local compression feature.

Implementations

impl S3SourceConfig

pub fn new(bucket: impl Into<String>) -> S3SourceConfig

Create a new config with the required bucket name and sensible defaults.

pub fn prefix(self, prefix: impl Into<String>) -> S3SourceConfig

Set the object key prefix filter.

pub fn region(self, region: impl Into<String>) -> S3SourceConfig

Set the AWS region.

pub fn endpoint_url(self, url: impl Into<String>) -> S3SourceConfig

Set a custom endpoint URL for S3-compatible services.

pub fn file_format(self, format: S3FileFormat) -> S3SourceConfig

Set the file format.

pub fn max_objects(self, max: usize) -> S3SourceConfig

Set the maximum number of objects to read.

pub fn concurrency(self, concurrency: usize) -> S3SourceConfig

Set the maximum number of concurrent object reads.

pub fn with_batch_size(self, batch_size: usize) -> S3SourceConfig

Set the per-page record count for Source::stream_pages.

Pass 0 to opt out of within-object chunking — every emitted StreamPage corresponds to exactly one S3 object.

pub fn compression(self, c: CompressionConfig) -> S3SourceConfig

Available on crate feature compression only.

Set the compression codec. Available only with the compression feature.

Trait Implementations

impl Clone for S3SourceConfig

fn clone(&self) -> S3SourceConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for S3SourceConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for S3SourceConfig

fn deserialize<__D>( __deserializer: __D, ) -> Result<S3SourceConfig, <__D as Deserializer<'de>>::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl JsonSchema for S3SourceConfig

fn schema_name() -> Cow<'static, str>

The name of the generated JSON Schema.

fn schema_id() -> Cow<'static, str>

Returns a string that uniquely identifies the schema produced by this type.

fn json_schema(generator: &mut SchemaGenerator) -> Schema

Generates a JSON Schema for this type.

fn inline_schema() -> bool

Whether JSON Schemas generated for this type should be included directly in parent schemas, rather than being re-used where possible using the $ref keyword.

impl Serialize for S3SourceConfig

fn serialize<__S>( &self, __serializer: __S, ) -> Result<<__S as Serializer>::Ok, <__S as Serializer>::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.