Struct S3SourceConfig 

pub struct S3SourceConfig {
    pub bucket: String,
    pub prefix: Option<String>,
    pub region: Option<String>,
    pub endpoint_url: Option<String>,
    pub file_format: S3FileFormat,
    pub max_objects: Option<usize>,
    pub concurrency: usize,
    pub batch_size: usize,
    pub verify_length: bool,
    pub verify_checksum: bool,
    pub compression: CompressionConfig,
}

Configuration for the S3 source connector.

Fields

bucket: String

S3 bucket name.

prefix: Option<String>

Object key prefix filter.

region: Option<String>

AWS region. None uses the SDK default.

endpoint_url: Option<String>

Custom endpoint URL for S3-compatible services (e.g. MinIO).

file_format: S3FileFormat

Format of the files to read.

max_objects: Option<usize>

Maximum number of objects to read.

concurrency: usize

Maximum number of concurrent object reads (default: 10).

batch_size: usize

Records per emitted StreamPage. For JsonLines and RawText formats, the object body is decoded line-by-line via tokio::io::AsyncBufReadExt and a page is yielded whenever the buffer reaches this size; multi-object scans flatten so a single page may contain lines from any object. For JsonArray, each object is buffered fully before its records are chunked into pages of this size (see the README “Streaming and batching” section for the caveat). Defaults to DEFAULT_BATCH_SIZE.

batch_size = 0 is the “no batching” sentinel: every page is one complete object — no within-object chunking. Useful for small lookup files, or for sinks (e.g. SQL COPY, BigQuery load jobs) that prefer one large request per file to many small ones.

verify_length: bool

Verify each object’s byte length against the Content-Length the store advertises, failing the read with FaucetError::Source on a short (truncated) or over-long transfer (#161). The check is cheap (a byte counter over the body that is read anyway) and defaults to true. Disable it only for an S3-compatible store that does not return a reliable Content-Length. When the store reports no length, the check is skipped (a debug log notes it) rather than failing.

verify_checksum: bool

Verify each object’s body against the checksum the store advertises — an x-amz-checksum-{crc32,crc32c,sha1,sha256} header when present, or the ETag as MD5 for a non-multipart upload (#161). Stronger than the length check but costs a hash over the full body, so it defaults to false. Enabling it sets ChecksumMode::Enabled on each GetObject so the store returns its stored checksum. When the store advertises no usable checksum for an object, verification is skipped for that object (a debug log notes it); the length check still applies.

compression: CompressionConfig

Available on crate feature compression only.

Compression codec applied to each downloaded object. Defaults to CompressionConfig::Auto — the codec is resolved per-object-key, so a single source can read a mix of compressed and uncompressed objects. Requires the crate-local compression feature.

Implementations

impl S3SourceConfig

pub fn new(bucket: impl Into<String>) -> Self

Create a new config with the required bucket name and sensible defaults.

pub fn prefix(self, prefix: impl Into<String>) -> Self

Set the object key prefix filter.

pub fn region(self, region: impl Into<String>) -> Self

Set the AWS region.

pub fn endpoint_url(self, url: impl Into<String>) -> Self

Set a custom endpoint URL for S3-compatible services.

pub fn file_format(self, format: S3FileFormat) -> Self

Set the file format.

pub fn max_objects(self, max: usize) -> Self

Set the maximum number of objects to read.

pub fn concurrency(self, concurrency: usize) -> Self

Set the maximum number of concurrent object reads.

pub fn with_batch_size(self, batch_size: usize) -> Self

Set the per-page record count for Source::stream_pages.

Pass 0 to opt out of within-object chunking — every emitted StreamPage corresponds to exactly one S3 object.

pub fn verify_length(self, verify: bool) -> Self

Enable or disable the per-object Content-Length verification (default true). See verify_length.

pub fn verify_checksum(self, verify: bool) -> Self

Enable or disable per-object checksum verification (default false). See verify_checksum.

pub fn compression(self, c: CompressionConfig) -> Self

Available on crate feature compression only.

Set the compression codec. Available only with the compression feature.

Trait Implementations

impl Clone for S3SourceConfig

fn clone(&self) -> S3SourceConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for S3SourceConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for S3SourceConfig

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl JsonSchema for S3SourceConfig

fn schema_name() -> Cow<'static, str>

The name of the generated JSON Schema.

fn schema_id() -> Cow<'static, str>

Returns a string that uniquely identifies the schema produced by this type.

fn json_schema(generator: &mut SchemaGenerator) -> Schema

Generates a JSON Schema for this type.

fn inline_schema() -> bool

Whether JSON Schemas generated for this type should be included directly in parent schemas, rather than being re-used where possible using the $ref keyword.

impl Serialize for S3SourceConfig

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.