Struct GcsSourceConfig 

pub struct GcsSourceConfig {
    pub bucket: String,
    pub prefix: Option<String>,
    pub object_keys: Option<Vec<String>>,
    pub auth: GcsCredentials,
    pub file_format: GcsFileFormat,
    pub max_objects: Option<usize>,
    pub concurrency: usize,
    pub batch_size: usize,
    pub storage_host: Option<String>,
    pub compression: CompressionConfig,
}

Available on crate features source-gcs and source-rest only.

Configuration for the GCS source connector.

Fields

bucket: String

GCS bucket name.

prefix: Option<String>

Object name prefix filter. Ignored when object_keys is set.

object_keys: Option<Vec<String>>

Explicit object names. When set, listing is skipped and prefix is ignored.

auth: GcsCredentials

Credential source.

file_format: GcsFileFormat

File format.

max_objects: Option<usize>

Hard cap on the number of objects read (after listing).

concurrency: usize

Maximum concurrent object reads (default: 10).

batch_size: usize

Records per emitted StreamPage. See “Streaming and batching” in the README. batch_size = 0 is the “no batching” sentinel and emits one page per object.

storage_host: Option<String>

Optional storage-host override (e.g. http://localhost:4443 for fake-gcs-server). Production users should leave this unset.

compression: CompressionConfig

Available on crate feature compression only.

Compression codec applied to each downloaded object. Defaults to CompressionConfig::Auto — the codec is resolved per-object-key, so a single source can read a mix of compressed and uncompressed objects. Requires the crate-local compression feature.

Implementations

impl GcsSourceConfig

pub fn new(bucket: impl Into<String>) -> GcsSourceConfig

Create a new config with the required bucket name and sensible defaults.

pub fn prefix(self, prefix: impl Into<String>) -> GcsSourceConfig

pub fn object_keys(self, keys: Vec<String>) -> GcsSourceConfig

pub fn auth(self, creds: GcsCredentials) -> GcsSourceConfig

pub fn file_format(self, format: GcsFileFormat) -> GcsSourceConfig

pub fn max_objects(self, max: usize) -> GcsSourceConfig

pub fn concurrency(self, concurrency: usize) -> GcsSourceConfig

pub fn with_batch_size(self, batch_size: usize) -> GcsSourceConfig

pub fn storage_host(self, host: impl Into<String>) -> GcsSourceConfig

pub fn compression(self, c: CompressionConfig) -> GcsSourceConfig

Available on crate feature compression only.

Set the compression codec. Available only with the compression feature.

Trait Implementations

impl Clone for GcsSourceConfig

fn clone(&self) -> GcsSourceConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for GcsSourceConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for GcsSourceConfig

fn deserialize<__D>( __deserializer: __D, ) -> Result<GcsSourceConfig, <__D as Deserializer<'de>>::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl JsonSchema for GcsSourceConfig

fn schema_name() -> Cow<'static, str>

The name of the generated JSON Schema.

fn schema_id() -> Cow<'static, str>

Returns a string that uniquely identifies the schema produced by this type.

fn json_schema(generator: &mut SchemaGenerator) -> Schema

Generates a JSON Schema for this type.

fn inline_schema() -> bool

Whether JSON Schemas generated for this type should be included directly in parent schemas, rather than being re-used where possible using the $ref keyword.

impl Serialize for GcsSourceConfig

fn serialize<__S>( &self, __serializer: __S, ) -> Result<<__S as Serializer>::Ok, <__S as Serializer>::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.