Struct ObjectStoreConfig 

pub struct ObjectStoreConfig {
    pub url: String,
    pub format: FileFormat,
    pub delimiter: Option<String>,
    pub checkpoint_store: Option<String>,
    pub cursor_id: Option<String>,
    pub polling_interval_ms: Option<u64>,
    pub max_object_bytes: Option<u64>,
    pub date_partition: bool,
    pub extension: Option<String>,
    pub compression: Compression,
    pub encryption: Option<EncryptionConfig>,
}

Configuration for a cloud object-store endpoint (S3, GCS, Azure Blob, R2, …).

As a sink, each flushed batch is written as one immutable object under url, named <prefix>/[YYYY/MM/DD/]<uuidv7>.<ext>. As a source, objects under url are listed in key order, fetched, split by delimiter, and emitted as messages; progress is persisted to checkpoint_store (the last processed object key) so a restart resumes without re-emitting. Objects are never mutated or deleted in place.

Fields

url: String

Object-store URL, e.g. s3://bucket/prefix, gs://bucket/prefix, az://account/container/prefix. Credentials are resolved from the environment by the object_store crate (same mechanism as the checkpoint backend); R2 uses s3:// plus a custom AWS_ENDPOINT_URL.

format: FileFormat

Record encoding within an object, shared with the file endpoint. Defaults to normal (one JSON CanonicalMessage per line). CSV is supported for sources only.

delimiter: Option<String>

Record delimiter within an object. Defaults to newline (“\n”). Can be a string or a hex sequence (e.g. “0x00”).

checkpoint_store: Option<String>

(Source only) Durable resume store URL recording the last processed object key, e.g. file:///var/lib/mqb/obj.json, s3://bucket/cursors, or postgres://…. Without it every restart re-lists and re-emits all objects.

cursor_id: Option<String>

(Source only) Cursor id namespacing the checkpoint key; enables durable resume.

polling_interval_ms: Option<u64>

(Source only) Idle poll interval in milliseconds when no new objects are found. Defaults to 1000.

max_object_bytes: Option<u64>

(Source only) Maximum size in bytes of a single object to fetch into memory. An object larger than this fails the read (surfaced as a consumer error) instead of being buffered whole. Unset means no limit (the whole object is materialized).

date_partition: bool

(Sink only) Prepend a YYYY/MM/DD/ path (write time, UTC) to each object key. Purely for readability / lifecycle rules — the uuidv7 name already sorts by time. Default true.

extension: Option<String>

(Sink only) Extension for written objects, without the dot. Defaults to a value derived from format, compression and encryption (e.g. jsonl, csv, bin, jsonl.gz, jsonl.lz4, jsonl.gz.enc); encrypted objects get a trailing .enc since they are ciphertext, not a directly decompressible .gz.

compression: Compression

Whole-object compression (none, gzip, lz4, zstd). Requires the compression feature.

encryption: Option<EncryptionConfig>

At-rest AEAD encryption applied after compression. Requires the encryption feature.

Trait Implementations

impl Clone for ObjectStoreConfig

fn clone(&self) -> ObjectStoreConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for ObjectStoreConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for ObjectStoreConfig

fn default() -> Self

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for ObjectStoreConfig

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Serialize for ObjectStoreConfig

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.