Crate sulfite 

sulfite

Overview

sulfite is a high-level S3 client built on AWS SDK for Rust for even better ease of use, reliability, and bandwidth saturation (>50 Gbps).

The name: SO3^2-, an anion, implying a companion to some other cation (application), is commonly used as a preservative in wines and dried fruits (preserve to S3). It’s S3 with an O in the middle, a play on oxidization.

Motivation

The AWS SDK is a little low-level for users to take advantage of the concurrency & parallelism, with the following challenges:

1. You need to orchestrate the parallel multipart download & upload for large files.
2. The built-in retry settings are too basic (limited to HTTP status codes, none for bytestream errors), and we allow installing higher-level retries.
3. The async API doesn’t agree well with the filesystem for high-throughput operations, when it comes to streaming small chunks from/to disk.

To address them, we provide implementations for the parallel multipart download & upload, and higher-level retries. We also make sure the on-disk file is adequately buffered to avoid async-sync overhead.

Structs

CommonPrefixInfo

A common prefix from a list_objects_v2 response (delimiter-based “directory”).

ListObjectsV2PageIter

Page-by-page iterator for list_objects_v2. Yields one page at a time; retries are applied per page request, so a failure on one page does not invalidate the iterator. MaxKeys is not set (SDK default, typically 1000 keys per page).

ObjectInfo

Metadata for an S3 object (from HEAD or LIST).

RetryConfig

Configuration for retry behavior (max retries, strategy, and which client status codes to retry). Use RetryConfig::default for default retry behavior (no high-level retries).

S3Client

S3ClientConfig

Configuration for the underlying AWS S3 client (region, endpoint, credentials, timeouts).

Enums

RetryStrategy

Configuration for how failed operations are retried.