s3-wire

Async, streaming S3-compatible client for Rust with explicit bounds on memory, retries, timeouts, and remote input.

Highlights

• Tokio-native uploads and downloads with backpressure
• Replay-aware retries for in-memory and file-backed request bodies
• Managed multipart uploads with bounded concurrency and abort cleanup
• SigV4 request signing and presigned GET and PUT URLs
• Typed object keys, ranges, conditions, checksums, and multipart state
• HTTPS by default, secret-redacting types, and bounded XML parsing
• Integration-tested against pinned MinIO, RustFS, and SeaweedFS releases

s3-wire requires Rust 1.97.1 and does not depend on another S3 client.

Install

cargo add s3-wire
cargo add tokio --features fs,macros,rt-multi-thread

The default transport uses HTTP/1.1. Enable the http2 crate feature when an endpoint and workload benefit from HTTP/2 negotiation.

Quick start

The default credential provider reads AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and optional AWS_SESSION_TOKEN. Remote endpoints must use HTTPS; plain HTTP is an explicit local-testing opt-in.

use s3_wire::{
    ByteStream, Endpoint, GetObjectRequest, ObjectKey, PutObjectRequest, S3Client, S3Config,
};

async fn put_then_get() -> Result<(), Box<dyn std::error::Error>> {
    let region = "us-east-1";
    let config = S3Config::builder()
        .endpoint(Endpoint::for_aws_region(region)?)
        .region(region)
        .bucket("artifact-bucket")
        .build()?;
    let client = S3Client::new(config)?;

    let key = ObjectKey::new("artifacts/report.json")?;
    let mut upload = PutObjectRequest::new(
        key.clone(),
        ByteStream::from_bytes(br#"{"status":"complete"}"#.as_slice()),
    );
    upload.content_type = Some("application/json".into());
    client.put_object(upload).await?;

    let download = client.get_object(GetObjectRequest::new(key)).await?;
    let mut destination = tokio::fs::File::create("report.json").await?;
    let written = download.body.write_to(&mut destination).await?;

    println!("downloaded {written} bytes");
    Ok(())
}

Run the complete CRUD, range, listing, and conditional-write example with:

cargo run --example basic

The example uses the standard AWS credential variables plus S3_BUCKET, and supports both AWS and custom endpoints.

Configuration and credentials

S3Config validates addressing, timeouts, retry policy, response limits, and the credential provider before a client is created:

use std::sync::Arc;
use std::time::Duration;

use s3_wire::{
    AddressingStyle, CachedCredentialsProvider, Credentials, Endpoint, RetryPolicy, S3Client,
    S3Config, StaticCredentialsProvider,
};

fn configured_client() -> Result<S3Client, Box<dyn std::error::Error>> {
    let credentials = Credentials::new("access-key", "secret-key", None)?;
    let static_provider = Arc::new(StaticCredentialsProvider::new(credentials));
    let cached_provider = Arc::new(CachedCredentialsProvider::new(static_provider));
    let retries = RetryPolicy::new(
        4,
        Duration::from_millis(100),
        Duration::from_secs(5),
        Duration::from_secs(20),
    )?;

    let config = S3Config::builder()
        .endpoint(Endpoint::new("https://s3.example.com")?)
        .region("us-east-1")
        .bucket("artifact-bucket")
        .addressing_style(AddressingStyle::Path)
        .connect_timeout(Duration::from_secs(5))
        .attempt_timeout(Duration::from_secs(30))
        .operation_timeout(Duration::from_secs(5 * 60))
        .idle_body_timeout(Duration::from_secs(30))
        .retry_policy(retries)
        .credentials_provider(cached_provider)
        .build()?;

    Ok(S3Client::new(config)?)
}

The default EnvironmentCredentialsProvider needs no explicit configuration. Use StaticCredentialsProvider for an injected immutable value, or implement the async CredentialsProvider trait for a workload-specific source. CachedCredentialsProvider coalesces concurrent refreshes and respects credential expiration.

Upload sources

Choose a body based on how it should behave if a request must be retried:

File uploads are hashed into an immutable temporary snapshot before the first request. A retry therefore sends the same bytes even if the original file changes.

Multipart uploads

Managed multipart handles part scheduling, a derived in-flight byte bound, one end-to-end transfer deadline, completion, and separately bounded cleanup that quiesces in-flight part requests before aborting:

use std::time::Duration;

use s3_wire::{ManagedMultipartUploadRequest, MultipartOptions, ObjectKey, S3Client};

async fn upload_large_file(client: &S3Client) -> Result<(), Box<dyn std::error::Error>> {
    let options = MultipartOptions::new(8 * 1024 * 1024, 4)?
        .with_transfer_timeout(Duration::from_secs(15 * 60))?;
    let request =
        ManagedMultipartUploadRequest::from_path(
            ObjectKey::new("artifacts/archive.tar")?,
            "archive.tar",
        )
        .with_content_type("application/x-tar")
        .with_options(options);

    client.multipart_upload(request).await?;
    Ok(())
}

Multipart selection is intentional: put_object never switches modes automatically. Call multipart_upload when application policy says a source should use multipart. Primitive create, upload-part, complete, list, and abort operations are also available when the application must own multipart state.

Dropping a managed upload stops scheduling parts, gives transmitted requests a bounded opportunity to settle, and then attempts an abort after an upload ID exists. If requests cannot settle, the error exposes cleanup failure because abort cannot be guaranteed to win that race. Process termination can still leave stale uploads, so long-running deployments should also run bounded stale-upload cleanup.

Listing and presigning

list_objects_v2_all follows continuation tokens up to a caller-supplied page limit. presigned_get and presigned_put return a redacted PresignedUrl:

use std::time::Duration;

use s3_wire::{ObjectKey, S3Client};

async fn share_download(client: &S3Client) -> Result<(), Box<dyn std::error::Error>> {
    let key = ObjectKey::new("artifacts/report.json")?;
    let url = client
        .presigned_get(&key, Duration::from_secs(300))
        .await?;

    // Exposure is explicit because the query string contains signing material.
    send_to_authorized_caller(url.expose());
    Ok(())
}

fn send_to_authorized_caller(_url: &str) {}

Presigned URLs are bearer credentials. Keep their lifetime short and do not place exposed URLs in logs, analytics, or error messages.

Error handling

S3Error separates a stable category from optional service metadata. Its Display and Debug implementations omit transport text and cleanup details that may contain credentials or signed URLs:

use s3_wire::{ErrorCategory, S3Error};

fn report(error: &S3Error) {
    match error.category() {
        ErrorCategory::NotFound => eprintln!("object does not exist"),
        ErrorCategory::Timeout => eprintln!("timeout during {:?}", error.timeout_phase()),
        ErrorCategory::Throttling => eprintln!("request was throttled"),
        _ => eprintln!("S3 request failed: {}", error.message()),
    }

    if let Some(request_id) = error.request_id() {
        eprintln!("request id: {request_id}");
    }
    if error.cleanup_failure().is_some() {
        eprintln!("multipart cleanup also failed");
    }
}

Retries are applied inside the client only when the classification, attempt and elapsed-time limits, operation deadline, and body replayability all permit another attempt.

Examples

Focused, runnable examples live in examples/README.md:

• Basic object lifecycle
• Streaming download to a file
• One-shot streaming upload
• Managed multipart upload
• Presigned GET and PUT
• Copy and batch delete
• Content-addressed artifact-store adapter

Compatibility

The client currently covers object upload, download, inspection, deletion, batch deletion, server-side copy, listing, range reads, conditional headers, presigning, and primitive or managed multipart uploads.

The pinned MinIO, RustFS, and SeaweedFS suites run in CI. An opt-in AWS suite exists but has not yet been executed for this release, so compatible-server results are not presented as proof of AWS compatibility. See S3 compatibility for the operation matrix, checksum behavior, test status, and unsupported API families.

Scope and limits

• The client is async-only and configured for one bucket at a time.
• AWS chunked SigV4 streaming is not implemented.
• Automatic upload-checksum calculation currently supports SHA-256.
• Managed multipart does not expose a destination If-None-Match condition.
• Bucket administration, ACLs, policies, version listing, metadata-service credentials, and encryption configuration are outside the current API.

See the security model for deployment responsibilities and architecture for retry, transport, and ownership details.

Documentation

Also see the API example, security reporting, and contributing.

Development

Run the fast validation set after local changes:

cargo fmt --all -- --check
cargo clippy --locked --all-targets -- -D warnings
cargo clippy --locked --all-targets --all-features -- -D warnings
cargo test --locked --all-targets
cargo test --locked --all-targets --all-features
RUSTDOCFLAGS="-D warnings" cargo doc --locked --no-deps --all-features

The pinned endpoint suites run with ./scripts/test-s3-compat.sh <minio|rustfs|seaweedfs>. Packaging and release checks are described in testing.

License

Licensed under the Apache License 2.0.