s3-wire

Async, streaming S3-compatible client for Rust with explicit bounds on memory, retries, timeouts, and remote input.

Highlights

• Tokio-native uploads and downloads with backpressure
• Replay-aware retries for in-memory and file-backed request bodies
• Managed multipart uploads with bounded concurrency, per-part checksums, and abort cleanup
• SigV4 request signing and presigned object and multipart operations
• Typed object keys, ranges, conditions, checksums, and multipart state
• HTTPS by default, secret-redacting types, and bounded XML parsing
• Integration-tested against pinned MinIO, RustFS, and SeaweedFS releases

s3-wire requires Rust 1.97.1 and does not depend on another S3 client. Patch releases do not intentionally raise the MSRV; see the support policy.

Install

cargo add s3-wire
cargo add tokio --features fs,macros,rt-multi-thread

The default transport uses HTTP/1.1. Enable the http2 crate feature when an endpoint and workload benefit from HTTP/2 negotiation.

Feature tiers

The AWS credential bridge is opt-in because it materially increases build size and may contact configured workload metadata endpoints. Encryption and ecosystem integrations should remain optional or live in separately versioned companion crates so the core client does not silently acquire their dependency or resource costs. The observer API is dependency-free and remains in core.

Quick start

The default credential provider reads AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and optional AWS_SESSION_TOKEN. Remote endpoints must use HTTPS; plain HTTP is an explicit local-testing opt-in.

use s3_wire::{
    ByteStream, Endpoint, GetObjectRequest, ObjectKey, PutObjectRequest, S3Client, S3Config,
};

async fn put_then_get() -> Result<(), Box<dyn std::error::Error>> {
    let region = "us-east-1";
    let config = S3Config::builder()
        .endpoint(Endpoint::for_aws_region(region)?)
        .region(region)
        .bucket("artifact-bucket")
        .build()?;
    let client = S3Client::new(config)?;

    let key = ObjectKey::new("artifacts/report.json")?;
    let mut upload = PutObjectRequest::new(
        key.clone(),
        ByteStream::from_bytes(br#"{"status":"complete"}"#.as_slice()),
    );
    upload.content_type = Some("application/json".into());
    client.put_object(upload).await?;

    let download = client.get_object(GetObjectRequest::new(key)).await?;
    let mut destination = tokio::fs::File::create("report.json").await?;
    let written = download.body.write_to(&mut destination).await?;

    println!("downloaded {written} bytes");
    Ok(())
}

Run the complete CRUD, range, listing, and conditional-write example with:

cargo run --example basic

The example uses the standard AWS credential variables plus S3_BUCKET, and supports both AWS and custom endpoints.

Configuration and credentials

S3Config validates addressing, timeouts, retry policy, response limits, and the credential provider before a client is created:

use std::sync::Arc;
use std::time::Duration;

use s3_wire::{
    AddressingStyle, CachedCredentialsProvider, Credentials, Endpoint, RetryPolicy, S3Client,
    S3Config, StaticCredentialsProvider,
};

fn configured_client() -> Result<S3Client, Box<dyn std::error::Error>> {
    let credentials = Credentials::new("access-key", "secret-key", None)?;
    let static_provider = Arc::new(StaticCredentialsProvider::new(credentials));
    let cached_provider = Arc::new(CachedCredentialsProvider::new(static_provider));
    let retries = RetryPolicy::new(
        4,
        Duration::from_millis(100),
        Duration::from_secs(5),
        Duration::from_secs(20),
    )?;

    let config = S3Config::builder()
        .endpoint(Endpoint::new("https://s3.example.com")?)
        .region("us-east-1")
        .bucket("artifact-bucket")
        .addressing_style(AddressingStyle::Path)
        .connect_timeout(Duration::from_secs(5))
        .attempt_timeout(Duration::from_secs(30))
        .operation_timeout(Duration::from_secs(5 * 60))
        .idle_body_timeout(Duration::from_secs(30))
        .retry_policy(retries)
        .credentials_provider(cached_provider)
        .build()?;

    Ok(S3Client::new(config)?)
}

The default EnvironmentCredentialsProvider needs no explicit configuration. Use StaticCredentialsProvider for an injected immutable value, or implement the async CredentialsProvider trait for a workload-specific source. CachedCredentialsProvider coalesces concurrent refreshes and respects credential expiration. With the opt-in aws-credentials feature, AwsDefaultCredentialsProvider and the region resolver helpers bridge to AWS's standard renewable profile, web-identity, ECS, IMDSv2, credential-process, and assume-role chains without changing the default dependency graph.

Upload sources

Choose a body based on how it should behave if a request must be retried:

File uploads are hashed into an immutable temporary snapshot before the first request. A retry therefore sends the same bytes even if the original file changes.

Multipart uploads

Managed multipart handles part scheduling, a derived in-flight byte bound, one end-to-end transfer deadline, completion, and separately bounded cleanup that quiesces in-flight part requests before aborting:

use std::time::Duration;

use s3_wire::{ManagedMultipartUploadRequest, MultipartOptions, ObjectKey, S3Client};

async fn upload_large_file(client: &S3Client) -> Result<(), Box<dyn std::error::Error>> {
    let options = MultipartOptions::new(8 * 1024 * 1024, 4)?
        .with_transfer_timeout(Duration::from_secs(15 * 60))?;
    let request =
        ManagedMultipartUploadRequest::from_path(
            ObjectKey::new("artifacts/archive.tar")?,
            "archive.tar",
        )
        .with_content_type("application/x-tar")
        .with_options(options);

    client.multipart_upload(request).await?;
    Ok(())
}

Multipart selection is intentional: put_object never switches modes automatically. Call multipart_upload when application policy says a source should use multipart. Primitive create, upload-part, upload-part-copy, complete, list-parts, list-uploads, and abort operations are also available when the application must own multipart state.

Dropping a managed upload stops scheduling parts, gives transmitted requests a bounded opportunity to settle, and then attempts an abort after an upload ID exists. If requests cannot settle, the error exposes cleanup failure because abort cannot be guaranteed to win that race. Process termination can still leave stale uploads, so long-running deployments should also run bounded stale-upload cleanup.

Listing and presigning

list_objects_v2_all follows continuation tokens up to a caller-supplied page limit. Presigned GET, PUT, HEAD, DELETE, multipart-create, part-upload, and multipart-abort requests return a redacted PresignedUrl:

use std::time::Duration;

use s3_wire::{ObjectKey, S3Client};

async fn share_download(client: &S3Client) -> Result<(), Box<dyn std::error::Error>> {
    let key = ObjectKey::new("artifacts/report.json")?;
    let url = client
        .presigned_get(&key, Duration::from_secs(300))
        .await?;

    // Exposure is explicit because the query string contains signing material.
    send_to_authorized_caller(url.expose());
    Ok(())
}

fn send_to_authorized_caller(_url: &str) {}

Presigned URLs are bearer credentials. Keep their lifetime short and do not place exposed URLs in logs, analytics, or error messages.

Error handling

S3Error separates a stable category from optional service metadata. Its Display and Debug implementations omit transport text and cleanup details that may contain credentials or signed URLs:

use s3_wire::{ErrorCategory, S3Error};

fn report(error: &S3Error) {
    match error.category() {
        ErrorCategory::NotFound => eprintln!("object does not exist"),
        ErrorCategory::Timeout => eprintln!("timeout during {:?}", error.timeout_phase()),
        ErrorCategory::Throttling => eprintln!("request was throttled"),
        _ => eprintln!("S3 request failed: {}", error.message()),
    }

    if let Some(request_id) = error.request_id() {
        eprintln!("request id: {request_id}");
    }
    if error.cleanup_failure().is_some() {
        eprintln!("multipart cleanup also failed");
    }
}

Retries are applied inside the client only when the classification, shared retry quota, attempt and elapsed-time limits, operation deadline, and body replayability all permit another attempt. attempts() and retry_stop_reason() explain the final decision, while an optional sanitized observer can record attempt lifecycle events without keys, endpoints, headers, or signing material.

Examples

Focused, runnable examples live in examples/README.md:

• Basic object lifecycle
• Streaming download to a file
• One-shot streaming upload
• Managed multipart upload
• Presigned GET and PUT
• Copy and batch delete
• Content-addressed artifact-store adapter

Compatibility

The client currently covers object upload, download, inspection, deletion, batch deletion, server-side copy, listing, range reads, conditional headers, presigning, and primitive or managed multipart uploads.

The pinned MinIO, RustFS, and SeaweedFS suites run in CI. An opt-in AWS suite exists but has not yet been executed for this release, so compatible-server results are not presented as proof of AWS compatibility. See S3 compatibility for the operation matrix, checksum behavior, test status, and unsupported API families.

Scope and limits

• The client is async-only; for_bucket creates cheap bucket-scoped handles that share one connection pool, credential provider, and retry quota.
• AWS chunked SigV4 streaming is not implemented.
• Automatic PutObject checksum calculation currently supports SHA-256; managed multipart parts support CRC32, CRC32C, CRC64NVME, and SHA-256.
• Managed multipart does not expose a destination If-None-Match condition.
• Bucket administration, ACLs, policies, version listing, and application-managed encryption configuration are outside the current API.

See the security model for deployment responsibilities and architecture for retry, transport, and ownership details.

Documentation

Also see the API example, security reporting, contributing, and the code of conduct.

Development

Run the fast validation set after local changes:

cargo fmt --all -- --check
cargo clippy --locked --all-targets -- -D warnings
cargo clippy --locked --all-targets --all-features -- -D warnings
cargo test --locked --all-targets
cargo test --locked --all-targets --all-features
RUSTDOCFLAGS="-D warnings" cargo doc --locked --no-deps --all-features

The pinned endpoint suites run with ./scripts/test-s3-compat.sh <minio|rustfs|seaweedfs>. Packaging and release checks are described in testing.

License

Licensed under the Apache License 2.0.