# s3-wire
[](https://crates.io/crates/s3-wire)
[](https://docs.rs/s3-wire)
[](https://github.com/runtrue/s3-rs/actions/workflows/ci.yml)
Async, streaming S3-compatible client for Rust with explicit bounds on memory, retries, timeouts, and remote input.
## Highlights
- Tokio-native uploads and downloads with backpressure
- Replay-aware retries for in-memory and file-backed request bodies
- Managed multipart uploads with bounded concurrency, per-part checksums, and abort cleanup
- SigV4 request signing and presigned object and multipart operations
- Typed object keys, ranges, conditions, checksums, and multipart state
- HTTPS by default, secret-redacting types, and bounded XML parsing
- Integration-tested against pinned MinIO, RustFS, and SeaweedFS releases
`s3-wire` requires Rust 1.97.1 and does not depend on another S3 client. Patch
releases do not intentionally raise the MSRV; see the [support policy](SUPPORT.md).
## Install
```sh
cargo add s3-wire
cargo add tokio --features fs,macros,rt-multi-thread
```
The default transport uses HTTP/1.1. Enable the `http2` crate feature when an
endpoint and workload benefit from HTTP/2 negotiation.
### Feature tiers
| Core | default feature set | Production data-plane API; no optional feature is required for ordinary S3 operations |
| Transport option | `http2` | Adds HTTP/2 negotiation without changing request semantics |
| Credential bridge | `aws-credentials` | Adds the AWS profile, web-identity, ECS, IMDSv2, credential-process, assume-role, and region provider chains |
| Verification only | `aws-compat`, `fuzzing` | Test and benchmark entry points; not application features or compatibility promises |
The AWS credential bridge is opt-in because it materially increases build size
and may contact configured workload metadata endpoints. Encryption and
ecosystem integrations should remain optional or live in separately versioned
companion crates so the core client does not silently acquire their dependency
or resource costs. The observer API is dependency-free and remains in core.
## Quick start
The default credential provider reads `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and optional `AWS_SESSION_TOKEN`. Remote endpoints must use HTTPS; plain HTTP is an explicit local-testing opt-in.
```rust,no_run
use s3_wire::{
ByteStream, Endpoint, GetObjectRequest, ObjectKey, PutObjectRequest, S3Client, S3Config,
};
async fn put_then_get() -> Result<(), Box<dyn std::error::Error>> {
let region = "us-east-1";
let config = S3Config::builder()
.endpoint(Endpoint::for_aws_region(region)?)
.region(region)
.bucket("artifact-bucket")
.build()?;
let client = S3Client::new(config)?;
let key = ObjectKey::new("artifacts/report.json")?;
let mut upload = PutObjectRequest::new(
key.clone(),
ByteStream::from_bytes(br#"{"status":"complete"}"#.as_slice()),
);
upload.content_type = Some("application/json".into());
client.put_object(upload).await?;
let download = client.get_object(GetObjectRequest::new(key)).await?;
let mut destination = tokio::fs::File::create("report.json").await?;
let written = download.body.write_to(&mut destination).await?;
println!("downloaded {written} bytes");
Ok(())
}
```
Run the complete CRUD, range, listing, and conditional-write example with:
```sh
cargo run --example basic
```
The example uses the standard AWS credential variables plus `S3_BUCKET`, and supports both AWS and custom endpoints.
## Configuration and credentials
`S3Config` validates addressing, timeouts, retry policy, response limits, and the credential provider before a client is created:
```rust,no_run
use std::sync::Arc;
use std::time::Duration;
use s3_wire::{
AddressingStyle, CachedCredentialsProvider, Credentials, Endpoint, RetryPolicy, S3Client,
S3Config, StaticCredentialsProvider,
};
fn configured_client() -> Result<S3Client, Box<dyn std::error::Error>> {
let credentials = Credentials::new("access-key", "secret-key", None)?;
let static_provider = Arc::new(StaticCredentialsProvider::new(credentials));
let cached_provider = Arc::new(CachedCredentialsProvider::new(static_provider));
let retries = RetryPolicy::new(
4,
Duration::from_millis(100),
Duration::from_secs(5),
Duration::from_secs(20),
)?;
let config = S3Config::builder()
.endpoint(Endpoint::new("https://s3.example.com")?)
.region("us-east-1")
.bucket("artifact-bucket")
.addressing_style(AddressingStyle::Path)
.connect_timeout(Duration::from_secs(5))
.attempt_timeout(Duration::from_secs(30))
.operation_timeout(Duration::from_secs(5 * 60))
.idle_body_timeout(Duration::from_secs(30))
.retry_policy(retries)
.credentials_provider(cached_provider)
.build()?;
Ok(S3Client::new(config)?)
}
```
The default `EnvironmentCredentialsProvider` needs no explicit configuration.
Use `StaticCredentialsProvider` for an injected immutable value, or implement
the async `CredentialsProvider` trait for a workload-specific source.
`CachedCredentialsProvider` coalesces concurrent refreshes and respects
credential expiration. With the opt-in `aws-credentials` feature,
`AwsDefaultCredentialsProvider` and the region resolver helpers bridge to AWS's
standard renewable profile, web-identity, ECS, IMDSv2, credential-process, and
assume-role chains without changing the default dependency graph.
## Upload sources
Choose a body based on how it should behave if a request must be retried:
| `ByteStream::from_bytes` | Yes | Retains the input bytes | Best for small, already-buffered values |
| `ByteStream::from_path` | Yes | Streams from a private disk snapshot | Requires temporary disk space |
| `ByteStream::from_stream` | No | Streams with backpressure | Caller supplies exact length and SHA-256 |
File uploads are hashed into an immutable temporary snapshot before the first request. A retry therefore sends the same bytes even if the original file changes.
## Multipart uploads
Managed multipart handles part scheduling, a derived in-flight byte bound, one
end-to-end transfer deadline, completion, and separately bounded cleanup that
quiesces in-flight part requests before aborting:
```rust,no_run
use std::time::Duration;
use s3_wire::{ManagedMultipartUploadRequest, MultipartOptions, ObjectKey, S3Client};
async fn upload_large_file(client: &S3Client) -> Result<(), Box<dyn std::error::Error>> {
let options = MultipartOptions::new(8 * 1024 * 1024, 4)?
.with_transfer_timeout(Duration::from_secs(15 * 60))?;
let request =
ManagedMultipartUploadRequest::from_path(
ObjectKey::new("artifacts/archive.tar")?,
"archive.tar",
)
.with_content_type("application/x-tar")
.with_options(options);
client.multipart_upload(request).await?;
Ok(())
}
```
Multipart selection is intentional: `put_object` never switches modes
automatically. Call `multipart_upload` when application policy says a source
should use multipart. Primitive create, upload-part, upload-part-copy, complete,
list-parts, list-uploads, and abort operations are also available when the
application must own multipart state.
Dropping a managed upload stops scheduling parts, gives transmitted requests a
bounded opportunity to settle, and then attempts an abort after an upload ID
exists. If requests cannot settle, the error exposes cleanup failure because
abort cannot be guaranteed to win that race. Process termination can still
leave stale uploads, so long-running deployments should also run bounded
stale-upload cleanup.
## Listing and presigning
`list_objects_v2_all` follows continuation tokens up to a caller-supplied page
limit. Presigned GET, PUT, HEAD, DELETE, multipart-create, part-upload, and
multipart-abort requests return a redacted `PresignedUrl`:
```rust,no_run
use std::time::Duration;
use s3_wire::{ObjectKey, S3Client};
async fn share_download(client: &S3Client) -> Result<(), Box<dyn std::error::Error>> {
let key = ObjectKey::new("artifacts/report.json")?;
let url = client
.presigned_get(&key, Duration::from_secs(300))
.await?;
// Exposure is explicit because the query string contains signing material.
send_to_authorized_caller(url.expose());
Ok(())
}
fn send_to_authorized_caller(_url: &str) {}
```
Presigned URLs are bearer credentials. Keep their lifetime short and do not place exposed URLs in logs, analytics, or error messages.
## Error handling
`S3Error` separates a stable category from optional service metadata. Its `Display` and `Debug` implementations omit transport text and cleanup details that may contain credentials or signed URLs:
```rust
use s3_wire::{ErrorCategory, S3Error};
fn report(error: &S3Error) {
match error.category() {
ErrorCategory::NotFound => eprintln!("object does not exist"),
ErrorCategory::Timeout => eprintln!("timeout during {:?}", error.timeout_phase()),
ErrorCategory::Throttling => eprintln!("request was throttled"),
_ => eprintln!("S3 request failed: {}", error.message()),
}
if let Some(request_id) = error.request_id() {
eprintln!("request id: {request_id}");
}
if error.cleanup_failure().is_some() {
eprintln!("multipart cleanup also failed");
}
}
```
Retries are applied inside the client only when the classification, shared
retry quota, attempt and elapsed-time limits, operation deadline, and body
replayability all permit another attempt. `attempts()` and
`retry_stop_reason()` explain the final decision, while an optional sanitized
observer can record attempt lifecycle events without keys, endpoints, headers,
or signing material.
## Examples
Focused, runnable examples live in [`examples/README.md`](examples/README.md):
- [Basic object lifecycle](examples/basic.rs)
- [Streaming download to a file](examples/download_file.rs)
- [One-shot streaming upload](examples/stream_upload.rs)
- [Managed multipart upload](examples/multipart_upload.rs)
- [Presigned GET and PUT](examples/presign.rs)
- [Copy and batch delete](examples/copy_and_delete.rs)
- [Content-addressed artifact-store adapter](examples/artifact_store.rs)
## Compatibility
The client currently covers object upload, download, inspection, deletion, batch deletion, server-side copy, listing, range reads, conditional headers, presigning, and primitive or managed multipart uploads.
The pinned MinIO, RustFS, and SeaweedFS suites run in CI. An opt-in AWS suite exists but has not yet been executed for this release, so compatible-server results are not presented as proof of AWS compatibility. See [S3 compatibility](docs/compatibility.md) for the operation matrix, checksum behavior, test status, and unsupported API families.
## Scope and limits
- The client is async-only; `for_bucket` creates cheap bucket-scoped handles
that share one connection pool, credential provider, and retry quota.
- AWS chunked SigV4 streaming is not implemented.
- Automatic `PutObject` checksum calculation currently supports SHA-256;
managed multipart parts support CRC32, CRC32C, CRC64NVME, and SHA-256.
- Managed multipart does not expose a destination `If-None-Match` condition.
- Bucket administration, ACLs, policies, version listing, and application-managed
encryption configuration are outside the current API.
See the [security model](docs/security-model.md) for deployment responsibilities and [architecture](docs/architecture.md) for retry, transport, and ownership details.
## Documentation
| [API reference](https://docs.rs/s3-wire) | Public types, methods, and crate-level quick start |
| [Architecture](docs/architecture.md) | Modules, request flow, retry rules, and transfer ownership |
| [S3 compatibility](docs/compatibility.md) | Operations, signing, checksums, tested services, and non-goals |
| [Security model](docs/security-model.md) | Trust boundaries, controls, secret handling, and operator duties |
| [Testing](docs/testing.md) | Local checks, MinIO, AWS, property tests, fuzzing, and benchmarks |
| [Performance and size](docs/performance.md) | Retained measurements, reproduction, and interpretation |
| [sandboxd integration](docs/sandboxd-integration.md) | Content-addressed publication, reads, garbage collection, and cleanup |
| [Examples](examples/README.md) | Runnable object, streaming, multipart, presigning, and integration flows |
| [Releasing](docs/releasing.md) | Package validation, tagging, publication, and post-release checks |
| [Support policy](SUPPORT.md) | Supported releases, MSRV, issue evidence, and project scope |
| [Roadmap](ROADMAP.md) | Direction, production-readiness priorities, and core non-goals |
Also see [the API example](examples/basic.rs), [security reporting](SECURITY.md),
[contributing](CONTRIBUTING.md), and the [code of conduct](CODE_OF_CONDUCT.md).
## Development
Run the fast validation set after local changes:
```sh
cargo fmt --all -- --check
cargo clippy --locked --all-targets -- -D warnings
cargo clippy --locked --all-targets --all-features -- -D warnings
cargo test --locked --all-targets
cargo test --locked --all-targets --all-features
RUSTDOCFLAGS="-D warnings" cargo doc --locked --no-deps --all-features
```
The pinned endpoint suites run with `./scripts/test-s3-compat.sh <minio|rustfs|seaweedfs>`. Packaging and release checks are described in [testing](docs/testing.md).
## License
Licensed under the [Apache License 2.0](LICENSE).