Skip to main content

Crate aws_s3_transfer

Crate aws_s3_transfer 

Source
Expand description

Transfer Manager for Amazon S3 - a high-performance data plane client for uploading and downloading objects and directories.

The transfer manager splits large transfers into concurrent part requests over the Amazon S3 service API, retries each failed part individually, and sizes its concurrency and memory use to the host. Client starts transfers; each returns a handle to wait for, control or abort the transfer.

§Examples

Load the default configuration:

let config = aws_s3_transfer::from_env().load().await;
let client = aws_s3_transfer::Client::new(config);

Download a bucket to a local directory:

let config = aws_s3_transfer::from_env().load().await;
let client = aws_s3_transfer::Client::new(config);

let handle = client
    .download_objects()
    .bucket("my-bucket")
    .destination("/tmp/my-bucket")
    .initiate()?;

// wait for transfer to complete
handle.join().await?;

§Operations

See the documentation for each client operation for more information:

§Configuration

from_env returns a ConfigLoader, which loads S3 client configuration from the shared AWS configuration sources, detects the host to size the automatic defaults, and applies any settings made on it. ConfigLoader::s3_config replaces the shared-configuration load with an existing aws_types::SdkConfig or S3ClientConfig.

Config::builder constructs a configuration from explicit values instead. It requires an S3ClientConfig, loads no shared configuration, and makes no EC2 instance metadata request; the client detects the host from local system information only. Use it when the S3 configuration is already complete and neither environment loading nor network detection is wanted.

RuntimeConfig selects where transfer work runs, which also determines the HTTP transport used when no HTTP client is configured; see Runtime and HTTP.

§Integrity

Uploads send the checksums selected by ChecksumStrategy, which S3 verifies against the bytes it receives. An expected value set with ChecksumStrategy::with_expected_value has S3 check the uploaded object against its source.

A completed download reports whether the bytes it delivered were validated against S3’s checksums: DownloadOutput::integrity_checks returns IntegrityChecks, whose ChecksumValidation is Validated or says why the bytes were not validated. Validation covers the bytes as received from S3, not the destination they were written to. A checksum mismatch fails join() with ErrorKind::IntegrityError. Streamed bytes can arrive before the final check, so a download has succeeded only when join() returns Ok.

A successful download to a path means the operating system accepted every byte and the file was renamed into place; the transfer manager does not sync the data to stable storage. The download module describes when each check applies.

§Runtime and HTTP

By default, transfer work runs on threads the transfer manager creates and owns (RuntimeConfig::managed), and requests use the transfer manager’s own HTTP transport, which drives each connection on the thread that opened it. RuntimeConfig::multi_thread_tokio runs transfer work on the caller’s Tokio runtime instead, and requests use the AWS SDK’s default HTTP client.

An HTTP client set on the aws_types::SdkConfig or S3 configuration is used with either runtime, in place of the transfer manager’s transport or the SDK default. To change TLS, DNS or connection settings, configure an HTTP client. RuntimeConfig describes what each transport provides.

aws-s3-transfer-ext is an experimental crate that adopts a managed runtime built by the caller; its API may change in any release.

§Modeled types

S3 request values, response metadata, and enums live in types, and their builders in types::builders. Operation modules re-export the modeled types and builders they use.

The sdk-v1 feature enables conversions between modeled values and AWS SDK for Rust S3 v1 types, conversion from an S3 configuration builder, and S3WalkContextBuilder construction from an SDK client.

§WebAssembly

The crate builds for wasm32-wasip1, wasm32-wasip2, and wasm32-wasip3. Other wasm targets are not supported. On these targets:

  • Build with RUSTFLAGS="--cfg tokio_unstable". The S3 client’s tokio integration enables tokio’s fs feature, which tokio accepts on wasm only with that flag.
  • The AWS SDK’s default HTTPS client is not built. Requests fail unless the aws_types::SdkConfig or S3 configuration supplies an HTTP client, for example one built on wasi:http. wasm32-wasip1 has no networking interface for such a client.
  • Managed execution behaves as RuntimeConfig::multi_thread_tokio, and a current-thread tokio runtime is supported. Client::shutdown_completion returns None. File reads and writes run inline on the thread that polls the transfer.
  • The __unstable_ext feature, which adopts a prebuilt managed runtime, is unavailable.
  • Memory detection is unavailable, so MemoryBudgetConfig::Auto uses its 2 GiB fallback. Use MemoryBudgetConfig::Limit to fit the module’s linear memory.

§Runtime diagnostics

Tracing output and AWS_S3_TM_DIAGNOSTICS are diagnostics, not stable API: tracing targets, event and span names, their fields, and the settings the environment variable accepts may change in minor releases. Events are grouped under these subsystem targets:

  • aws_s3_transfer::concurrency: concurrency resolution and control.
  • aws_s3_transfer::scheduling: scheduler capacity decisions, worker pool growth, and memory-budget admission.
  • aws_s3_transfer::memory: buffer-pool admission, physical storage, and reclamation.
  • aws_s3_transfer::execution: per-work-item dispatch and completion.
  • aws_s3_transfer::transfer: transfer lifecycle and state transitions.

Other events use the target of the module that emits them.

AWS_S3_TM_DIAGNOSTICS enables optional diagnostic collection without changing the client configuration API. Settings are namespaced by subsystem and resolved once when a client configuration is built. A standalone memory::BufferPool resolves only the memory settings.

The value is a comma-separated list of case-sensitive key=value settings:

AWS_S3_TM_DIAGNOSTICS=memory.snapshot=1000ms,memory.detail=1,transfer.detail=1

Whitespace around entries, keys, and values is ignored. Later valid assignments replace earlier ones. Unknown keys are ignored for forward compatibility. A malformed recognized setting produces a warning and keeps the preceding value. Unsupported detail levels use the highest level understood by this version and produce a warning.

The environment variable controls collection cost. Tracing filters independently control whether collected records are emitted.

§Memory pool

Memory diagnostics report pool capacity, admission pressure, allocation fallback, preparation, and reclamation.

  • memory.snapshot=off disables periodic reports. A positive integer followed by ms enables reports; values below 100ms use 100ms.
  • memory.detail=0 uses counters updated only at pressure and lifecycle boundaries.
  • memory.detail=1 also counts every optimistic allocation attempt and bitmap word inspected, adding relaxed atomic updates to the acquisition path.

Memory snapshots use the aws_s3_transfer::memory tracing target at DEBUG:

AWS_S3_TM_DIAGNOSTICS=memory.snapshot=1000ms
RUST_LOG=aws_s3_transfer::memory=debug

Snapshot reporting reuses the memory pool’s maintenance thread. It does not create a diagnostics-only thread. Detailed counters and periodic reporting are independent; both are disabled by default.

§Transfer state machines

Transfer diagnostics report where an operation spends time and how work moves through its direction-specific state machine.

  • transfer.detail=0 disables optional transfer-state collection.
  • transfer.detail=1 collects an aggregate summary for each upload, download, upload_objects and download_objects transfer.
  • transfer.detail=2 also collects individual state transitions.

Transfer summaries use the operation’s tracing target at DEBUG; level-2 transitions use the same target at TRACE. Each operation logs under its own target below aws_s3_transfer::transfer, so the transfer target enables all of them:

AWS_S3_TM_DIAGNOSTICS=transfer.detail=1
RUST_LOG=aws_s3_transfer::transfer=debug

Re-exports§

pub use self::client::Client;
pub use self::config::Config;

Modules§

client
The transfer manager client and its operations.
config
Client configuration and loading: S3 settings, part size, concurrency, read-ahead, runtime and memory.
error
Errors returned by transfers, and how to classify them.
io
Upload bodies, custom part streams, and walkers for local directories and S3 buckets.
memory
Payload-memory configuration and shared pooled storage.
metrics
Transfer progress and I/O metrics, and client memory metrics.
operation
Inputs, builders, handles and outputs for each operation.
types
S3 values and response metadata, transfer policies and outcomes, and byte-size and throughput units.

Structs§

SchedulingCtl
Scheduling controls for a transfer.
ShutdownCompletion
Observes managed-runtime completion without retaining the client or owner.
ShutdownError
Failure to observe transfer-manager-owned runtime completion.

Enums§

ShutdownErrorKind
Classifies a failure to observe managed-runtime shutdown.

Functions§

from_env
Create a ConfigLoader that loads configuration from the environment.