Crate s3pe 

A small, dependable client for S3-compatible object stores (AWS S3, Garage, MinIO, Ceph, …): nine verbs, SigV4 signing, retries and integrity checks, plus the s3pe command line.

Nine verbs, one client, nothing else to learn (plus copy_s3, delete_many_s3, list_dir_s3, presign_s3, sync_up_s3 and sync_down_s3 since 0.3.0):

use s3pe::{read_s3, write_s3, list_s3};

// endpoint and credentials come from the environment (see below)
let manifest = read_s3("s3://my-bucket/data/manifest.json")?;
write_s3("s3://my-bucket/reports/figure1.png", std::fs::read("figure1.png")?, None)?;
for object in list_s3("s3://my-bucket/data/")? {
    println!("{} {}", object.size, object.key);
}

Or with an explicit Client:

let s3 = s3pe::Client::builder()
    .endpoint("https://s3.example.org")
    .credentials("AKIA…", "…")
    .bucket("my-bucket")
    .build()?;
let bytes = s3.read("data/manifest.json")?;
s3.upload("sales.parquet", "etl/sales.parquet", None)?;

What happens in every call

Configuration

Anything not passed to the builder comes from the environment:

Setting	Variables	Default
endpoint	S3PE_ENDPOINT, AWS_ENDPOINT_URL	required
region	AWS_REGION, AWS_DEFAULT_REGION	us-east-1
credentials	AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN	required
bucket	S3PE_BUCKET	none: use s3://bucket/key

URLs are always path-style (https://host/bucket/key).

What “dependable” means here

• Requests are signed with AWS Signature V4 over the real SHA-256 of the payload (no aws-chunked, which some stores, Garage among them, reject).
• Connection failures and HTTP 429/500/502/503/504 are retried twice, with a 1 s then 2 s pause. Other errors are reported at once, typed (Error::NotFound, Error::AccessDenied, …).
• After an upload the server’s ETag is compared with the local MD5; after a download the byte count is compared with Content-Length and, when the ETag is an MD5, with the MD5 of what arrived.
• Downloads to a file are written to a .s3pe-part file and renamed when complete.
• Bodies above 64 MiB go up as a multipart upload, 16 MiB parts, four at a time, aborted on failure; the final ETag is checked too.
• One line per transfer is logged through the log facade (target s3pe): ↑ s3://bucket/key (12.3 MB, 0.8 s).

Command line

cargo install s3pe also installs the s3pe command:

s3pe ls s3://my-bucket/data/
s3pe get s3://my-bucket/data/manifest.json
s3pe put sales.parquet s3://my-bucket/etl/
s3pe cat s3://my-bucket/data/manifest.json | jq .version
s3pe rm s3://my-bucket/etl/old.parquet s3://my-bucket/etl/older.parquet
s3pe cp s3://my-bucket/etl/sales.parquet s3://my-bucket/archive/sales-2026.parquet
s3pe presign s3://my-bucket/reports/report.pdf --expires 3600
s3pe sync ./out s3://my-bucket/reports/ --last manifest.json

The same package exists for Python (pip install s3pe), R (install.packages("s3pe")) and Go (go get github.com/StrategicProjects/s3pe).

Modules

sigv4

AWS Signature Version 4 for S3, header style.

Structs

Bucket

A bucket as listed by the store.

Client

A configured connection to one S3 endpoint.

ClientBuilder

Builds a Client; anything not set comes from the environment, then from the defaults (see the crate documentation for the variables).

Listing

One level of a bucket: the objects directly under a prefix and the common prefixes (“folders”) below it (D17).

Object

One object as listed or described by the store.

PutOptions

Options of write_with and upload_with (D17).

SyncOptions

Options of sync_up and sync_down (D18).

SyncReport

What a sync did.

Transfer

What a completed upload, download or copy reports.

Enums

Error

What can go wrong when talking to the store.

Constants

DEFAULT_MAX_TRIES

Attempts per call: the first and two retries.

DEFAULT_MULTIPART_THRESHOLD

Bodies larger than this go up as a multipart upload (D15).

DEFAULT_PARALLEL

Parts sent at the same time (D15).

DEFAULT_PART_SIZE

Size of each part of a multipart upload (D15).

DEFAULT_REGION

Signing region used when none is given.

DEFAULT_TIMEOUT

Seconds a single call may take before it is abandoned.

MAX_PRESIGN_SECS

Longest validity of a pre-signed URL (seven days).

MIN_PART_SIZE

Smallest part S3 accepts, except for the last one.

Functions

buckets_s3

The buckets the credentials can see, with a client built from the environment.

content_type_for

The content type a key’s extension suggests (application/octet-stream when unknown). The same table is used by every edition of s3pe.

copy_s3

Copies an object inside the store, with a client built from the environment.

delete_many_s3

Deletes several objects in batches, with a client built from the environment.

delete_s3

Deletes an object, with a client built from the environment.

download_s3

Downloads an object to a file, with a client built from the environment.

exists_s3

Whether an object exists, with a client built from the environment.

expected_etag

The ETag the store would give a local file uploaded by this client: its MD5, or the multipart ETag (D15) when it is above the threshold.

format_size

12.3 MB, 1.2 kB, 512 B (SI units, one decimal).

head_s3

Metadata of an object (None if absent), with a client built from the environment.

list_dir_s3

One level of a prefix (objects and “folders”), with a client built from the environment.

list_s3

Lists the objects under a prefix, with a client built from the environment.

multipart_etag

The ETag S3 gives a multipart upload: md5(concat(part MD5 bytes))-N.

presign_s3

A pre-signed URL, with a client built from the environment.

read_s3

Reads an object into memory, with a client built from the environment.

sync_down_s3

Downloads a prefix to a directory, with a client built from the environment.

sync_up_s3

Uploads a directory to a prefix, with a client built from the environment.

upload_s3

Uploads a file, with a client built from the environment.

write_s3

Writes bytes to an object, with a client built from the environment.

Type Aliases

Result

Result with this crate’s Error.