s3pe

A small, dependable client and command line for S3-compatible object stores (AWS S3, Garage, MinIO, Ceph, …): nine verbs, SigV4 signing over the real payload hash, retries that know when to stop, and an integrity check on every transfer.

Website: https://strategicprojects.github.io/s3pe/rust/ · API reference: https://docs.rs/s3pe · all editions

Install

cargo add s3pe                       # the library (and the CLI as a build target)
cargo add s3pe --no-default-features # the library only, without clap
cargo install s3pe                   # the `s3pe` command

Rust 1.85 or later. TLS by rustls with bundled roots: no system OpenSSL.

Library

use s3pe::{read_s3, write_s3, list_s3};

// endpoint and credentials come from the environment
let manifest = read_s3("s3://my-bucket/data/manifest.json")?;
write_s3("s3://my-bucket/reports/figure1.png", std::fs::read("figure1.png")?, None)?;
for object in list_s3("s3://my-bucket/data/")? {
    println!("{:>10} {}", object.size, object.key);
}
# Ok::<(), Box<dyn std::error::Error>>(())

With an explicit client:

let s3 = s3pe::Client::builder()
    .endpoint("https://s3.example.org")
    .credentials("AKIA…", "…")
    .bucket("my-bucket")
    .build()?;
let bytes = s3.read("data/manifest.json")?;
s3.upload("sales.parquet", "etl/sales.parquet", None)?;      // streaming
s3.download("data/regions.geojson", "regions.geojson")?;     // streaming, .s3pe-part then rename
if let Some(meta) = s3.head("data/manifest.json")? {
    println!("{} bytes, {}", meta.size, meta.content_type.unwrap_or_default());
}
s3.delete("etl/old.parquet")?;
# Ok::<(), s3pe::Error>(())

Big files, copies, batches, folders, metadata, pre-signed URLs, sync

use s3pe::{PutOptions, SyncOptions};
use std::time::Duration;

s3.upload("backup.tar", "backups/", None)?;              // > 64 MiB: multipart, 4 parts at a time
s3.copy("etl/sales.parquet", "archive/sales-2026.parquet")?;
let keys: Vec<String> = s3.list("tmp/")?.into_iter().map(|o| o.key).collect();
s3.delete_many(&keys)?;                                   // DeleteObjects, 1000 per request
let level = s3.list_dir("")?;                             // level.objects, level.prefixes ("folders")
let opts = PutOptions {
    content_type: None,
    metadata: vec![("author".into(), "ana".into())],       // x-amz-meta-author
    headers: vec![("Cache-Control".into(), "max-age=3600".into())],
};
s3.write_with("reports/sales.csv", csv_bytes, &opts)?;
let url = s3.presign("reports/report.pdf", "GET", Duration::from_secs(3600))?;
let report = s3.sync_up("./out", "reports/", &SyncOptions { delete: false, last: Some("manifest.json".into()) })?;
s3.sync_down("reports/", "./mirror", &SyncOptions::default())?;
# Ok::<(), s3pe::Error>(())

The multipart knobs: .multipart_threshold(), .part_size(), .parallel() on the builder, or S3PE_MULTIPART_THRESHOLD, S3PE_PART_SIZE, S3PE_PARALLEL (defaults 64 MiB, 16 MiB, 4). headers are sent as given, so x-amz-server-side-encryption: AES256 turns on SSE where the store supports it. No ACLs and no versioning, by design.

Configuration

Path-style URLs always (https://host/bucket/key), which every store accepts. Keys are bare (relative to the bucket) or full s3://bucket/key URLs.

What happens in every call

What "dependable" means

• AWS Signature V4 over the real SHA-256 of the payload: no UNSIGNED-PAYLOAD, no aws-chunked (which some stores reject).
• Connection failures and HTTP 429/500/502/503/504 are retried twice, with a 1 s then 2 s pause. Other errors come back at once, typed: Error::NotFound, Error::AccessDenied, Error::Http { status, code, .. }, Error::Transport, Error::Config, Error::Integrity.
• After an upload the server's ETag is compared with the local MD5; after a download the byte count with Content-Length and, when the ETag is an MD5, the MD5 of what arrived.
• One line per transfer through the log facade (target s3pe): ↑ s3://my-bucket/etl/sales.parquet (12.3 MB, 0.8 s).

Command line

export S3PE_ENDPOINT=https://s3.example.org
export AWS_ACCESS_KEY_ID=...  AWS_SECRET_ACCESS_KEY=...

s3pe buckets
s3pe ls s3://my-bucket/data/              # keys
s3pe ls -l s3://my-bucket/data/           # date, size, etag, key
s3pe cat s3://my-bucket/data/manifest.json | jq .version
s3pe get s3://my-bucket/data/regions.geojson        # to ./regions.geojson
s3pe put sales.parquet s3://my-bucket/etl/           # keeps the file name
s3pe head s3://my-bucket/etl/sales.parquet
s3pe rm s3://my-bucket/etl/old.parquet s3://my-bucket/etl/older.parquet   # one request
s3pe ls -d s3://my-bucket/                            # one level: "folders" and files
s3pe cp s3://my-bucket/etl/sales.parquet s3://my-bucket/archive/sales-2026.parquet
s3pe put report.pdf s3://my-bucket/reports/ --metadata author=ana --header "Cache-Control: max-age=3600"
s3pe presign s3://my-bucket/reports/report.pdf --expires 3600
s3pe sync ./out s3://my-bucket/reports/ --last manifest.json   # upload what changed, manifest last
s3pe sync s3://my-bucket/reports/ ./mirror --delete            # download what changed
s3pe --endpoint http://127.0.0.1:9000 -b my-bucket ls data/   # a local MinIO or Garage

Exit codes: 0 ok, 2 not found, 3 access denied, 4 configuration, 1 anything else. -q or S3PE_QUIET=1 keeps stderr to errors.

Architecture

The family

The same package exists for Python (pip install s3pe), R (install.packages("s3pe")) and Go (go get github.com/StrategicProjects/s3pe), all in one repository, with one version and one changelog. The Python package is a binding on this crate.

Made in Pernambuco 🌵

s3pe is developed in Recife, Brazil. MIT license.