s3pe

A small, dependable client and command line for S3-compatible object stores (AWS S3, Garage, MinIO, Ceph, …): nine verbs, SigV4 signing over the real payload hash, retries that know when to stop, and an integrity check on every transfer.

Website: https://strategicprojects.github.io/s3pe/rust/ · API reference: https://docs.rs/s3pe · all editions

Install

cargo add s3pe                       # the library (and the CLI as a build target)
cargo add s3pe --no-default-features # the library only, without clap
cargo install s3pe                   # the `s3pe` command

Rust 1.85 or later. TLS by rustls with bundled roots: no system OpenSSL.

Library

use s3pe::{read_s3, write_s3, list_s3};

// endpoint and credentials come from the environment
let manifest = read_s3("s3://my-bucket/data/manifest.json")?;
write_s3("s3://my-bucket/reports/figure1.png", std::fs::read("figure1.png")?, None)?;
for object in list_s3("s3://my-bucket/data/")? {
    println!("{:>10} {}", object.size, object.key);
}
# Ok::<(), Box<dyn std::error::Error>>(())

With an explicit client:

let s3 = s3pe::Client::builder()
    .endpoint("https://s3.example.org")
    .credentials("AKIA…", "…")
    .bucket("my-bucket")
    .build()?;
let bytes = s3.read("data/manifest.json")?;
s3.upload("sales.parquet", "etl/sales.parquet", None)?;      // streaming
s3.download("data/regions.geojson", "regions.geojson")?;     // streaming, .s3pe-part then rename
if let Some(meta) = s3.head("data/manifest.json")? {
    println!("{} bytes, {}", meta.size, meta.content_type.unwrap_or_default());
}
s3.delete("etl/old.parquet")?;
# Ok::<(), s3pe::Error>(())

Configuration

Path-style URLs always (https://host/bucket/key), which every store accepts. Keys are bare (relative to the bucket) or full s3://bucket/key URLs.

What happens in every call

What "dependable" means

• AWS Signature V4 over the real SHA-256 of the payload: no UNSIGNED-PAYLOAD, no aws-chunked (which some stores reject).
• Connection failures and HTTP 429/500/502/503/504 are retried twice, with a 1 s then 2 s pause. Other errors come back at once, typed: Error::NotFound, Error::AccessDenied, Error::Http { status, code, .. }, Error::Transport, Error::Config, Error::Integrity.
• After an upload the server's ETag is compared with the local MD5; after a download the byte count with Content-Length and, when the ETag is an MD5, the MD5 of what arrived.
• One line per transfer through the log facade (target s3pe): ↑ s3://my-bucket/etl/sales.parquet (12.3 MB, 0.8 s).

Command line

export S3PE_ENDPOINT=https://s3.example.org
export AWS_ACCESS_KEY_ID=...  AWS_SECRET_ACCESS_KEY=...

s3pe buckets
s3pe ls s3://my-bucket/data/              # keys
s3pe ls -l s3://my-bucket/data/           # date, size, etag, key
s3pe cat s3://my-bucket/data/manifest.json | jq .version
s3pe get s3://my-bucket/data/regions.geojson        # to ./regions.geojson
s3pe put sales.parquet s3://my-bucket/etl/           # keeps the file name
s3pe head s3://my-bucket/etl/sales.parquet
s3pe rm s3://my-bucket/etl/old.parquet
s3pe --endpoint http://127.0.0.1:9000 -b my-bucket ls data/   # a local MinIO or Garage

Exit codes: 0 ok, 2 not found, 3 access denied, 4 configuration, 1 anything else. -q or S3PE_QUIET=1 keeps stderr to errors.

Architecture

The family

The same package exists for Python (pip install s3pe), R (install.packages("s3pe")) and Go (go get github.com/StrategicProjects/s3pe), all in one repository, with one version and one changelog. The Python package is a binding on this crate.

Made in Pernambuco 🌵

s3pe is developed in Recife, Brazil. MIT license.