s3pe 0.3.0

Small, dependable S3 client and CLI for any S3-compatible store: read, write, list and delete objects with SigV4 signing, retries and integrity checks
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158

# s3pe

[![crates.io](https://img.shields.io/crates/v/s3pe.svg)](https://crates.io/crates/s3pe)
[![docs.rs](https://img.shields.io/docsrs/s3pe)](https://docs.rs/s3pe)
[![CI](https://github.com/StrategicProjects/s3pe/actions/workflows/ci.yml/badge.svg)](https://github.com/StrategicProjects/s3pe/actions/workflows/ci.yml)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.23606939.svg)](https://doi.org/10.5281/zenodo.23606939)

A small, dependable client and command line for S3-compatible object
stores (AWS S3, Garage, MinIO, Ceph, …): nine verbs, SigV4 signing over the
real payload hash, retries that know when to stop, and an integrity check
on every transfer.

**Website:** <https://strategicprojects.github.io/s3pe/rust/> ·
**API reference:** <https://docs.rs/s3pe> · [all editions](https://strategicprojects.github.io/s3pe/)

## Install

```bash
cargo add s3pe                       # the library (and the CLI as a build target)
cargo add s3pe --no-default-features # the library only, without clap
cargo install s3pe                   # the `s3pe` command
```

Rust 1.85 or later. TLS by rustls with bundled roots: no system OpenSSL.

## Library

```rust
use s3pe::{read_s3, write_s3, list_s3};

// endpoint and credentials come from the environment
let manifest = read_s3("s3://my-bucket/data/manifest.json")?;
write_s3("s3://my-bucket/reports/figure1.png", std::fs::read("figure1.png")?, None)?;
for object in list_s3("s3://my-bucket/data/")? {
    println!("{:>10} {}", object.size, object.key);
}
# Ok::<(), Box<dyn std::error::Error>>(())
```

With an explicit client:

```rust
let s3 = s3pe::Client::builder()
    .endpoint("https://s3.example.org")
    .credentials("AKIA…", "…")
    .bucket("my-bucket")
    .build()?;
let bytes = s3.read("data/manifest.json")?;
s3.upload("sales.parquet", "etl/sales.parquet", None)?;      // streaming
s3.download("data/regions.geojson", "regions.geojson")?;     // streaming, .s3pe-part then rename
if let Some(meta) = s3.head("data/manifest.json")? {
    println!("{} bytes, {}", meta.size, meta.content_type.unwrap_or_default());
}
s3.delete("etl/old.parquet")?;
# Ok::<(), s3pe::Error>(())
```

### Big files, copies, batches, folders, metadata, pre-signed URLs, sync

```rust
use s3pe::{PutOptions, SyncOptions};
use std::time::Duration;

s3.upload("backup.tar", "backups/", None)?;              // > 64 MiB: multipart, 4 parts at a time
s3.copy("etl/sales.parquet", "archive/sales-2026.parquet")?;
let keys: Vec<String> = s3.list("tmp/")?.into_iter().map(|o| o.key).collect();
s3.delete_many(&keys)?;                                   // DeleteObjects, 1000 per request
let level = s3.list_dir("")?;                             // level.objects, level.prefixes ("folders")
let opts = PutOptions {
    content_type: None,
    metadata: vec![("author".into(), "ana".into())],       // x-amz-meta-author
    headers: vec![("Cache-Control".into(), "max-age=3600".into())],
};
s3.write_with("reports/sales.csv", csv_bytes, &opts)?;
let url = s3.presign("reports/report.pdf", "GET", Duration::from_secs(3600))?;
let report = s3.sync_up("./out", "reports/", &SyncOptions { delete: false, last: Some("manifest.json".into()) })?;
s3.sync_down("reports/", "./mirror", &SyncOptions::default())?;
# Ok::<(), s3pe::Error>(())
```

The multipart knobs: `.multipart_threshold()`, `.part_size()`, `.parallel()`
on the builder, or `S3PE_MULTIPART_THRESHOLD`, `S3PE_PART_SIZE`,
`S3PE_PARALLEL` (defaults 64 MiB, 16 MiB, 4). `headers` are sent as given,
so `x-amz-server-side-encryption: AES256` turns on SSE where the store
supports it. No ACLs and no versioning, by design.

### Configuration

| Setting | Environment | Default |
|---|---|---|
| endpoint | `S3PE_ENDPOINT`, then `AWS_ENDPOINT_URL` | required (`https://host[:port]`) |
| region | `AWS_REGION`, then `AWS_DEFAULT_REGION` | `us-east-1` |
| credentials | `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` (`AWS_SESSION_TOKEN`) | required |
| bucket | `S3PE_BUCKET` | none: use `s3://bucket/key` |

Path-style URLs always (`https://host/bucket/key`), which every store
accepts. Keys are bare (relative to the bucket) or full `s3://bucket/key`
URLs.

### What happens in every call

<img src="https://raw.githubusercontent.com/StrategicProjects/s3pe/main/site/assets/request-flow.svg" alt="Resolve the key, hash the payload, sign with SigV4, send, retry on 429 and 5xx, verify the ETag and Content-Length; success logs one line, errors are typed." width="100%" />

### What "dependable" means

- AWS Signature V4 over the real SHA-256 of the payload: no
  `UNSIGNED-PAYLOAD`, no `aws-chunked` (which some stores reject).
- Connection failures and HTTP 429/500/502/503/504 are retried twice, with
  a 1 s then 2 s pause. Other errors come back at once, typed:
  `Error::NotFound`, `Error::AccessDenied`, `Error::Http { status, code, .. }`,
  `Error::Transport`, `Error::Config`, `Error::Integrity`.
- After an upload the server's ETag is compared with the local MD5; after a
  download the byte count with `Content-Length` and, when the ETag is an
  MD5, the MD5 of what arrived.
- One line per transfer through the [`log`](https://docs.rs/log) facade
  (target `s3pe`): `↑ s3://my-bucket/etl/sales.parquet (12.3 MB, 0.8 s)`.

## Command line

```bash
export S3PE_ENDPOINT=https://s3.example.org
export AWS_ACCESS_KEY_ID=...  AWS_SECRET_ACCESS_KEY=...

s3pe buckets
s3pe ls s3://my-bucket/data/              # keys
s3pe ls -l s3://my-bucket/data/           # date, size, etag, key
s3pe cat s3://my-bucket/data/manifest.json | jq .version
s3pe get s3://my-bucket/data/regions.geojson        # to ./regions.geojson
s3pe put sales.parquet s3://my-bucket/etl/           # keeps the file name
s3pe head s3://my-bucket/etl/sales.parquet
s3pe rm s3://my-bucket/etl/old.parquet s3://my-bucket/etl/older.parquet   # one request
s3pe ls -d s3://my-bucket/                            # one level: "folders" and files
s3pe cp s3://my-bucket/etl/sales.parquet s3://my-bucket/archive/sales-2026.parquet
s3pe put report.pdf s3://my-bucket/reports/ --metadata author=ana --header "Cache-Control: max-age=3600"
s3pe presign s3://my-bucket/reports/report.pdf --expires 3600
s3pe sync ./out s3://my-bucket/reports/ --last manifest.json   # upload what changed, manifest last
s3pe sync s3://my-bucket/reports/ ./mirror --delete            # download what changed
s3pe --endpoint http://127.0.0.1:9000 -b my-bucket ls data/   # a local MinIO or Garage
```

Exit codes: 0 ok, 2 not found, 3 access denied, 4 configuration, 1 anything
else. `-q` or `S3PE_QUIET=1` keeps stderr to errors.

## Architecture

<img src="https://raw.githubusercontent.com/StrategicProjects/s3pe/main/site/assets/architecture.svg" alt="The Rust crate (ureq, rustls, hmac, sha2, plus the s3pe command line) is one of four editions that share one specification; the Python package is a binding on it. All talk to any S3-compatible store." width="100%" />

## The family

The same package exists for Python (`pip install s3pe`), R
(`install.packages("s3pe")`) and Go (`go get github.com/StrategicProjects/s3pe`),
all in [one repository](https://github.com/StrategicProjects/s3pe), with one
version and one changelog. The Python package is a binding on this crate.

## Made in Pernambuco 🌵

s3pe is developed in Recife, Brazil. MIT license.