garage-sdk 0.1.1

# Garage S3 Object Storage: Deployment Guide

This guide is for operators. It assumes you already run Garage and want a secure,
scalable setup for private uploads and public asset delivery.

If you only need the Rust SDK, start with the README:
[README.md](../README.md)

Highlights:
* **Garage v2.x** on **bare metal** (3 nodes)
* **Cloudflare → Envoy → Garage**
* **Uploads** via `s3.example.com` (apps sign requests normally)
* **Public assets** via `img.example.com` (browsers anonymous; **Envoy signs upstream** to Garage)
* Buckets/keys per app + one **read-only “public-cdn”** key for Envoy
* How to **add/remove nodes** safely

---

# 1) Architecture you’re building

### Hostnames

* `s3.example.com`
  Used by your apps/services to upload/download via S3 (SigV4). Cloudflare can front it, but your apps must sign requests.

* `img.example.com`
  Used by browsers and the public. Requests are anonymous. Envoy rewrites + signs them upstream using a special read-only key.

### Why two hostnames?

Garage does **not** support anonymous/public bucket access the way AWS does. If you hit Garage anonymously you get `AccessDenied`. So you must either:

* generate presigned URLs, or
* put a proxy in front that signs requests (Envoy signing = what you chose)

---

# 2) Prerequisites

## Nodes

* 3 servers (node1/node2/node3), each with stable storage for Garage data
* Time sync (NTP) recommended (SigV4 is time-sensitive)

## Ports you must allow

Between Garage nodes:

* **3901/tcp** (Garage RPC/cluster)

From Envoy to Garage:

* **3900/tcp** (S3 API)

Do **not** expose Garage admin widely:

* **3903/tcp** should remain local/private

### UFW example (tight)

On each Garage node, allow 3901 only from the other Garage nodes, and 3900 only from Envoy network/IP:

```bash
# node-to-node RPC
sudo ufw allow from <NODE1_IP> to any port 3901 proto tcp
sudo ufw allow from <NODE2_IP> to any port 3901 proto tcp
sudo ufw allow from <NODE3_IP> to any port 3901 proto tcp

# Envoy-to-Garage S3
sudo ufw allow from <ENVOY_IP_or_DOCKER_SUBNET> to any port 3900 proto tcp
```

---

# 3) Install Garage (bare metal)

## Option A: package manager (if your distro offers it)

Some distros have Garage packages.

## Option B: download Garage binary

1. Put the binary in:

* `/usr/local/bin/garage`

2. Make executable:

```bash
sudo chmod +x /usr/local/bin/garage
```

3. Confirm:

```bash
garage --version
```

---

# 4) Configure Garage on each node

Create directories:

```bash
sudo mkdir -p /var/lib/garage/meta /var/lib/garage/data
sudo chown -R root:root /var/lib/garage
sudo chmod 700 /var/lib/garage
```

## Create `/etc/garage.toml` (on every node)

Use the same `rpc_secret` on all nodes.

### Example (node1)

```toml
replication_factor = 3
consistency_mode = "consistent"

metadata_dir = "/var/lib/garage/meta"
data_dir = "/var/lib/garage/data"
db_engine = "lmdb"

# MUST be identical on all nodes
rpc_secret = "REPLACE_WITH_A_LONG_RANDOM_SECRET"

# Cluster RPC
rpc_bind_addr = "0.0.0.0:3901"
rpc_public_addr = "NODE1_PUBLIC_IP:3901"

[s3_api]
api_bind_addr = "0.0.0.0:3900"
s3_region = "garage"
# root_domain is only for vhost-style buckets. You're path-style, so it can be omitted.
# root_domain = ".s3.example.com"

[admin]
# keep private
api_bind_addr = "127.0.0.1:3903"
metrics_require_token = true
```

On node2/node3 change only:

```toml
rpc_public_addr = "NODE2_PUBLIC_IP:3901"
rpc_public_addr = "NODE3_PUBLIC_IP:3901"
```

---

# 5) Run Garage using systemd

Create `/etc/systemd/system/garage.service`:

```ini
[Unit]
Description=Garage Data Store
After=network-online.target
Wants=network-online.target

[Service]
Environment='RUST_LOG=garage=info' 'RUST_BACKTRACE=1'
ExecStart=/usr/local/bin/garage server
StateDirectory=garage
DynamicUser=true
ProtectHome=true
NoNewPrivileges=true
LimitNOFILE=42000

[Install]
WantedBy=multi-user.target
```

Enable/start:

```bash
sudo systemctl daemon-reload
sudo systemctl enable --now garage
sudo systemctl status garage --no-pager
```

Logs:

```bash
sudo journalctl -u garage -f
```

---

# 6) Form the 3-node cluster

On each node:

```bash
garage node id
```

Pick one node as the “hub” (node1). From node2 and node3:

```bash
garage node connect <NODE1_ID>@NODE1_PUBLIC_IP:3901
```

Check cluster:

```bash
garage status
```

You should see all three nodes as HEALTHY.

---

# 7) Assign layout (zones + capacity) and apply

This is how Garage knows where/what to store.

### Assign each node a distinct zone (recommended)

```bash
garage layout assign -z zone1 -c 500G <NODE1_ID_PREFIX>
garage layout assign -z zone2 -c 500G <NODE2_ID_PREFIX>
garage layout assign -z zone3 -c 500G <NODE3_ID_PREFIX>
```

Show layout:

```bash
garage layout show
```

Apply the next version (if current is 1, apply 2):

```bash
garage layout apply --version <CURRENT_VERSION_PLUS_1>
```

> Notes:
>
> * `-c 500G` is the **declared capacity** Garage uses for balancing. It does not “reserve” disk from other apps.
> * Zone labels matter for failure-domain placement.

---

# 8) Create buckets and keys (per-app)

Example bucket for one site/app:

```bash
garage bucket create site1-assets
```

Create an app key (for uploads):

```bash
garage key create site1-app
garage bucket allow --read --write --owner site1-assets --key site1-app
garage key info site1-app
```

Repeat for other apps:

* `site2-assets` + `site2-app`
* `site3-assets` + `site3-app`

### Important

* The **secret key is shown only once** at creation time. If you lose it, create a new key.

---

# 9) Public asset delivery (CDN-style URLs) with Envoy signing

## 9.1 Create a single Envoy “public-cdn” read key

```bash
garage key create public-cdn

garage bucket allow --read site1-assets --key public-cdn
garage bucket allow --read site2-assets       --key public-cdn
garage bucket allow --read site3-assets       --key public-cdn

garage key info public-cdn
```

This key is read-only and used only by Envoy to sign GET/HEAD upstream.

---

## 9.2 Envoy in Docker Compose: inject `public-cdn` credentials

In `docker-compose.yml`:

```yaml
services:
  envoy:
    image: envoyproxy/envoy:tools-v1.36-latest
    container_name: envoy
    restart: unless-stopped
    environment:
      AWS_ACCESS_KEY_ID: "GK..."
      AWS_SECRET_ACCESS_KEY: "..."
    volumes:
      - ./envoy.yaml:/etc/envoy/envoy.yaml:ro
      - ./certs:/etc/envoy/certs:ro
    ports:
      - "80:80"
      - "443:443"
      - "9901:9901"
```

**Critical:** after changing env vars you must **recreate** the container:

```bash
docker compose up -d --force-recreate --no-deps envoy
```

Verify creds exist in the running container:

```bash
docker inspect envoy --format '{{range .Config.Env}}{{println .}}{{end}}' | grep '^AWS_'
```

---

## 9.3 Envoy config: two different endpoints

### A) `s3.example.com` (no signing by Envoy)

Your apps sign their own requests. Envoy should just route.

### B) `img.example.com` (Envoy signs upstream)

* Map URL prefixes to buckets
* Rewrite paths from `/site1/<key>` → `/<bucket>/<key>`
* Sign with aws_request_signing upstream filter
* Disable path normalization/merge-slashes to avoid breaking signatures

### Cluster used for public reads (signed)

Add a cluster such as `garage_public` that includes the signing filter:

```yaml
clusters:
  - name: garage_public
    connect_timeout: 5s
    type: STRICT_DNS
    dns_lookup_family: V4_ONLY
    lb_policy: ROUND_ROBIN
    load_assignment:
      cluster_name: garage_public
      endpoints:
        - lb_endpoints:
            - endpoint: { address: { socket_address: { address: node1, port_value: 3900 } } }
            - endpoint: { address: { socket_address: { address: node2, port_value: 3900 } } }
            - endpoint: { address: { socket_address: { address: node3, port_value: 3900 } } }

    typed_extension_protocol_options:
      envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
        "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
        explicit_http_config:
          http_protocol_options: {}
        http_filters:
          - name: envoy.filters.http.aws_request_signing
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.aws_request_signing.v3.AwsRequestSigning
              service_name: s3
              region: garage
              use_unsigned_payload: true
              credential_provider:
                custom_credential_provider_chain: true
                environment_credential_provider: {}
          - name: envoy.filters.http.upstream_codec
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.upstream_codec.v3.UpstreamCodec
```

### Routes for public reads

Under the `img.example.com` listener/virtual host:

```yaml
routes:
  - match: { prefix: "/site1/" }
    route:
      cluster: garage_public
      prefix_rewrite: "/site1-assets/"
      timeout: 0s

  - match: { prefix: "/site2/" }
    route:
      cluster: garage_public
      prefix_rewrite: "/site2-assets/"
      timeout: 0s

  - match: { prefix: "/site3/" }
    route:
      cluster: garage_public
      prefix_rewrite: "/site3-assets/"
      timeout: 0s
```

### Expected public URL format

If your app uploads to bucket `site1-assets` with key `uploads/logo.png`, then the public URL is:

`https://img.example.com/site1/uploads/logo.png`

That URL can be embedded in websites, used in other CDNs, etc.

---

# 10) How apps upload and return the public URL

## Upload endpoint

Apps use S3 clients against:

* `https://s3.example.com`
* bucket: `site1-assets`
* key: e.g. `uploads/<uuid>.png`

Then **return**:

* `https://img.example.com/site1/<key>`

This keeps upload credentials private and gives you stable public URLs.

---

# 11) Put assets behind Cloudflare or other CDNs

### Cloudflare

* Keep `img.example.com` proxied (orange cloud) for caching + global edge.
* Consider adding Cache Rules for images (e.g. “Cache Everything” for `/site1/*` etc).
* If you use query strings for anything, decide whether to include them in cache key.

### Other CDNs

Any CDN can cache `img.example.com/...` because it’s normal HTTP GET/HEAD from the CDN’s perspective.

---

# 12) Adding a new node to the cluster (scale out)

1. Install Garage + same `rpc_secret`
2. Start service
3. Connect to an existing node:

   ```bash
   garage node connect <EXISTING_ID>@EXISTING_IP:3901
   ```
4. Assign layout:

   ```bash
   garage layout assign -z zone4 -c 500G <NEW_NODE_ID_PREFIX>
   ```
5. Apply next layout version:

   ```bash
   garage layout show
   garage layout apply --version <CURRENT+1>
   ```

Garage will rebalance progressively.

---

# 13) Removing a node (planned decommission)

1. Remove it from layout:

```bash
garage layout remove <OLD_NODE_ID_PREFIX>
garage layout show
garage layout apply --version <CURRENT+1>
```

2. Stop the daemon on that node once cluster is stable:

```bash
sudo systemctl stop garage
sudo systemctl disable garage
```

3. Remove it from Envoy upstream list if you hard-coded node IPs.

> Rule: ensure you still have enough nodes to satisfy `replication_factor=3` (i.e., at least 3 storage nodes) or reduce replication factor only with careful planning.

---

# 14) Troubleshooting checklist

### “AccessDenied: anonymous”

* You’re hitting Garage unsigned (wrong hostname/route/cluster)
* Envoy signing creds missing (in Docker Compose you must **recreate** container)
* Route points to wrong cluster (unsigned)

### “NoSuchKey”

* Signing works; object not uploaded at that key.

### One node timing out

* Firewall missing 3900/3901 rules
* Garage not listening/bound
* Disk full / IO stuck
* Remove that node from Envoy rotation until fixed

---

Back to the SDK overview: [README.md](../README.md)