---
title: File Uploads and Storage
description: Receiving files from multipart forms, keeping them on disk or in S3, attaching images with resized styles to records, serving them, and testing uploads.
category: Going Further
order: 4
---
Glacier handles file uploads in three layers. A controller reads the files
in a `multipart/form-data` request with `file` and `files`. A
`storage::Store` keeps files in a folder or in an S3 bucket. An
`attachment::Attachment` ties a file to a record: the original plus
resized copies of an image, under paths built from a pattern, described by
four columns on the record. There's no way yet to declare an attachment in
a model: you write the few model methods that join these pieces, and this
guide shows them.
After reading this guide, you will know:
- How to read uploaded files in an action, and which of their details to
trust.
- How to limit the size of uploads.
- How to write an upload form, and a problem with browser upload forms
that Glacier hasn't fixed yet.
- How to keep files on disk or in S3, and how presigned S3 URLs work.
- How to attach an image to a model, with resized styles.
- How to serve stored files, publicly or through the app.
- How to test uploads.
- What Glacier's file handling doesn't do yet.
The examples come from `uploads`, an app made with `glacier new uploads`
and a `Photo` scaffold whose photos each have an image.
## Receiving Uploads
### Multipart Forms
A browser sends a form with a file field as `multipart/form-data`, a body
made of parts, one per field. Glacier reads such a body in two ways:
- Its text fields become params, as a URL-encoded form's do, so
`params` and `Photo.build(params)` work unchanged.
- Its files are left out of the params and read with `file(name)` and
`files`.
### `file` and `files`
`file("image")` returns the file sent in the field named `image`, as an
`Option<Upload>`: `None` if the request isn't multipart or has no such
file. The name is matched exactly as the form sent it, so a field named
`photo[image]` is read with `file("photo[image]")`. If several files
share the name, it returns the first.
`files` returns every file in the request, in the order sent, as
`(field name, Upload)` pairs. The `uploads` app uses it to add a photo for
each file chosen in an `<input type="file" multiple>`:
```html
<form method="post" action="{{ bulk_photos_path() }}" enctype="multipart/form-data">
{{ csrf_field() }}
<input type="file" name="images[]" multiple accept="image/png,image/jpeg,image/gif,image/webp">
<button>Add photos</button>
</form>
```
```glacier
# Adds a photo for every file sent, whatever its field name.
def bulk do
added = 0
for (_field, upload) in files do
photo = Photo.new()
photo.title = upload.filename.clone()
photo.image = Some(upload)
if photo.save()! do
added = added + 1
end
end
redirect Photo.collection_path(), notice: "#{added} photos added."
end
```
An `Upload` (`glacier::upload::Upload`) has three public fields and four
methods:
| `filename` | The file's name as the client sent it, cut short at any `;` (see below) |
| `content_type` | The `Content-Type` the client gave the part, or `application/octet-stream` if it gave none |
| `bytes` | The file's contents, as `Bytes` |
| `len()`, `is_empty()` | The size of `bytes` |
| `detected_type()` | The type the bytes show (see [Which Details to Trust](#which-details-to-trust)) |
| `is_spoofed()` | Whether the file claims to be something it isn't |
A browser sends a file field left empty as a part with an empty file name
and no content. Glacier leaves such a part out, so an empty field reads as
no file at all.
The multipart parser splits a part's `Content-Disposition` header at every
`;`, even inside a quoted name, so a file sent as `a;b.png` arrives with the
`filename` `a`. It reads only the plain `filename=` parameter and ignores
`filename*=`, the form some clients use for non-ASCII names. The `uploads`
app's validations then refuse the name, since it has lost its extension:
```glacier
test "cuts a file name at a semicolon" do
res = post "/photos", [title: "Semi", image: glacier::testing::file("a;b.png", "image/png", png(10, 10))]
assert res.status == :unprocessable_entity
assert res.body =~ "Image must be named .png, .jpg, .jpeg, .gif or .webp"
end
```
### Which Details to Trust
`filename` and `content_type` are whatever the client chose to send, and
Glacier doesn't clean up the name. Treat both as untrusted input, and use
the two checks an `Upload` makes from its bytes, as Rails' Paperclip does:
- `detected_type()` is the type the file's first bytes name
(`image/jpeg` for a JPEG called `cat.png`). For bytes without a known
signature it falls back to the extension's type, then to `text/plain`
for text and `application/octet-stream` for anything else. Attachments
record this type, not the declared one.
- `is_spoofed()` is true when the declared kind (`image` of `image/png`)
disagrees with the extension's or with the bytes: text sent as
`image/png`, or a photo called `report.pdf`. It knows common formats
only, so bytes it can't name and extensions it doesn't know never count
against a file.
Then:
- Never build a file system path from `filename` yourself. The attachment
code in this guide cleans the name (see
[Storing an Upload](#storing-an-upload)), and the disk store refuses
paths that climb out of its folder.
- Check the name's extension and the type against a list of the ones you
accept, as the [validations below](#validating-uploads) do. Files on disk
are [served](#serving-stored-files) with a `Content-Type` chosen from
their extension, so the extension decides how a browser treats them.
- To know what a file really contains, read its bytes. Making an image's
styles reads them, but only for an attachment with at least one style,
and it proves only that the bytes decode as a JPEG, PNG, GIF or WebP
image. The format is recognised from the file's first few bytes, not
from its name or type; anything appended after a valid image isn't
noticed; and the original is stored unchanged, whatever else it
contains.
### Size Limits
The server reads a request's whole body into memory before the app sees
it, and refuses a body larger than `server.max_body_size` bytes with `413
Payload Too Large`. The limit is 2 MiB unless you change it, and it covers
the whole request: every file in it, plus the other fields. The `uploads`
app raises it in `config/default.toml`:
```toml
# The largest request body accepted, uploads included: 10 MiB.
max_body_size = 10485760
```
`App::max_body_size(bytes)` sets it in code instead. With this setting,
the app, running locally (on port 4517 here), accepted a 3,309,623-byte PNG sent with
`curl` (with the CSRF token in a header, as [An Upload Form](#an-upload-form)
explains) and refused an 11,064,558-byte one before any CSRF check:
```console
$ curl -s -w ' %{http_code}\n' -F title=Large -F "image=@large.png;type=image/png" http://127.0.0.1:4517/photos
Payload Too Large 413
```
There is one limit for the whole app, not one per route or per field. It
bounds the size of the raw body, not the memory a request can use. On a
multipart request, every call to `params`, `file` or `files`
parses the whole body again and copies every file in it into new `Bytes`
(`params()` copies the files and then drops them). So an action that calls
`params()` and then `file()`, as this app's do, holds the body and two
copies of each file at once; read the files once and keep them.
Making an image's styles can take far more memory than the file itself, as
[Styles and Resizing](#styles-and-resizing) explains.
NOTE: The limit is applied where the server reads the request from the
network. The test client hands requests to the app directly, so a test can
send a body larger than the limit without a 413.
## An Upload Form (updated)
A form that sends files needs `enctype="multipart/form-data"`, and an
`<input type="file">`. Glacier's form helpers, `form_for` and `form_tag`,
have no option for the `enctype` and there is no file field helper, so the
`uploads` app writes its `<form>` tag by hand, with `csrf_field()` for the
CSRF token and a `_method` field for the edit form. `photo.form_action()`
gives the action and method that `form_for` would have used:
```html
<form method="post" action="{{ action }}" enctype="multipart/form-data">
{{ csrf_field() }}
{%- if method == "patch" %}
<input type="hidden" name="_method" value="patch">
{%- end %}
<p>
<label>Title<br><input name="title" value="{{ photo.text("title") }}"></label>
{%- for message in photo.errors_on("title") %}
<small class="error">Title {{ message }}</small>
{%- end %}
</p>
<p>
<label>Image<br><input type="file" name="image" accept="image/png,image/jpeg,image/gif,image/webp"></label>
{%- for message in photo.errors_on("image") %}
<small class="error">Image {{ message }}</small>
{%- end %}
</p>
<button>Save</button>
</form>
```
The `accept` attribute only filters what the browser's file picker shows.
It's no check at all on what reaches the server.
The CSRF check and `MethodOverride` read `_csrf_token` and `_method` from
a multipart body as from a URL-encoded one, so a browser submits this
form, edit form included, as it is.
## Storing Files (updated)
### `storage::Store`
`glacier::storage::Store` keeps files under string keys such as
`photos/1/thumb/desk.png`, in one of two places:
- `Store::Disk(folder)`: files in a folder on the server.
- `Store::S3(signer)`: objects in an S3 bucket, or a service that speaks
S3's API, such as MinIO.
Both have the same methods:
| `put(key, bytes, content_type)!` | Saves the bytes as `key`, replacing any file already there |
| `get(key)!` | The file's bytes, or `None` if there's no such file |
| `delete(key)!` | Removes the file; removing a missing file isn't an error |
| `exists(key)!` | Whether there's a file saved as `key` |
| `size(key)!` | Its size in bytes, or `None` |
| `list(prefix)!` | The keys of every file under `prefix` (`""` for all), sorted |
| `copy(from, to)!` | Copies a file within the store |
| `rename(from, to)!` | Moves a file within the store (`move` is a Rust keyword) |
| `url(key, expires_in)` | A URL a browser can fetch the file from (see below) |
A test in the `uploads` app uses a disk store directly:
```glacier
test "keeps files in a folder" do
store = Store::Disk("tmp/storage".into())
store.put("notes/hello.txt", "Hello".into(), "text/plain")!
assert store.get("notes/hello.txt")! == Some("Hello".into())
assert store.url("notes/hello.txt", 300) == "/uploads/notes/hello.txt"
store.delete("notes/hello.txt")!
assert store.get("notes/hello.txt")! == nil
store.delete("notes/hello.txt")!
end
```
and lists, copies and moves files:
```glacier
test "lists, copies, moves and measures files" do
store = Store::Disk("tmp/storage/more".into())
store.put("reports/2026/q1.txt", "first".into(), "text/plain")!
store.put("reports/2026/q2.txt", "second".into(), "text/plain")!
assert store.list("reports/2026/")! == ["reports/2026/q1.txt", "reports/2026/q2.txt"]
assert store.exists("reports/2026/q1.txt")!
assert store.size("reports/2026/q2.txt")! == Some(6)
store.copy("reports/2026/q1.txt", "archive/q1.txt")!
store.rename("reports/2026/q2.txt", "archive/q2.txt")!
assert store.list("archive/")! == ["archive/q1.txt", "archive/q2.txt"]
refute store.exists("reports/2026/q2.txt")!
end
```
On S3, `list` pages through the bucket's listing, `size` asks for the
object's headers only, and `rename` is a copy, then a delete.
A disk store creates folders as needed, and keeps every key inside its
folder: leading slashes are dropped, and a key with a `..` segment is
refused with an error (`400 Bad Request` if it reaches a response):
```glacier
test "refuses keys that climb out of the folder" do
store = Store::Disk("tmp/storage".into())
assert store.put("../escaped.txt", "no".into(), "text/plain").await.is_err()
end
```
`content_type` is only used by S3, which stores it with the object and
sends it back as the object's `Content-Type`. A disk store writes just the
bytes.
### Choosing a Store
Glacier reads the app's store from `[storage]` in the configuration, and
`glacier::storage::current()` gives it: an S3 bucket when `bucket` is set
(with the three `AWS_*` variables below, and `endpoint` for MinIO and the
like), otherwise the folder in `root`, `storage` if the config names
neither. Attachments keep their files there:
```toml
# Uploaded files: a folder, relative to where the server runs. Set
# `bucket` (and AWS_REGION, AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in
# the environment, and `endpoint` for MinIO) to keep them in S3 instead.
[storage]
root = "storage"
```
The tests keep their files out of the way, in a folder the app's
`.gitignore` lists along with `storage/`:
```toml
# Files stored by tests go here (git-ignored), not into storage/.
[storage]
root = "tmp/storage"
```
`storage()` is the app's store in `.glacier`.
A `bucket` without the `AWS_*` variables stops the app at start-up.
`storage::set_current(store)` sets the store from code instead, and
`storage::from_config(&config)` reads the section without setting it.
### Named Stores (new)
An app can keep some files elsewhere: avatars in a public bucket, say,
and the rest in a private one. `[storage.<name>]` describes another
store, with the same `bucket`, `endpoint` and `root` settings (a folder
under `storage/` named after it by default):
```toml
# Another store, `glacier::storage::named("avatars")`.
[storage.avatars]
root = "tmp/avatars"
```
`glacier::storage::named("avatars")?` gives it:
```glacier
test "the app's store, and named ones from the config" do
assert storage().url("a.png", 60) == "/uploads/a.png"
avatars = glacier::storage::named("avatars")?
avatars.put("ada.png", "png".into(), "image/png")!
assert std::path::Path.new("tmp/avatars/ada.png").exists()
end
```
### Files on Disk
For a disk store, `url(key, …)` is `/uploads/` followed by the key, such as
`/uploads/photos/1/medium/desk.png`, whatever the folder is called. It
ignores `expires_in`, and the key isn't percent-encoded. Glacier doesn't
serve that path by itself: the app has to, and the `uploads` app does it
with the `Static` middleware, pointed at the store's folder, in its
settings (`src/app.glacier`):
```glacier
# Before the logger, so logs show the overridden method.
middleware MethodOverride
middleware Logger
# Files kept on disk (`[storage] root`) are served from /uploads,
# where their URLs point.
if let Store.Disk(root) = glacier::storage::current() do
middleware Static.new(root).at("/uploads")
end
```
[Serving Stored Files](#serving-stored-files) covers what that means for
who can see them.
### Files in S3
`glacier::storage::S3Signer` holds a bucket, its region and an access key,
and signs requests with AWS Signature Version 4. `Store::S3` uses it to
`PUT`, `GET` and `DELETE` objects, over HTTPS unless an endpoint says
otherwise, sending each file in one request. There are three ways to make one:
| `S3Signer::new(bucket, region, access_key_id, secret_access_key)` | For `https://<bucket>.s3.<region>.amazonaws.com` |
| `S3Signer::from_env(bucket)` | The same, from `AWS_REGION`, `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`; `None` if any of them is missing or empty |
| `.with_endpoint("http://localhost:9000")` | Another S3-compatible service, such as MinIO, with `/bucket/key` paths |
`.with_host(host)` replaces the host name: requests are sent to it, it's
the host that's signed, and presigned URLs point at it. On a signer made
by `new` or `from_env`, the bucket isn't put in the path, so the host has
to name the bucket itself, as `examplebucket.s3.amazonaws.com` does. `from_env` reads only those three variables: not
`AWS_SESSION_TOKEN`, `AWS_DEFAULT_REGION`, `~/.aws` files or instance roles,
so temporary credentials don't work yet.
A `404` from S3 makes `get` return `None`. Any other failure, from an
unreachable service or a non-success status, is an error that answers
`500` if it reaches the response. Glacier sets no timeout on S3 requests,
so an endpoint that accepts the connection and then stops answering holds
up the request that's waiting on it for as long as it hangs. Glacier sends
no ACL with an upload, so who can read an object is up to the bucket's own
settings.
WARNING: The S3 store treats a `404` as "not found" for every request, not
only `get`. S3 answers `404` (`NoSuchBucket`) when the bucket doesn't
exist, so a `put` to a misspelt bucket name returns without an error and
stores nothing. Check the bucket name, or read a file back once after
deploying.
NOTE: The `uploads` app's tests don't talk to AWS or MinIO, so the S3 store
hasn't been run against a real bucket for this guide. What the tests check
is the signing, which runs locally: Glacier's own tests check its signed
request headers against AWS's published example, and the app checks a
presigned URL (below) the same way.
### Presigned URLs (updated)
For an S3 store, `url(key, expires_in)` is a presigned URL: a `GET` URL for
the object that works for `expires_in` seconds without credentials, so a
private object can be shown in an `<img>` or linked to directly.
`S3Signer::presigned_url(key, expires_in)` makes one from the signer, and
`presigned_url_at(key, expires_in, time)` signs as at a given time, which
lets a test compare the result with the URL AWS's documentation gives for
the same request:
```glacier
test "signs a URL as AWS's documented example does" do
s3 = glacier::storage::S3Signer::new("examplebucket", "us-east-1", "AKIAIOSFODNN7EXAMPLE", "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY").with_host("examplebucket.s3.amazonaws.com")
at = "2013-05-24T00:00:00Z".parse::<Timestamp>().unwrap()
url = s3.presigned_url_at("test.txt", 86400, at)
assert url == "https://examplebucket.s3.amazonaws.com/test.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIOSFODNN7EXAMPLE%2F20130524%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20130524T000000Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=aeeed9bbccd4d02ee5c0109b86d86835f995330da4c265957d157751f604d404"
end
```
```glacier
test "gives S3 files presigned URLs" do
s3 = glacier::storage::S3Signer::new("my-bucket", "eu-west-2", "AKIA…", "secret")
store = Store::S3(s3)
url = store.url("photos/1/thumb/desk.png", 300)
assert url =~ "https://my-bucket.s3.eu-west-2.amazonaws.com/photos/1/thumb/desk.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&"
assert url =~ "&X-Amz-Expires=300&"
end
```
```glacier
test "uses /bucket/key paths for another endpoint" do
s3 = glacier::storage::S3Signer::new("photos", "us-east-1", "minioadmin", "minioadmin").with_endpoint("http://localhost:9000")
assert s3.presigned_url("1/thumb/desk.png", 60) =~ "http://localhost:9000/photos/1/thumb/desk.png?"
end
```
A presigned URL can't be withdrawn early: anyone who has it can fetch the
object until it expires. AWS refuses an `X-Amz-Expires` longer than seven
days (604,800 seconds); Glacier doesn't check the value.
`S3Signer::presigned_upload_url(key, expires_in)` presigns an upload
instead: a URL the browser can `PUT` a file to, so a large file goes
straight to S3 without passing through the app. The app then records
the key it chose; nothing checks what was uploaded there.
### Direct Uploads from a Form (new)
`S3Signer::presigned_post(key, policy)` presigns a form instead, with
limits S3 enforces: how long it works, the file's size and type, and
where the browser goes after uploading. A key ending `${filename}` lets
the upload keep the file's own name under that prefix:
```glacier
test "presigns a form that uploads straight to S3" do
s3 = glacier::storage::S3Signer::new("my-bucket", "eu-west-2", "AKIA…", "secret")
policy = glacier::storage::PostPolicy::new(600).max_size(5_000_000).content_type("image/png").redirect("https://app.example/avatars/uploaded")
upload = s3.presigned_post("avatars/7/${filename}", &policy)
assert upload.url == "https://my-bucket.s3.eu-west-2.amazonaws.com/"
names = upload.fields.iter().map(fn (name, _) -> name.as_str() end).collect::<Vec<_>>()
assert names == ["key", "Content-Type", "success_action_redirect", "x-amz-algorithm", "x-amz-credential", "x-amz-date", "policy", "x-amz-signature"]
end
```
The result has the form's `url` and its hidden `fields`, in order. A
plain HTML form sends the file with no JavaScript, the file last:
```html
<form action="{{ upload.url }}" method="post" enctype="multipart/form-data">
{% for (name, value) in upload.fields.iter() %}
<input type="hidden" name="{{ name }}" value="{{ value }}">
{% end %}
<input type="file" name="file">
<button>Upload</button>
</form>
```
S3 checks the upload against the policy and refuses one that's too big,
of another type, or under another key. With `redirect(url)`, it then
sends the browser to `url` with `bucket`, `key` and `etag` in the query,
so the app can record the key. `PostPolicy` takes `max_size(bytes)` or
`size_between(min, max)`, and `content_type(type)` (sent as a field) or
`content_type_starts_with("image/")`, for which the page must set the
form's `Content-Type` field to the file's type itself. S3 checks that
field, not the file's bytes, so check what arrived before trusting it.
## Attachments
`has_attached :image` in a model keeps an uploaded file for each record:
the original, plus a resized copy for each style, in the app's store at
keys built from a pattern, and four columns on the record describing the
file. Give it an upload and save the record; Glacier does the rest.
### The Columns
For an attachment called `image`, the columns are `image_file_name`,
`image_content_type`, `image_file_size` and `image_updated_at`. They're
the names the Paperclip library gives its columns, so a table that already
has them can be used without renaming them, as long as their types suit
(`image_file_size` is read as an `i32`). Its stored files are another
matter: the path pattern below has no placeholder that splits the id into
folders such as `000/000/007`, as that library's default path does, so
files kept under such paths would have to be moved.
The `uploads` app made the scaffold and a migration to add the columns.
The output below is from running the same commands again in a fresh app,
so the timestamps in the migration names differ from the app's own files:
```console
$ glacier generate scaffold Photo title:string
create db/migrations/m20261004131149_create_photos.glacier
create src/models/photo.glacier
create tests/factories/photo.glacier
create tests/models/photo.test.glacier
update tests/glacier.rs
create src/controllers/photos_controller.glacier
create src/views/photos/index.html
create src/views/photos/show.html
create src/views/photos/new.html
create src/views/photos/edit.html
create src/views/photos/_form.html
create tests/controllers/photos_controller.test.glacier
update tests/glacier.rs
update src/routes.glacier
Next: glacier db migrate, then visit /photos
$ glacier generate migration add_image_to_photos
create db/migrations/m20261004131150_add_image_to_photos.glacier
```
`attachment :image` in a migration adds the four columns (and `change_table`
removes them again when it's undone; see [Migrations](migrations.html)):
```glacier
migration do
change_table :photos do
# image_file_name, image_content_type, image_file_size, image_updated_at
attachment :image
end
end
```
### Declaring an Attachment
```glacier
# An image, kept with a resized copy per style: `image_file_name` and the
# other columns describe it, and `photo.image = file("image")` gives the
# upload to keep when the photo is saved.
has_attached :image, styles: [medium: "640x480", thumb: "100x100"]
```
`has_attached :image` gives the model the four columns as fields, all
`protected` (a request can't set them through params) and optional (a
photo may have no image), and one more, `image`, an
`Option<Upload>` that isn't stored: the upload to keep when the record is
next saved. It takes these options:
| `styles: [medium: "640x480", thumb: "100x100"]` | A resized copy per style, each scaled to fit its box (Paperclip's `>` and the like are accepted and ignored) |
| `content_types: ["image/png", "image/jpeg"]` | The types it may be, by what the bytes show; anything else is "is invalid" |
| `path: ":class/:id/:style/:filename"` | Where its files go, the default shown |
The model's `Photo.IMAGE` is the attachment (a
`glacier::attachment::Attachment`, a path pattern and the styles), whose
path pattern can use four placeholders, replaced as plain text:
| `:class` | The model's table, such as `photos` |
| `:id` | The record's id |
| `:style` | `original`, or a style's name |
| `:filename` | The stored file name |
`Photo.IMAGE.path("photos", 7, "thumb", "desk.png")` is
`photos/7/thumb/desk.png`, and `photo.image_path("thumb")` is that key for
a photo's own image. Keep `:id` and `:style` in a pattern, or different
records or styles will share keys.
### Storing an Upload
Set `image` and save:
```glacier
def create do
photo = Photo.build(params)
photo.image = file("image")
if photo.save()! do
redirect photo, notice: "Photo created."
else
render :new, photo, title: "New photo", status: 422
end
end
```
When a record with an upload in `image` is saved:
1. The columns are set from the upload: the file name, cleaned with
`glacier::attachment::sanitize_filename` (each of
`& $ + , / : ; = ? @ < > [ ] { } | \ ^ ~ % #` and the space becomes
`_`; other characters, including quotes, parentheses and non-ASCII
letters, are kept), the type the bytes show (not the one declared),
the size and the time.
2. It's validated: a file whose contents aren't what its type or
extension say is refused with "has contents that are not what they are
reported to be", as Paperclip's spoof check does, and one outside
`content_types` with "is invalid". The model's own validations see the
new columns too.
3. The row is written, then the original is put in the store unchanged at
the `original` style's key, then each style is made and put in turn.
The previous image's files are removed after that, if its name was
different (a new file with the same name has already been written over
them).
Step 3's writes happen in one transaction (a savepoint inside one already
open): if storing a file fails, the row is left as it was, the record is
restored to how it was before `save`, and the error is returned.
```glacier
test "creates a photo with an image" do
bytes = png(1600, 1200)
res = post "/photos", [title: "Shelf", image: glacier::testing::file("shelf.png", "image/png", bytes.clone())]
assert res.status == :see_other
photo = Photo.find_by(title: "Shelf")!.unwrap()
assert photo.image_file_name == Some("shelf.png".to_string())
assert photo.image_content_type == Some("image/png".to_string())
assert photo.image_file_size == Some(bytes.len() as i32)
assert photo.image_updated_at.is_some()
end
```
```glacier
test "makes the file name safe for paths and URLs" do
post "/photos", [title: "Hash", image: glacier::testing::file("my photo #1.png", "image/png", png(10, 10))]
photo = Photo.find_by(title: "Hash")!.unwrap()
assert photo.image_file_name == Some("my_photo__1.png".to_string())
end
```
WARNING: The store isn't transactional. Files are put one by one and
storing stops at the first failure, leaving whatever it has already
written: a file that can't be decoded as an image fails with `422
Unprocessable Entity` after the original is stored, and a `put` that fails
part-way through the styles, such as an S3 error, leaves some styles
behind. The row still describes the previous image, but if the new file
has the same name, the files written so far have replaced the old ones,
leaving a mix of the two. If removing the previous files fails, the save
fails too, after the new files are stored. Validating the type first, as
below, keeps out a file that claims to be a PNG and isn't:
```glacier
test "refuses a file that claims to be an image and isn't" do
res = post "/photos", [title: "Fake", image: glacier::testing::file("fake.png", "image/png", "not a png")]
assert res.status == :unprocessable_entity
assert res.body =~ "Image has contents that are not what they are reported to be"
assert Photo.find_by(title: "Fake")!.is_none()
end
test "keeps the type the bytes show, not the one declared" do
res = post "/photos", [title: "Renamed", image: glacier::testing::file("renamed.png", "image/png", encoded(20, 20, image::ImageFormat::Jpeg))]
assert res.status == :see_other
photo = Photo.find_by(title: "Renamed")!.unwrap()
assert photo.image_content_type.as_deref() == Some("image/jpeg")
end
```
A file that is a real image of an allowed kind but fails to decode
part-way can still get this far:
```glacier
test "keeps the old image's files when the new one can't be stored" do
post "/photos", [title: "Stool", image: glacier::testing::file("stool-old.png", "image/png", png(50, 50))]
before = Photo.find_by(title: "Stool")!.unwrap()
# A PNG cut short: it is what it says, so it passes the validations, but
# its styles can't be made.
broken = png(50, 50)[..40].to_vec()
res = patch before, [title: "Stool", image: glacier::testing::file("stool-new.png", "image/png", broken)]
assert res.status == :unprocessable_entity
for style in ["original", "medium", "thumb"] do
assert store().get(&Photo.IMAGE.path("photos", before.id, style, "stool-old.png"))!.is_some()
end
# The save is undone: the photo still describes the old image.
after = Photo.find(before.id)!
assert after.image_file_name == Some("stool-old.png".to_string())
assert after.image_file_size == before.image_file_size
assert after.image_updated_at == before.image_updated_at
assert store().get(&Photo.IMAGE.path("photos", before.id, "thumb", "stool-new.png"))!.is_none()
end
```
### Styles and Resizing
`Style::fit(name, width, height)` is the only kind of style: the image is
scaled to fit within the box with its proportions kept. There's no
cropping to fill the box, no format conversion and no quality setting. The style is written in the format the
image's bytes turn out to be (not the declared type or the extension),
using the `image` crate's default encoder settings, and Glacier reads
JPEG, PNG, GIF and WebP. Before resizing, an image is turned upright from
its EXIF orientation; the original is stored as it was sent. Some results
of this:
- The `image` crate's WebP encoder only writes lossless WebP, so a lossy
WebP upload gets lossless styles, which can be larger than the original.
- An animated GIF's styles have only its first frame.
- Styles carry no EXIF data, but the stored original keeps all of it,
including any GPS location a phone recorded. Anyone who can fetch the
original can read it, and with the `uploads` app's public `/uploads`
folder that is anyone who has its URL.
```glacier
test "keeps the original and a resized copy per style" do
post "/photos", [title: "Desk", image: glacier::testing::file("desk.png", "image/png", png(1600, 1200))]
photo = Photo.find_by(title: "Desk")!.unwrap()
path = fn style -> Photo.IMAGE.path("photos", photo.id, style, "desk.png") end
assert path("thumb") == format!("photos/{}/thumb/desk.png", photo.id)
original = store().get(&path("original"))!.unwrap()
assert size_of(&original) == (1600, 1200)
assert size_of(&store().get(&path("medium"))!.unwrap()) == (640, 480)
assert size_of(&store().get(&path("thumb"))!.unwrap()) == (100, 75)
end
```
```glacier
test "resizes JPEG and WebP images, keeping their format" do
post "/photos", [title: "Jpeg", image: glacier::testing::file("view.jpg", "image/jpeg", encoded(1200, 1200, image::ImageFormat::Jpeg))]
post "/photos", [title: "Webp", image: glacier::testing::file("view.webp", "image/webp", encoded(300, 600, image::ImageFormat::WebP))]
jpeg = Photo.find_by(title: "Jpeg")!.unwrap()
webp = Photo.find_by(title: "Webp")!.unwrap()
thumb = store().get(&Photo.IMAGE.path("photos", jpeg.id, "thumb", "view.jpg"))!.unwrap()
assert image::guess_format(&thumb).unwrap() == image::ImageFormat::Jpeg
assert size_of(&thumb) == (100, 100)
medium = store().get(&Photo.IMAGE.path("photos", webp.id, "medium", "view.webp"))!.unwrap()
assert image::guess_format(&medium).unwrap() == image::ImageFormat::WebP
assert size_of(&medium) == (240, 480)
end
```
An image smaller than a style's box is enlarged to fit it:
```glacier
test "enlarges a small image to fill its style's box" do
post "/photos", [title: "Stamp", image: glacier::testing::file("stamp.png", "image/png", png(40, 30))]
photo = Photo.find_by(title: "Stamp")!.unwrap()
medium = store().get(&Photo.IMAGE.path("photos", photo.id, "medium", "stamp.png"))!.unwrap()
assert size_of(&medium) == (640, 480)
end
```
Every style is put with the upload's declared content type, which only
the S3 store keeps.
NOTE: Styles are made during the request, before the response is sent,
and each style decodes the whole image again. Resizing is ordinary
synchronous code, not handed to a separate blocking thread, so while it
works it occupies one of the worker threads of the multi-threaded Tokio
runtime that `#[glacier::main]` starts (one per CPU core by default), and
that thread runs nothing else in the meantime. Enough uploads at once can
occupy them all. Decoding takes memory and time in proportion to an
image's pixel count, not its file size, and Glacier sets no limit on
either. Glacier decodes with the `image` crate's `into_decoder` and
`DynamicImage::from_decoder`. That passes the crate's default limits to
the decoder, but the check that the decoded image fits within 512 MiB is
made only by `ImageReader::decode`, which Glacier doesn't call, so the full
pixel buffer is allocated unchecked. Sent to the running app, a
1,166,428-byte PNG declaring 20,000 × 20,000 pixels was accepted, after
307 seconds in a debug build, with the server using over 1.3 GB of memory
while it worked. Glacier has [background jobs](background_jobs.html), but
attachments don't use them: styles are always made during the request.
### URLs and Paths
A style's URL is the store's URL for its key: `photo.image_url("medium")`,
or `None` when there's no image (`"original"` for the file as sent). On S3
it's a presigned URL good for five minutes. The `show` action passes them
to its template:
```glacier
def show do
photo = Photo.find(params[:id])!
medium = photo.image_url("medium")
original = photo.image_url("original")
render photo, medium: medium, original: original, title: "Photo"
end
```
```html
{% if let Some(src) = medium %}
<p><img src="{{ src }}" alt="{{ photo.title }}"></p>
{% end %}
{% if let Some(href) = original %}
<p><a href="{{ href }}">Original</a> ({{ photo.image_file_size.unwrap_or(0) }} bytes)</p>
{% end %}
```
Glacier doesn't add anything such as `image_updated_at` to the URL, so a
replaced image with the same name has the same URL as the old one. With
the disk store, browsers revalidate every time, and the `ETag` that
`Static` sends (made from the file's size and modification time) normally
lets them notice the change.
### Replacing and Deleting
Saving a new upload over an old one removes the old one's original and
styles once the new one is stored, if the name changed, since a new name
means new keys:
```glacier
test "replaces the image's files on update" do
post "/photos", [title: "Chair", image: glacier::testing::file("chair-old.png", "image/png", png(50, 50))]
photo = Photo.find_by(title: "Chair")!.unwrap()
res = patch photo, [title: "Chair", image: glacier::testing::file("chair-new.png", "image/png", png(60, 60))]
assert res.status == :see_other
assert store().get(&Photo.IMAGE.path("photos", photo.id, "thumb", "chair-old.png"))!.is_none()
assert store().get(&Photo.IMAGE.path("photos", photo.id, "thumb", "chair-new.png"))!.is_some()
end
```
```glacier
test "replaces the files in place when the new image has the same name" do
post "/photos", [title: "Vase", image: glacier::testing::file("vase.png", "image/png", png(50, 50))]
photo = Photo.find_by(title: "Vase")!.unwrap()
patch photo, [title: "Vase", image: glacier::testing::file("vase.png", "image/png", png(80, 40))]
original = store().get(&Photo.IMAGE.path("photos", photo.id, "original", "vase.png"))!.unwrap()
assert size_of(&original) == (80, 40)
end
```
`destroy` removes a record's files after deleting it (`delete`, which
skips callbacks, leaves them):
```glacier
# `destroy` removes the image's files with the photo.
def destroy do
Photo.find(params[:id])!.destroy()!
redirect Photo.collection_path(), notice: "Photo deleted."
end
```
```glacier
test "deletes the files with the photo" do
post "/photos", [title: "Rug", image: glacier::testing::file("rug.png", "image/png", png(50, 50))]
photo = Photo.find_by(title: "Rug")!.unwrap()
delete photo
for style in ["original", "medium", "thumb"] do
assert store().get(&Photo.IMAGE.path("photos", photo.id, style, "rug.png"))!.is_none()
end
end
```
If you change the styles later, files for the old styles aren't removed or
remade: removing deletes only the styles the attachment lists now.
### Validating Uploads
The spoof check is built in, and `content_types:` limits the types. The
columns are ordinary fields too, set before validation, so a `validate`
block can check them with messages of its own (see
[Models](models.html#custom-validations)). `Photo` accepts only image
types, by what the bytes show, and only file names with an image
extension:
```glacier
# Uploads are checked for contents that aren't what they claim to be;
# these rules narrow them to images.
validate do
if let Some(kind) = &self.image_content_type do
if not ["image/png", "image/jpeg", "image/gif", "image/webp"].contains(&kind.as_str()) do
errors.add("image", "must be a PNG, JPEG, GIF or WebP image")
end
end
if let Some(name) = &self.image_file_name do
name = name.to_ascii_lowercase()
if not [".png", ".jpg", ".jpeg", ".gif", ".webp"].any(fn ext -> name.ends_with(ext) end) do
errors.add("image", "must be named .png, .jpg, .jpeg, .gif or .webp")
end
end
end
```
```glacier
test "refuses a file that isn't a PNG, JPEG, GIF or WebP image" do
res = post "/photos", [title: "Page", image: glacier::testing::file("page.html", "text/html", "<script>alert(1)</script>")]
assert res.status == :unprocessable_entity
assert res.body =~ "Image must be a PNG, JPEG, GIF or WebP image"
assert res.body =~ "Image must be named .png, .jpg, .jpeg, .gif or .webp"
assert Photo.count()! == 0
assert store().get("photos/1/original/page.html")!.is_none()
end
```
The extension check matters as much as the type check, because files on
disk are served with a type chosen from their extension. Leave SVG out of
such a list unless you serve files from another domain: an SVG can carry
scripts.
## Serving Stored Files (updated)
There are four ways to let browsers have stored files.
**Publicly, from disk.** `Static::new(folder).at("/uploads")`, as in
`src/app.glacier` above, serves every file in the folder to anyone who asks,
for `GET` and `HEAD`, with an `ETag` and `Cache-Control: public, max-age=0,
must-revalidate`:
```glacier
test "serves the stored files under /uploads" do
post "/photos", [title: "Lamp", image: glacier::testing::file("lamp.png", "image/png", png(800, 600))]
photo = Photo.find_by(title: "Lamp")!.unwrap()
url = format!("/uploads/photos/{}/medium/lamp.png", photo.id)
assert photo.image_url("medium") == Some(url.clone())
page = get photo
assert page.body =~ format!("<img src=\"{url}\"")
image = get url
assert image.status == :ok
assert image.headers["content-type"] == "image/png"
end
```
Keys built from sequential ids and file names are easy to guess, so treat
anything served this way as public. Two more things follow from how
`Static` works:
- It chooses the `Content-Type` from the file's extension: `.html` is
served as `text/html` and `.svg` as `image/svg+xml`. An uploaded `.html`
or `.svg` file would run as a page on your site, scripts included, which
is why the validations above allow only image extensions.
- It runs before routing, so the browser pipeline's security headers, such
as `X-Content-Type-Options: nosniff`, aren't added to its responses.
**Through an action.** To decide per request who may have a file, don't
mount `Static` on the folder. Read the file with `store.get` in an action,
after whatever check you need, and send it. Glacier has no helper for
sending a file from an action yet, so the `uploads` app builds the
response itself, and
sends it as a download so the browser doesn't display it:
```glacier
# Sends the original image as a download, through the app, so that this
# action decides who may have it.
def download do
photo = Photo.find(params[:id])!
name = photo.image_file_name.clone().ok_or_else(Error.not_found)?
key = photo.image_path("original").ok_or_else(Error.not_found)?
bytes = glacier::storage::current().get(&key)!.ok_or_else(Error.not_found)?
disposition = format!("attachment; filename=\"{}\"", name.replace("\"", ""))
response = Response.new(StatusCode::OK).with_body_bytes(bytes)
response = response.with_header("content-type", "application/octet-stream")
response.with_header("content-disposition", disposition)
end
```
```glacier
post "/photos/bulk", PhotosController.bulk, as: :bulk_photos
get "/photos/:id/download", PhotosController.download, as: :download_photo
resources "/photos", PhotosController
```
```glacier
test "sends the original through the app" do
post "/photos", [title: "Map", image: glacier::testing::file("map.png", "image/png", png(30, 20))]
photo = Photo.find_by(title: "Map")!.unwrap()
res = get format!("/photos/{}/download", photo.id)
assert res.status == :ok
assert res.headers["content-disposition"] == "attachment; filename=\"map.png\""
assert res.headers["content-type"] == "application/octet-stream"
assert res.body =~ "PNG"
end
```
The action uses only `get`, which both stores have; with an S3 store,
every download then passes through your server.
**By a signed link that expires.** `storage().signed_url(key, seconds)`
is a link to a private file that works for that long and no longer. On
S3 it's a presigned URL; on disk it's `/files/<key>?token=…`, which the
`SignedFiles` middleware serves, refusing a link that was altered or has
expired:
```glacier
# Private files, by signed links that expire: `storage().signed_url(key, 300)`.
middleware SignedFiles
```
```glacier
test "a signed link to a private file works until it expires" do
storage().put("private/report.txt", "Quarterly".into(), "text/plain")!
url = storage().signed_url("private/report.txt", 60)
assert url =~ "/files/private/report.txt?token="
res = get url
assert res.status == :ok
assert res.body == "Quarterly"
assert res.header("content-security-policy")? == "sandbox"
# Another file with the same token, or no token at all: refused.
assert get(url.replace("report", "secrets")).status == :forbidden
assert get("/files/private/report.txt").status == :forbidden
travel(2.minutes())
assert get(url).status == :forbidden
end
```
The token signs the key and the time it expires with the app's secret
key, so it can't be stretched or moved to another file. A signed file
is sent with the type its extension names, `nosniff`, and a
`Content-Security-Policy: sandbox`, so an uploaded page or SVG can't run
scripts as the app; and with `Cache-Control: private, no-store`. As with
a presigned URL, a link can't be withdrawn before it expires.
**From S3, by presigned URL.** With an S3 store, `url` gives a presigned
URL that lets the browser fetch a private object straight from S3 until it
expires. The `uploads` app's `image_url` asks for five minutes (300
seconds). S3 sends the object with the content type it was stored with,
which for an attachment is the type the client declared.
## Testing Uploads
### Files in Test Forms
`glacier::testing::file(filename, content_type, bytes)` makes a file to
put in a test form. A form holding one is sent as `multipart/form-data`,
as a browser sends it, and a nested keyword list names its files as it
names other fields, so `[images: [a: file(…)]]` sends a file field named
`images[a]`:
```glacier
test "adds a photo per file" do
res = post "/photos/bulk", [images: [a: glacier::testing::file("cup.png", "image/png", png(20, 20)), b: glacier::testing::file("mug.png", "image/png", png(20, 20))]]
assert res.status == :see_other
assert get(res.location).body =~ "2 photos added."
assert Photo.find_by(title: "mug.png")!.is_some()
end
```
Unlike `post` and `insert`, `file` isn't a name the `.glacier` compiler
knows, and test files don't import it, so it's written with its full path.
Setting the same field name twice in a form keeps only the last file, so a
test can't send several files under one name, as `<input multiple>` does.
`file("", "application/octet-stream", "")` sends what a browser sends for
an empty file field:
```glacier
test "an empty file field is no file" do
res = post "/photos", [title: "Nothing", image: glacier::testing::file("", "application/octet-stream", "")]
assert res.status == :see_other
photo = Photo.find_by(title: "Nothing")!.unwrap()
assert photo.image_file_name == nil
end
```
As with any form, the test client sends the CSRF token for an upload in the
`X-CSRF-Token` header, using the token from a page it loaded earlier (the
`setup` block loads `/photos/new`). That is also why these tests can't
catch the [browser form problem](#an-upload-form).
### Test Images
Resizing needs real images, so the tests make them in memory with the
`image` crate, a dev-dependency of the app, rather than keeping image
files in the repository:
```toml
# Makes test images to upload.
[dev-dependencies]
image = { version = "0.25", default-features = false, features = ["png", "jpeg", "webp"] }
```
```glacier
# A PNG of the given size, made in memory.
def png(width: u32, height: u32) -> Array<u8> do
out = std::io::Cursor::new(Array.new())
image::DynamicImage::new_rgb8(width, height).write_to(&mut out, image::ImageFormat::Png).unwrap()
out.into_inner()
end
```
```glacier
# The width and height of a stored image.
def size_of(bytes: &[u8]) -> (u32, u32) do
picture = image::load_from_memory(bytes).unwrap()
(picture.width(), picture.height())
end
def store() -> Store do
Store::Disk("tmp/storage".into())
end
```
### What the Store Holds (new)
`assert_stored(key)!` fails the test unless the app's store has a file
at `key`, and `assert_missing(key)!` if it does:
```glacier
test "asserts what the store holds" do
storage().put("exports/1.csv", "a,b".into(), "text/csv")!
assert_stored("exports/1.csv")!
storage().delete("exports/1.csv")!
assert_missing("exports/1.csv")!
end
```
### Where Test Files Go
The test configuration points the store at `tmp/storage`, which the app's
`.gitignore` excludes. Each test gets a fresh database but not a fresh
folder, and the tests run in parallel, so ids repeat between tests while
the folder is shared. The `uploads` tests give every test its own file
names, so that two tests never write the same key; files left behind are
overwritten by the next run.
## What Isn't Built Yet (updated)
Glacier's file handling doesn't yet have:
- **A generator field type** for files, or a form helper for file fields
and `enctype`; and a record has no built-in way to hold several files
under one name.
- **Separate tables for file details**: they live in the record's own
columns.
- **JavaScript for direct uploads**: `presigned_post` gives a form that
uploads without it, but there's no script for progress bars or for
uploading in the background while the rest of a form is filled in.
- **Styles made on demand**, previews of PDFs and videos, and file
analysis (an image's dimensions aren't recorded).
- **Other resize modes**, format conversion and quality settings: styles
only fit within a box.
- **Limits on images**: nothing caps an image's dimensions or the memory
and time its styles take.
- **Background processing for files**: resizing, storing and deleting all
happen during the request. Glacier has
[background jobs](background_jobs.html), but attachments and stores
don't use them.
- **Other services and mirroring**: there are only the disk and S3
stores.
- **Other AWS credentials**: no session tokens, profiles or instance
roles.
- **Timeouts on S3 requests.**
- **Content checks beyond the spoof check**: an attachment compares a
file's bytes with its declared type and extension for common types
only, and a `Store` used directly checks nothing.
- **Streaming**: a request body is read into memory whole, up to
`server.max_body_size`, and S3 uploads are sent in one request.
- **A helper for sending a file from an action**: build the response,
as the download above does.