---
title: File Uploads and Storage
description: Receiving files from multipart forms, keeping them on disk or in S3, attaching images with resized styles to records, serving them, and testing uploads.
category: Going Further
order: 4
---
Glacier handles file uploads in three layers. A controller reads the files
in a `multipart/form-data` request with `ctx.file` and `ctx.files`. A
`storage::Store` keeps files in a folder or in an S3 bucket. An
`attachment::Attachment` ties a file to a record: the original plus
resized copies of an image, under paths built from a pattern, described by
four columns on the record. There's no way yet to declare an attachment in
a model: you write the few model methods that join these pieces, and this
guide shows them.
After reading this guide, you will know:
- How to read uploaded files in an action, and which of their details to
trust.
- How to limit the size of uploads.
- How to write an upload form, and a problem with browser upload forms
that Glacier hasn't fixed yet.
- How to keep files on disk or in S3, and how presigned S3 URLs work.
- How to attach an image to a model, with resized styles.
- How to serve stored files, publicly or through the app.
- How to test uploads.
- What Glacier's file handling doesn't do yet.
The examples come from `uploads`, an app made with `glacier new uploads`
and a `Photo` scaffold whose photos each have an image.
## Receiving Uploads
### Multipart Forms
A browser sends a form with a file field as `multipart/form-data`, a body
made of parts, one per field. Glacier reads such a body in two ways:
- Its text fields become params, as a URL-encoded form's do, so
`ctx.params()` and `Photo.build(ctx.params())` work unchanged.
- Its files are left out of the params and read with `ctx.file(name)` and
`ctx.files()`.
### `ctx.file` and `ctx.files`
`ctx.file("image")` returns the file sent in the field named `image`, as an
`Option<Upload>`: `None` if the request isn't multipart or has no such
file. The name is matched exactly as the form sent it, so a field named
`photo[image]` is read with `ctx.file("photo[image]")`. If several files
share the name, it returns the first.
`ctx.files()` returns every file in the request, in the order sent, as
`(field name, Upload)` pairs. The `uploads` app uses it to add a photo for
each file chosen in an `<input type="file" multiple>`:
```html
<form method="post" action="{{ bulk_photos_path() }}" enctype="multipart/form-data">
{{ csrf_field() }}
<input type="file" name="images[]" multiple accept="image/png,image/jpeg,image/gif,image/webp">
<button>Add photos</button>
</form>
```
```glacier
# Adds a photo for every file sent, whatever its field name.
def bulk(ctx) do
store = ctx.state::<Store>()?
added = 0
for (_field, upload) in ctx.files() do
photo = Photo.new()
photo.title = upload.filename.clone()
if photo.save_with_image(store, Some(upload))! do
added = added + 1
end
end
redirect Photo.collection_path(), notice: format!("{added} photos added.")
end
```
An `Upload` (`glacier::upload::Upload`) has three public fields and four
methods:
| `filename` | The file's name as the client sent it, cut short at any `;` (see below) |
| `content_type` | The `Content-Type` the client gave the part, or `application/octet-stream` if it gave none |
| `bytes` | The file's contents, as `Bytes` |
| `len()`, `is_empty()` | The size of `bytes` |
| `detected_type()` | The type the bytes show (see [Which Details to Trust](#which-details-to-trust)) |
| `is_spoofed()` | Whether the file claims to be something it isn't |
A browser sends a file field left empty as a part with an empty file name
and no content. Glacier leaves such a part out, so an empty field reads as
no file at all.
The multipart parser splits a part's `Content-Disposition` header at every
`;`, even inside a quoted name, so a file sent as `a;b.png` arrives with the
`filename` `a`. It reads only the plain `filename=` parameter and ignores
`filename*=`, the form some clients use for non-ASCII names. The `uploads`
app's validations then refuse the name, since it has lost its extension:
```glacier
test "cuts a file name at a semicolon" do
res = post "/photos", [title: "Semi", image: glacier::testing::file("a;b.png", "image/png", png(10, 10))]
assert res.status == :unprocessable_entity
assert res.body =~ "Image must be named .png, .jpg, .jpeg, .gif or .webp"
end
```
### Which Details to Trust
`filename` and `content_type` are whatever the client chose to send, and
Glacier doesn't clean up the name. Treat both as untrusted input, and use
the two checks an `Upload` makes from its bytes, as Rails' Paperclip does:
- `detected_type()` is the type the file's first bytes name
(`image/jpeg` for a JPEG called `cat.png`). For bytes without a known
signature it falls back to the extension's type, then to `text/plain`
for text and `application/octet-stream` for anything else. Attachments
record this type, not the declared one.
- `is_spoofed()` is true when the declared kind (`image` of `image/png`)
disagrees with the extension's or with the bytes: text sent as
`image/png`, or a photo called `report.pdf`. It knows common formats
only, so bytes it can't name and extensions it doesn't know never count
against a file.
Then:
- Never build a file system path from `filename` yourself. The attachment
code in this guide cleans the name (see
[Storing an Upload](#storing-an-upload)), and the disk store refuses
paths that climb out of its folder.
- Check the name's extension and the type against a list of the ones you
accept, as the [validations below](#validating-uploads) do. Files on disk
are [served](#serving-stored-files) with a `Content-Type` chosen from
their extension, so the extension decides how a browser treats them.
- To know what a file really contains, read its bytes. Making an image's
styles reads them, but only for an attachment with at least one style,
and it proves only that the bytes decode as a JPEG, PNG, GIF or WebP
image. The format is recognised from the file's first few bytes, not
from its name or type; anything appended after a valid image isn't
noticed; and the original is stored unchanged, whatever else it
contains.
### Size Limits
The server reads a request's whole body into memory before the app sees
it, and refuses a body larger than `server.max_body_size` bytes with `413
Payload Too Large`. The limit is 2 MiB unless you change it, and it covers
the whole request: every file in it, plus the other fields. The `uploads`
app raises it in `config/default.toml`:
```toml
# The largest request body accepted, uploads included: 10 MiB.
max_body_size = 10485760
```
`App::max_body_size(bytes)` sets it in code instead. With this setting,
the app, running locally (on port 4517 here), accepted a 3,309,623-byte PNG sent with
`curl` (with the CSRF token in a header, as [An Upload Form](#an-upload-form)
explains) and refused an 11,064,558-byte one before any CSRF check:
```console
$ curl -s -w ' %{http_code}\n' -F title=Large -F "image=@large.png;type=image/png" http://127.0.0.1:4517/photos
Payload Too Large 413
```
There is one limit for the whole app, not one per route or per field. It
bounds the size of the raw body, not the memory a request can use. On a
multipart request, every call to `ctx.params()`, `ctx.file` or `ctx.files`
parses the whole body again and copies every file in it into new `Bytes`
(`params()` copies the files and then drops them). So an action that calls
`params()` and then `file()`, as this app's do, holds the body and two
copies of each file at once; read the files once and keep them.
Making an image's styles can take far more memory than the file itself, as
[Styles and Resizing](#styles-and-resizing) explains.
NOTE: The limit is applied where the server reads the request from the
network. The test client hands requests to the app directly, so a test can
send a body larger than the limit without a 413.
## An Upload Form
A form that sends files needs `enctype="multipart/form-data"`, and an
`<input type="file">`. Glacier's form helpers, `form_for` and `form_tag`,
have no option for the `enctype` and there is no file field helper, so the
`uploads` app writes its `<form>` tag by hand, with `csrf_field()` for the
CSRF token and a `_method` field for the edit form. `photo.form_action()`
gives the action and method that `form_for` would have used:
```html
<form method="post" action="{{ action }}" enctype="multipart/form-data">
{{ csrf_field() }}
{%- if method == "patch" %}
<input type="hidden" name="_method" value="patch">
{%- end %}
<p>
<label>Title<br><input name="title" value="{{ photo.text("title") }}"></label>
{%- for message in photo.errors_on("title") %}
<small class="error">Title {{ message }}</small>
{%- end %}
</p>
<p>
<label>Image<br><input type="file" name="image" accept="image/png,image/jpeg,image/gif,image/webp"></label>
{%- for message in photo.errors_on("image") %}
<small class="error">Image {{ message }}</small>
{%- end %}
</p>
<button>Save</button>
</form>
```
The `accept` attribute only filters what the browser's file picker shows.
It's no check at all on what reaches the server.
WARNING: In this version of Glacier, a browser can't submit this form. The
CSRF check reads the `_csrf_token` field only from URL-encoded bodies, not
multipart ones, so it answers `403 Forbidden` ("Invalid CSRF token"). In
the same way, `MethodOverride` reads `_method` only from URL-encoded bodies,
so an edit form posted as multipart reaches the router as a `POST` and gets
`405 Method Not Allowed`. Both were checked against the running app with
`curl`, which also showed that the same upload succeeds when the token is
sent in the `X-CSRF-Token` header and the request uses the real `PATCH`
method. Until this is fixed, an upload has to come from a client that can
set the header and method, such as JavaScript's `fetch` with a `FormData`
body, or from an API client on a pipeline without CSRF checks (see
[Building JSON APIs](json_apis.html)). The app's tests pass regardless,
because the test client sends the token in the header and makes real
`PATCH` requests.
## Storing Files
### `storage::Store`
`glacier::storage::Store` keeps files under string keys such as
`photos/1/thumb/desk.png`, in one of two places:
- `Store::Disk(folder)`: files in a folder on the server.
- `Store::S3(signer)`: objects in an S3 bucket, or a service that speaks
S3's API, such as MinIO.
Both have the same four methods:
| `put(key, bytes, content_type)!` | Saves the bytes as `key`, replacing any file already there |
| `get(key)!` | The file's bytes, or `None` if there's no such file |
| `delete(key)!` | Removes the file; removing a missing file isn't an error |
| `url(key, expires_in)` | A URL a browser can fetch the file from (see below) |
A test in the `uploads` app uses a disk store directly:
```glacier
test "keeps files in a folder" do
store = Store::Disk("tmp/storage".into())
store.put("notes/hello.txt", "Hello".into(), "text/plain")!
assert store.get("notes/hello.txt")! == Some("Hello".into())
assert store.url("notes/hello.txt", 300) == "/uploads/notes/hello.txt"
store.delete("notes/hello.txt")!
assert store.get("notes/hello.txt")! == nil
store.delete("notes/hello.txt")!
end
```
A disk store creates folders as needed, and keeps every key inside its
folder: leading slashes are dropped, and a key with a `..` segment is
refused with an error (`400 Bad Request` if it reaches a response):
```glacier
test "refuses keys that climb out of the folder" do
store = Store::Disk("tmp/storage".into())
assert store.put("../escaped.txt", "no".into(), "text/plain").await.is_err()
end
```
`content_type` is only used by S3, which stores it with the object and
sends it back as the object's `Content-Type`. A disk store writes just the
bytes.
### Choosing a Store
A store is app state, registered with `App::state` and read in actions
with `ctx.state::<Store>()?`. The `uploads` app picks one from its
configuration:
```rust
/// An S3 bucket when `storage.bucket` is set (with the AWS_* variables),
/// otherwise the folder in `storage.root`.
fn store(config: &Config) -> glacier::Result<Store> {
if let Some(bucket) = config.get_opt::<String>("storage.bucket")? {
let s3 = S3Signer::from_env(&bucket).ok_or_else(|| {
Error::internal("storage.bucket needs AWS_REGION, AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY")
})?;
return Ok(Store::S3(s3));
}
let root = config.get_opt::<String>("storage.root")?;
Ok(Store::Disk(root.unwrap_or_else(|| "storage".into()).into()))
}
```
```toml
# Uploaded files: a folder, relative to where the server runs. Set
# `bucket` (and AWS_REGION, AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in
# the environment) to keep them in S3 instead; see src/lib.rs.
[storage]
root = "storage"
```
The tests keep their files out of the way, in a folder the app's
`.gitignore` lists along with `storage/`:
```toml
# Files stored by tests go here (git-ignored), not into storage/.
[storage]
root = "tmp/storage"
```
The `storage` settings are the app's own: Glacier reads no storage
configuration itself.
### Files on Disk
For a disk store, `url(key, …)` is `/uploads/` followed by the key, such as
`/uploads/photos/1/medium/desk.png`, whatever the folder is called. It
ignores `expires_in`, and the key isn't percent-encoded. Glacier doesn't
serve that path by itself: the app has to, and the `uploads` app does it
with the `Static` middleware, pointed at the store's folder:
```rust
let store = store(&config)?;
let mut app = App::new()
.config(config)?
.database(repo)
.migrations(migrations::all())
// Before the logger, so logs show the overridden method.
.middleware(MethodOverride)
.middleware(Logger)
// From public/ in debug builds, compiled in for release.
.middleware(glacier::static_files!("public"));
// Files kept on disk are served from /uploads, where Store::url points.
if let Store::Disk(root) = &store {
app = app.middleware(Static::new(root.clone()).at("/uploads"));
}
Ok(app.state(store).routes(routes::draw))
```
[Serving Stored Files](#serving-stored-files) covers what that means for
who can see them.
### Files in S3
`glacier::storage::S3Signer` holds a bucket, its region and an access key,
and signs requests with AWS Signature Version 4. `Store::S3` uses it to
`PUT`, `GET` and `DELETE` objects, over HTTPS unless an endpoint says
otherwise, sending each file in one request. There are three ways to make one:
| `S3Signer::new(bucket, region, access_key_id, secret_access_key)` | For `https://<bucket>.s3.<region>.amazonaws.com` |
| `S3Signer::from_env(bucket)` | The same, from `AWS_REGION`, `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`; `None` if any of them is missing or empty |
| `.with_endpoint("http://localhost:9000")` | Another S3-compatible service, such as MinIO, with `/bucket/key` paths |
`.with_host(host)` replaces the host name: requests are sent to it, it's
the host that's signed, and presigned URLs point at it. On a signer made
by `new` or `from_env`, the bucket isn't put in the path, so the host has
to name the bucket itself, as `examplebucket.s3.amazonaws.com` does. `from_env` reads only those three variables: not
`AWS_SESSION_TOKEN`, `AWS_DEFAULT_REGION`, `~/.aws` files or instance roles,
so temporary credentials don't work yet.
A `404` from S3 makes `get` return `None`. Any other failure, from an
unreachable service or a non-success status, is an error that answers
`500` if it reaches the response. Glacier sets no timeout on S3 requests,
so an endpoint that accepts the connection and then stops answering holds
up the request that's waiting on it for as long as it hangs. Glacier sends
no ACL with an upload, so who can read an object is up to the bucket's own
settings.
WARNING: The S3 store treats a `404` as "not found" for every request, not
only `get`. S3 answers `404` (`NoSuchBucket`) when the bucket doesn't
exist, so a `put` to a misspelt bucket name returns without an error and
stores nothing. Check the bucket name, or read a file back once after
deploying.
NOTE: The `uploads` app's tests don't talk to AWS or MinIO, so the S3 store
hasn't been run against a real bucket for this guide. What the tests check
is the signing, which runs locally: Glacier's own tests check its signed
request headers against AWS's published example, and the app checks a
presigned URL (below) the same way.
### Presigned URLs
For an S3 store, `url(key, expires_in)` is a presigned URL: a `GET` URL for
the object that works for `expires_in` seconds without credentials, so a
private object can be shown in an `<img>` or linked to directly.
`S3Signer::presigned_url(key, expires_in)` makes one from the signer, and
`presigned_url_at(key, expires_in, time)` signs as at a given time, which
lets a test compare the result with the URL AWS's documentation gives for
the same request:
```glacier
test "signs a URL as AWS's documented example does" do
s3 = glacier::storage::S3Signer::new("examplebucket", "us-east-1", "AKIAIOSFODNN7EXAMPLE", "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY").with_host("examplebucket.s3.amazonaws.com")
at = "2013-05-24T00:00:00Z".parse::<Timestamp>().unwrap()
url = s3.presigned_url_at("test.txt", 86400, at)
assert url == "https://examplebucket.s3.amazonaws.com/test.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIOSFODNN7EXAMPLE%2F20130524%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20130524T000000Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=aeeed9bbccd4d02ee5c0109b86d86835f995330da4c265957d157751f604d404"
end
```
```glacier
test "gives S3 files presigned URLs" do
s3 = glacier::storage::S3Signer::new("my-bucket", "eu-west-2", "AKIA…", "secret")
store = Store::S3(s3)
url = store.url("photos/1/thumb/desk.png", 300)
assert url =~ "https://my-bucket.s3.eu-west-2.amazonaws.com/photos/1/thumb/desk.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&"
assert url =~ "&X-Amz-Expires=300&"
end
```
```glacier
test "uses /bucket/key paths for another endpoint" do
s3 = glacier::storage::S3Signer::new("photos", "us-east-1", "minioadmin", "minioadmin").with_endpoint("http://localhost:9000")
assert s3.presigned_url("1/thumb/desk.png", 60) =~ "http://localhost:9000/photos/1/thumb/desk.png?"
end
```
A presigned URL can't be withdrawn early: anyone who has it can fetch the
object until it expires. AWS refuses an `X-Amz-Expires` longer than seven
days (604,800 seconds); Glacier doesn't check the value. Presigned URLs
are for downloads only: Glacier can't presign an upload.
## Attachments
`glacier::attachment` keeps an uploaded file for a record: the original,
plus a resized copy for each style, at keys built from a pattern, and four
columns on the record describing the file. Glacier has
no declaration for it in a model yet, so it takes a migration, four
fields, a constant and some model methods.
### The Columns
For an attachment called `image`, the columns are `image_file_name`,
`image_content_type`, `image_file_size` and `image_updated_at`. They're
the names the Paperclip library gives its columns, so a table that already
has them can be used without renaming them, as long as their types suit
the model's fields (`image_file_size` is read as an `i32`). Its stored files
are another matter: the path pattern below has no placeholder that splits
the id into folders such as `000/000/007`, as that library's default path
does, so files kept under such paths would have to be moved.
The `uploads` app made the scaffold and a migration to add the columns.
The output below is from running the same commands again in a fresh app,
so the timestamps in the migration names differ from the app's own files:
```console
$ glacier generate scaffold Photo title:string
create src/migrations/m20261004131149_create_photos.rs
update src/migrations/mod.rs
create src/models/photo.glacier
update src/models/mod.rs
create tests/factories/photo.glacier
create tests/models/photo.test.glacier
update tests/glacier.rs
create src/controllers/photos_controller.glacier
create src/views/photos/index.html
create src/views/photos/show.html
create src/views/photos/new.html
create src/views/photos/edit.html
create src/views/photos/_form.html
create tests/controllers/photos_controller.test.glacier
update tests/glacier.rs
update src/controllers/mod.rs
update src/routes.rs
Next: glacier db migrate, then visit /photos
$ glacier generate migration add_image_to_photos
create src/migrations/m20261004131150_add_image_to_photos.rs
update src/migrations/mod.rs
```
The generators have no field type for files, so the migration's columns
were written by hand, in place of the one it was generated with (see
[Migrations](migrations.html)):
```rust
pub fn up(s: &mut Schema) {
s.alter_table("photos", |t| {
t.string("image_file_name");
t.string("image_content_type");
t.integer("image_file_size");
t.timestamp("image_updated_at");
});
}
```
The model's fields are `protected`, so that a request can't set them
through params, and optional, since a photo may have no image:
```glacier
field image_file_name: Option<String>, protected
field image_content_type: Option<String>, protected
field image_file_size: Option<i32>, protected
field image_updated_at: Option<Timestamp>, protected
```
### Declaring an Attachment
An `Attachment` is a path pattern and a list of styles. The `uploads` app
keeps its one as a Rust constant in `src/attachments.rs`, declared as a
module in `src/lib.rs`:
```rust
/// A photo's image: the original as uploaded, plus two resized copies.
pub const PHOTO_IMAGE: Attachment = Attachment {
path: ":class/:id/:style/:filename",
styles: &[Style::fit("medium", 640, 480), Style::fit("thumb", 100, 100)],
};
```
The model brings it in with `alias Attachments.PHOTO_IMAGE`, which also
brings in the rest of that module, including the `pub use` lines that
make `Store` and `Upload` available to the model by those names.
The path pattern can use four placeholders, replaced as plain text:
| `:class` | The name you pass, such as `"photos"` |
| `:id` | The record's id |
| `:style` | `original`, or a style's name |
| `:filename` | The stored file name |
`PHOTO_IMAGE.path("photos", 7, "thumb", "desk.png")` is
`photos/7/thumb/desk.png`. Nothing is filled in for you: the class name,
id and file name are whatever the caller passes. Keep `:id` and `:style`
in the pattern, or different records or styles will share keys.
### Storing an Upload
`PHOTO_IMAGE.store(store, class, id, upload)!` saves an upload for a
record and returns a `Stored` value, whose fields `file_name`,
`content_type`, `file_size` (an `i32`) and `updated_at` (a UTC
`NaiveDateTime`) are what the four columns should hold. It:
1. Cleans the file name with `glacier::attachment::sanitize_filename`:
each of `& $ + , / : ; = ? @ < > [ ] { } | \ ^ ~ % #`
and the space becomes `_`. Other characters, including quotes,
parentheses and non-ASCII letters, are kept.
2. Puts the original, unchanged, at the `original` style's key.
3. Makes and puts each style in turn.
It needs the record's id, so the record has to be saved first. The
`Photo` model wraps it all in one method, which the controller's `create`
and `update` call:
```glacier
# Saves the photo with an uploaded image, if one was sent: its name, the
# type its bytes show and whether it claims to be what it isn't are
# checked by the validations, and its files are stored once the photo
# has an id. An earlier image's files are removed only after the
# new ones have all been stored.
def save_with_image(self, store: &Store, upload: Option<Upload>) -> bool do
previous = self.image_file_name.clone()
if let Some(upload) = &upload do
self.image_file_name = glacier::attachment::sanitize_filename(&upload.filename)
self.image_content_type = upload.detected_type()
self.image_spoofed = upload.is_spoofed()
end
saved = self.save()!
if saved do
if let Some(upload) = &upload do
stored = PHOTO_IMAGE.store(store, "photos", self.id, upload)!
self.image_file_size = stored.file_size
self.image_updated_at = stored.updated_at.and_utc()
self.save()!
if let Some(name) = previous do
if name != stored.file_name do
PHOTO_IMAGE.remove(store, "photos", self.id, &name)!
end
end
end
end
saved
end
```
```glacier
def create(ctx) do
photo = Photo.build(ctx.params())
if photo.save_with_image(ctx.state::<Store>()?, ctx.file("image"))! do
redirect photo, notice: "Photo created."
else
render :new, photo, title: "New photo", status: 422
end
end
```
The name and type go into the columns before the first save, so the
validations can check them. The files are stored only if that save
succeeds, and then the size and time are written by a second save. Only
after that are the previous image's files removed, and only if its name
was different: a new file with the same name has already been written over
them. Removing them before storing the new files would lose them if
`store` failed.
```glacier
test "creates a photo with an image" do
bytes = png(1600, 1200)
res = post "/photos", [title: "Shelf", image: glacier::testing::file("shelf.png", "image/png", bytes.clone())]
assert res.status == :see_other
photo = Photo.find_by(title: "Shelf")!.unwrap()
assert photo.image_file_name == Some("shelf.png".to_string())
assert photo.image_content_type == Some("image/png".to_string())
assert photo.image_file_size == Some(bytes.len() as i32)
assert photo.image_updated_at.is_some()
end
```
```glacier
test "makes the file name safe for paths and URLs" do
post "/photos", [title: "Hash", image: glacier::testing::file("my photo #1.png", "image/png", png(10, 10))]
photo = Photo.find_by(title: "Hash")!.unwrap()
assert photo.image_file_name == Some("my_photo__1.png".to_string())
end
```
WARNING: Neither `store` nor `save_with_image` is transactional. `store`
puts the original first, then makes and puts each style in turn, and stops
at the first failure, leaving whatever it has already written. A file it
can't decode as an image fails with `422 Unprocessable Entity` after the
original is stored, and a `put` that fails part-way through the styles,
such as an S3 error, leaves some styles missing. By then `save_with_image`
has saved the record with the new file name and type, but not the new size
and time: the record names files that lack some or all of their styles,
while `image_file_size` and `image_updated_at` still describe the previous
image. On an update the previous image's files are still in the store, but
nothing refers to them any more; if the new file has the same name, the
files written so far have replaced the old ones, leaving a mix of the two.
If the second save fails after the new files are stored, the record keeps
the previous size and time. If removing the previous files fails after the
new ones are stored, the request fails with an error even though the photo
was saved, and some of the previous files may be left. Validating the
name, the detected type and the spoof check first, as below, keeps out a
file that claims to be a PNG and isn't, and records the type a file
really has:
```glacier
test "refuses a file that claims to be an image and isn't" do
res = post "/photos", [title: "Fake", image: glacier::testing::file("fake.png", "image/png", "not a png")]
assert res.status == :unprocessable_entity
assert res.body =~ "Image has contents that are not what they are reported to be"
assert Photo.find_by(title: "Fake")!.is_none()
end
test "keeps the type the bytes show, not the one declared" do
res = post "/photos", [title: "Renamed", image: glacier::testing::file("renamed.png", "image/png", encoded(20, 20, image::ImageFormat::Jpeg))]
assert res.status == :see_other
photo = Photo.find_by(title: "Renamed")!.unwrap()
assert photo.image_content_type.as_deref() == Some("image/jpeg")
end
```
A file that is a real image of an allowed kind but fails to decode
part-way can still get this far.
```glacier
test "keeps the old image's files when the new one can't be stored" do
post "/photos", [title: "Stool", image: glacier::testing::file("stool-old.png", "image/png", png(50, 50))]
before = Photo.find_by(title: "Stool")!.unwrap()
# A PNG cut short: it is what it says, so it passes the validations, but
# its styles can't be made.
broken = png(50, 50)[..40].to_vec()
res = patch before, [title: "Stool", image: glacier::testing::file("stool-new.png", "image/png", broken)]
assert res.status == :unprocessable_entity
for style in ["original", "medium", "thumb"] do
assert store().get(&PHOTO_IMAGE.path("photos", before.id, style, "stool-old.png"))!.is_some()
end
after = Photo.find(before.id)!
assert after.image_file_name == Some("stool-new.png".to_string())
assert after.image_file_size == before.image_file_size
assert after.image_updated_at == before.image_updated_at
assert store().get(&PHOTO_IMAGE.path("photos", before.id, "thumb", "stool-new.png"))!.is_none()
end
```
### Styles and Resizing
`Style::fit(name, width, height)` is the only kind of style: the image is
scaled to fit within the box with its proportions kept. There's no
cropping to fill the box, no format conversion and no quality setting. The style is written in the format the
image's bytes turn out to be (not the declared type or the extension),
using the `image` crate's default encoder settings, and Glacier reads
JPEG, PNG, GIF and WebP. Before resizing, an image is turned upright from
its EXIF orientation; the original is stored as it was sent. Some results
of this:
- The `image` crate's WebP encoder only writes lossless WebP, so a lossy
WebP upload gets lossless styles, which can be larger than the original.
- An animated GIF's styles have only its first frame.
- Styles carry no EXIF data, but the stored original keeps all of it,
including any GPS location a phone recorded. Anyone who can fetch the
original can read it, and with the `uploads` app's public `/uploads`
folder that is anyone who has its URL.
```glacier
test "keeps the original and a resized copy per style" do
post "/photos", [title: "Desk", image: glacier::testing::file("desk.png", "image/png", png(1600, 1200))]
photo = Photo.find_by(title: "Desk")!.unwrap()
path = fn style -> PHOTO_IMAGE.path("photos", photo.id, style, "desk.png") end
assert path("thumb") == format!("photos/{}/thumb/desk.png", photo.id)
original = store().get(&path("original"))!.unwrap()
assert size_of(&original) == (1600, 1200)
assert size_of(&store().get(&path("medium"))!.unwrap()) == (640, 480)
assert size_of(&store().get(&path("thumb"))!.unwrap()) == (100, 75)
end
```
```glacier
test "resizes JPEG and WebP images, keeping their format" do
post "/photos", [title: "Jpeg", image: glacier::testing::file("view.jpg", "image/jpeg", encoded(1200, 1200, image::ImageFormat::Jpeg))]
post "/photos", [title: "Webp", image: glacier::testing::file("view.webp", "image/webp", encoded(300, 600, image::ImageFormat::WebP))]
jpeg = Photo.find_by(title: "Jpeg")!.unwrap()
webp = Photo.find_by(title: "Webp")!.unwrap()
thumb = store().get(&PHOTO_IMAGE.path("photos", jpeg.id, "thumb", "view.jpg"))!.unwrap()
assert image::guess_format(&thumb).unwrap() == image::ImageFormat::Jpeg
assert size_of(&thumb) == (100, 100)
medium = store().get(&PHOTO_IMAGE.path("photos", webp.id, "medium", "view.webp"))!.unwrap()
assert image::guess_format(&medium).unwrap() == image::ImageFormat::WebP
assert size_of(&medium) == (240, 480)
end
```
An image smaller than a style's box is enlarged to fit it:
```glacier
test "enlarges a small image to fill its style's box" do
post "/photos", [title: "Stamp", image: glacier::testing::file("stamp.png", "image/png", png(40, 30))]
photo = Photo.find_by(title: "Stamp")!.unwrap()
medium = store().get(&PHOTO_IMAGE.path("photos", photo.id, "medium", "stamp.png"))!.unwrap()
assert size_of(&medium) == (640, 480)
end
```
Every style is put with the upload's declared content type, which only
the S3 store keeps.
NOTE: Styles are made during the request, before the response is sent,
and each style decodes the whole image again. Resizing is ordinary
synchronous code, not handed to a separate blocking thread, so while it
works it occupies one of the worker threads of the multi-threaded Tokio
runtime that `#[glacier::main]` starts (one per CPU core by default), and
that thread runs nothing else in the meantime. Enough uploads at once can
occupy them all. Decoding takes memory and time in proportion to an
image's pixel count, not its file size, and Glacier sets no limit on
either. Glacier decodes with the `image` crate's `into_decoder` and
`DynamicImage::from_decoder`. That passes the crate's default limits to
the decoder, but the check that the decoded image fits within 512 MiB is
made only by `ImageReader::decode`, which Glacier doesn't call, so the full
pixel buffer is allocated unchecked. Sent to the running app, a
1,166,428-byte PNG declaring 20,000 × 20,000 pixels was accepted, after
307 seconds in a debug build, with the server using over 1.3 GB of memory
while it worked. Glacier has [background jobs](background_jobs.html), but
attachments don't use them: styles are always made during the request.
### URLs and Paths
A style's URL is the store's URL for its key. The `Photo` model has a
method for it:
```glacier
# The URL of a style ("original", "medium" or "thumb"), if there's an image.
def image_url(self, store: &Store, style: &str) -> Option<String> do
self.image_file_name.as_ref().map(fn name ->
store.url(&PHOTO_IMAGE.path("photos", self.id, style, name), 300)
end)
end
```
Templates can't reach the store, so the `show` action works out the URLs
and passes them as props:
```glacier
def show(ctx) do
photo = Photo.find(ctx.id()?)!
store = ctx.state::<Store>()?
medium = photo.image_url(store, "medium")
original = photo.image_url(store, "original")
render photo, medium: medium, original: original, title: "Photo"
end
```
```html
{% if let Some(src) = medium %}
<p><img src="{{ src }}" alt="{{ photo.title }}"></p>
{% end %}
{% if let Some(href) = original %}
<p><a href="{{ href }}">Original</a> ({{ photo.image_file_size.unwrap_or(0) }} bytes)</p>
{% end %}
```
Glacier doesn't add anything such as `image_updated_at` to the URL, so a
replaced image with the same name has the same URL as the old one. With
the disk store, browsers revalidate every time, and the `ETag` that
`Static` sends (made from the file's size and modification time) normally
lets them notice the change.
### Replacing and Deleting
`PHOTO_IMAGE.remove(store, class, id, file_name)!` deletes the original and
every style. `save_with_image` calls it for the previous file once the new
one is stored, if the new one's name is different, since a new name means
new keys:
```glacier
test "replaces the image's files on update" do
post "/photos", [title: "Chair", image: glacier::testing::file("chair-old.png", "image/png", png(50, 50))]
photo = Photo.find_by(title: "Chair")!.unwrap()
res = patch photo, [title: "Chair", image: glacier::testing::file("chair-new.png", "image/png", png(60, 60))]
assert res.status == :see_other
assert store().get(&PHOTO_IMAGE.path("photos", photo.id, "thumb", "chair-old.png"))!.is_none()
assert store().get(&PHOTO_IMAGE.path("photos", photo.id, "thumb", "chair-new.png"))!.is_some()
end
```
```glacier
test "replaces the files in place when the new image has the same name" do
post "/photos", [title: "Vase", image: glacier::testing::file("vase.png", "image/png", png(50, 50))]
photo = Photo.find_by(title: "Vase")!.unwrap()
patch photo, [title: "Vase", image: glacier::testing::file("vase.png", "image/png", png(80, 40))]
original = store().get(&PHOTO_IMAGE.path("photos", photo.id, "original", "vase.png"))!.unwrap()
assert size_of(&original) == (80, 40)
end
```
Deleting a record doesn't delete its files, so the app's `destroy` action
removes them first:
```glacier
# Deletes the image's files: the original and each style.
def remove_image(self, store: &Store) do
if let Some(name) = &self.image_file_name do
PHOTO_IMAGE.remove(store, "photos", self.id, name)!
end
end
```
```glacier
def destroy(ctx) do
photo = Photo.find(ctx.id()?)!
photo.remove_image(ctx.state::<Store>()?)!
photo.delete()!
redirect Photo.collection_path(), notice: "Photo deleted."
end
```
```glacier
test "deletes the files with the photo" do
post "/photos", [title: "Rug", image: glacier::testing::file("rug.png", "image/png", png(50, 50))]
photo = Photo.find_by(title: "Rug")!.unwrap()
delete photo
for style in ["original", "medium", "thumb"] do
assert store().get(&PHOTO_IMAGE.path("photos", photo.id, style, "rug.png"))!.is_none()
end
end
```
If you change the styles later, files for the old styles aren't removed or
remade: `remove` deletes only the styles the attachment lists now.
### Validating Uploads
Glacier has no attachment validations, but the columns are ordinary
fields, so a `validate` block can check them (see
[Models](models.html#custom-validations)). `Photo` refuses a spoofed file
(through a virtual field `save_with_image` sets), and accepts only image
types, by what the bytes show, and only file names with an image
extension:
```glacier
validate do
if self.image_spoofed do
errors.add("image", "has contents that are not what they are reported to be")
end
if let Some(kind) = &self.image_content_type do
if not ["image/png", "image/jpeg", "image/gif", "image/webp"].contains(&kind.as_str()) do
errors.add("image", "must be a PNG, JPEG, GIF or WebP image")
end
end
if let Some(name) = &self.image_file_name do
name = name.to_ascii_lowercase()
if not [".png", ".jpg", ".jpeg", ".gif", ".webp"].iter().any(fn ext -> name.ends_with(ext) end) do
errors.add("image", "must be named .png, .jpg, .jpeg, .gif or .webp")
end
end
end
```
```glacier
test "refuses a file that isn't a PNG, JPEG, GIF or WebP image" do
res = post "/photos", [title: "Page", image: glacier::testing::file("page.html", "text/html", "<script>alert(1)</script>")]
assert res.status == :unprocessable_entity
assert res.body =~ "Image must be a PNG, JPEG, GIF or WebP image"
assert res.body =~ "Image must be named .png, .jpg, .jpeg, .gif or .webp"
assert Photo.count()! == 0
assert store().get("photos/1/original/page.html")!.is_none()
end
```
The extension check matters as much as the type check, because files on
disk are served with a type chosen from their extension. Leave SVG out of
such a list unless you serve files from another domain: an SVG can carry
scripts.
## Serving Stored Files
There are three ways to let browsers have stored files.
**Publicly, from disk.** `Static::new(folder).at("/uploads")`, as in
`src/lib.rs` above, serves every file in the folder to anyone who asks,
for `GET` and `HEAD`, with an `ETag` and `Cache-Control: public, max-age=0,
must-revalidate`:
```glacier
test "serves the stored files under /uploads" do
post "/photos", [title: "Lamp", image: glacier::testing::file("lamp.png", "image/png", png(800, 600))]
photo = Photo.find_by(title: "Lamp")!.unwrap()
url = format!("/uploads/photos/{}/medium/lamp.png", photo.id)
assert photo.image_url(&store(), "medium") == Some(url.clone())
page = get photo
assert page.body =~ format!("<img src=\"{url}\"")
image = get url
assert image.status == :ok
assert image.headers["content-type"] == "image/png"
end
```
Keys built from sequential ids and file names are easy to guess, so treat
anything served this way as public. Two more things follow from how
`Static` works:
- It chooses the `Content-Type` from the file's extension: `.html` is
served as `text/html` and `.svg` as `image/svg+xml`. An uploaded `.html`
or `.svg` file would run as a page on your site, scripts included, which
is why the validations above allow only image extensions.
- It runs before routing, so the browser pipeline's security headers, such
as `X-Content-Type-Options: nosniff`, aren't added to its responses.
**Through an action.** To decide per request who may have a file, don't
mount `Static` on the folder. Read the file with `store.get` in an action,
after whatever check you need, and send it. Glacier has no helper for
sending a file from an action yet, so the `uploads` app builds the
response itself, and
sends it as a download so the browser doesn't display it:
```glacier
# Sends the original image as a download, through the app, so that this
# action decides who may have it.
def download(ctx) do
photo = Photo.find(ctx.id()?)!
name = photo.image_file_name.clone().ok_or_else(Error.not_found)?
key = PHOTO_IMAGE.path("photos", photo.id, "original", &name)
bytes = ctx.state::<Store>()?.get(&key)!.ok_or_else(Error.not_found)?
disposition = format!("attachment; filename=\"{}\"", name.replace("\"", ""))
response = Response.new(StatusCode::OK).with_body_bytes(bytes)
response = response.with_header("content-type", "application/octet-stream")
Ok(response.with_header("content-disposition", disposition))
end
```
```rust
post "/photos/bulk" => Photos::bulk, as bulk_photos;
get "/photos/:id/download" => Photos::download, as download_photo;
resources "/photos" => Photos;
```
```glacier
test "sends the original through the app" do
post "/photos", [title: "Map", image: glacier::testing::file("map.png", "image/png", png(30, 20))]
photo = Photo.find_by(title: "Map")!.unwrap()
res = get format!("/photos/{}/download", photo.id)
assert res.status == :ok
assert res.headers["content-disposition"] == "attachment; filename=\"map.png\""
assert res.headers["content-type"] == "application/octet-stream"
assert res.body =~ "PNG"
end
```
The action uses only `get`, which both stores have; with an S3 store,
every download then passes through your server.
**From S3, by presigned URL.** With an S3 store, `url` gives a presigned
URL that lets the browser fetch a private object straight from S3 until it
expires. The `uploads` app's `image_url` asks for five minutes (300
seconds). S3 sends the object with the content type it was stored with,
which for an attachment is the type the client declared.
## Testing Uploads
### Files in Test Forms
`glacier::testing::file(filename, content_type, bytes)` makes a file to
put in a test form. A form holding one is sent as `multipart/form-data`,
as a browser sends it, and a nested keyword list names its files as it
names other fields, so `[images: [a: file(…)]]` sends a file field named
`images[a]`:
```glacier
test "adds a photo per file" do
res = post "/photos/bulk", [images: [a: glacier::testing::file("cup.png", "image/png", png(20, 20)), b: glacier::testing::file("mug.png", "image/png", png(20, 20))]]
assert res.status == :see_other
assert get(res.location).body =~ "2 photos added."
assert Photo.find_by(title: "mug.png")!.is_some()
end
```
Unlike `post` and `insert`, `file` isn't a name the `.glacier` compiler
knows, and test files don't import it, so it's written with its full path.
Setting the same field name twice in a form keeps only the last file, so a
test can't send several files under one name, as `<input multiple>` does.
`file("", "application/octet-stream", "")` sends what a browser sends for
an empty file field:
```glacier
test "an empty file field is no file" do
res = post "/photos", [title: "Nothing", image: glacier::testing::file("", "application/octet-stream", "")]
assert res.status == :see_other
photo = Photo.find_by(title: "Nothing")!.unwrap()
assert photo.image_file_name == nil
end
```
As with any form, the test client sends the CSRF token for an upload in the
`X-CSRF-Token` header, using the token from a page it loaded earlier (the
`setup` block loads `/photos/new`). That is also why these tests can't
catch the [browser form problem](#an-upload-form).
### Test Images
Resizing needs real images, so the tests make them in memory with the
`image` crate, a dev-dependency of the app, rather than keeping image
files in the repository:
```toml
# Makes test images to upload.
[dev-dependencies]
image = { version = "0.25", default-features = false, features = ["png", "jpeg", "webp"] }
```
```glacier
# A PNG of the given size, made in memory.
def png(width: u32, height: u32) -> Vec<u8> do
out = std::io::Cursor::new(Vec::new())
image::DynamicImage::new_rgb8(width, height).write_to(&mut out, image::ImageFormat::Png).unwrap()
out.into_inner()
end
```
```glacier
# The width and height of a stored image.
def size_of(bytes: &[u8]) -> (u32, u32) do
picture = image::load_from_memory(bytes).unwrap()
(picture.width(), picture.height())
end
def store() -> Store do
Store::Disk("tmp/storage".into())
end
```
### Where Test Files Go
The test configuration points the store at `tmp/storage`, which the app's
`.gitignore` excludes. Each test gets a fresh database but not a fresh
folder, and the tests run in parallel, so ids repeat between tests while
the folder is shared. The `uploads` tests give every test its own file
names, so that two tests never write the same key; files left behind are
overwritten by the next run.
## What Isn't Built Yet
Glacier's file handling doesn't yet have:
- **Browser upload forms**: the CSRF check and `MethodOverride` don't read
multipart bodies, as described [above](#an-upload-form).
- **A model declaration** for attachments, a generator field type for
files, or a form helper for file fields and `enctype`. Attachments are
wired up by hand, as in this guide, and a record has no built-in way to
hold several files.
- **Separate tables for file details**: they live in the record's own
columns.
- **Direct uploads** from the browser to S3: presigned URLs are `GET`
only.
- **Styles made on demand**, previews of PDFs and videos, and file
analysis (an image's dimensions aren't recorded).
- **Other resize modes**, format conversion and quality settings: styles
only fit within a box.
- **Limits on images**: nothing caps an image's dimensions or the memory
and time its styles take.
- **Background processing for files**: resizing, storing and deleting all
happen during the request. Glacier has
[background jobs](background_jobs.html), but attachments and stores
don't use them.
- **Removing files with the record**: deleting a record doesn't delete its
files; the app has to, as `destroy` does above.
- **Other services and mirroring**: there are only the disk and S3
stores.
- **Other AWS credentials**: no session tokens, profiles or instance
roles.
- **Timeouts on S3 requests.**
- **Content checks**: Glacier doesn't compare a file's bytes with its
declared type or extension, except that making image styles fails for a
file that doesn't decode as an image.
- **Streaming**: a request body is read into memory whole, up to
`server.max_body_size`, and S3 uploads are sent in one request.
- **Expiring or signed URLs for disk files**, and a helper for sending a
file from an action.