# defter
A self-hosted Zotero sync and file storage server: keep one Zotero library in
sync across your machines — metadata **and** attachment files such as PDFs —
without [Zotero Storage](https://www.zotero.org/storage). Metadata lives in SQLite, files in S3-compatible object
storage (Hetzner Object Storage, AWS S3, Cloudflare R2, RustFS, …). Stock Zotero 10 clients sync against
it, and [commonmeta](https://codeberg.org/front-matter/commonmeta-rs)'s `sync`
command can enrich the library through the same API.
**Status: early development (0.3).** The sync protocol, file storage and query
API are implemented and tested; interfaces may still change.
defter builds on [zhost](https://github.com/mulatta/zhost) (MIT, see
`NOTICE`), which worked out the subset of the Zotero Web API v3 a client
needs. defter swaps PostgreSQL for SQLite (rusqlite), `rust-s3` for the
`object_store` crate, and adds a web UI and the endpoints `commonmeta sync`
uses.
## How it works
- **Client redirect, no patching.** Zotero reads its sync server from the
hidden pref `extensions.zotero.api.url`; pointing it at defter redirects
sync and login.
- **Server.** A Rust/[axum](https://github.com/tokio-rs/axum) service. Library
objects are stored as opaque JSON keyed by `(kind, key)`, with the columns
queries need derived on every write. One library version counter, bumped by
each write, keeps the client's `since` reads and `If-Unmodified-Since-Version`
writes coherent.
- **Files.** Uploads stream through defter, which checks the md5 while writing
them to the bucket; downloads redirect to a short-lived pre-signed URL, so
the bytes come straight from the bucket.
- **Auth.** One user, one library. API keys are configured, each read/write or
read-only. The Zotero client gets a read/write key through its browser login,
which you approve on defter's login page by entering that key.
- **Web UI.** A dashboard (library statistics, storage, connection
instructions), shown after signing in with an API key, and the login
consent page for Zotero, rendered on the server from
[askama](https://askama.rs) templates and styled with Tailwind CSS.
[htmx](https://htmx.org) refreshes the dashboard in place; every page works
without JavaScript and is served under a strict Content-Security-Policy.
The full contract is in [`docs/api.md`](docs/api.md).
## Running
```sh
cargo install --path . # or: cargo install defter
defter start --public-url https://zotero.example.org \
--s3-bucket <your-bucket> \
--s3-endpoint https://fsn1.your-objectstorage.com --s3-region fsn1 \
--s3-access-key-id-file /etc/defter/s3-access-key-id \
--s3-secret-access-key-file /etc/defter/s3-secret-access-key
```
Every flag has a `DEFTER_*` environment variable; see `env.example` and
`defter start --help`. S3 credentials also fall back to the standard `AWS_*`
variables. Without `--s3-bucket`, files go to `<data-dir>/files` — fine for
development, not meant for production.
### API keys
The simplest setup is one read/write key in the environment, e.g. in
`/etc/defter/env`:
```sh
DEFTER_API_KEY=<output of: openssl rand -hex 32>
```
When Zotero logs in, it opens defter's login page in the browser; you approve
the login by entering that key, and Zotero receives it. Anyone can start a
login, so the key is what protects the library: generate it, don't choose it
(defter warns about keys shorter than 32 characters), and keep the file
readable by root and the `defter` group only.
To give scripts or agents read-only access as well, provision keys as files
instead, each read/write or read-only:
```sh
openssl rand -hex 32 > app-key # read/write, for Zotero
openssl rand -hex 32 > cli-key # read-only
defter start --keys rw:/path/app-key,ro:/path/cli-key …
```
With no key configured at all, defter generates a read/write key in
`<data-dir>/api-key`; `defter key` prints it.
### Deployment
defter listens on `127.0.0.1:8189`; put it behind a reverse proxy with HTTPS
and set `--public-url` to the proxy's address. The data API is protected by
keys and safe to expose, and so is `/login`, which hands out a key only to
whoever enters one. The dashboard (`/`) shows a sign-in form until an API key
is entered (a read-only key is enough), and `/ui/status` takes the key as a
header. Behind an SSO proxy, add `--login-authorized-user you@example.org` to
also require its identity on login approval.
Clients download from pre-signed URLs on `--s3-endpoint`, so the bucket
endpoint must be reachable from them; otherwise add `--proxy-downloads`.
### Debian with systemd
As root, with Rust installed through [rustup](https://rustup.rs):
```sh
# the binary, straight to /usr/local/bin (the service cannot run one from ~);
# chmod in case a restrictive umask (e.g. 027) made it unreadable to defter
cargo install --locked --root /usr/local defter
chmod 0755 /usr/local/bin/defter
useradd --system --no-create-home --shell /usr/sbin/nologin defter
install -d -m 0750 -o root -g defter /etc/defter
# env.example and defter.service from this repository:
install -m 0640 -o root -g defter env.example /etc/defter/env # then edit it
# the S3 keys, one line each, root:defter 0640:
# /etc/defter/s3-access-key-id, /etc/defter/s3-secret-access-key
install -m 0644 defter.service /etc/systemd/system/defter.service
systemctl daemon-reload && systemctl enable --now defter
journalctl -u defter -f
```
To upgrade: `cargo install --locked --root /usr/local defter` and
`chmod 0755 /usr/local/bin/defter` again, then `systemctl restart defter`.
A binary the `defter` user cannot execute makes systemd fail with status
203/EXEC (behind Cloudflare: error 525).
[`defter.service`](defter.service) runs defter as the `defter` user with its
data in `/var/lib/defter` (systemd's `StateDirectory`, which defter picks up
without `--data-dir`) and the rest of the file system read-only.
### Sharing port 443 with dragoman
On a machine where [dragoman](https://codeberg.org/front-matter/dragoman)
already serves HTTPS on port 443 behind a CDN, defter needs no port, proxy or
certificate of its own: dragoman's `--sni-route` passes connections for
defter's hostname, still encrypted, to defter, which serves HTTPS on a
loopback port with the same CDN origin certificate (if it covers defter's
hostname, e.g. `*.example.org`).
```sh
# dragoman (/etc/dragoman/env), then restart it:
DRAGOMAN_SNI_ROUTES=defter.example.org=127.0.0.1:8443
# defter (/etc/defter/env):
DEFTER_PORT=8443
DEFTER_PUBLIC_URL=https://defter.example.org
DEFTER_TLS_CERT=/etc/defter/origin.pem
DEFTER_TLS_KEY=/etc/defter/origin-key.pem
```
Give defter its own copy of the certificate and key, readable by its group:
```sh
install -m 0644 -o root -g defter /etc/dragoman/origin.pem /etc/defter/origin.pem
install -m 0640 -o root -g defter /etc/dragoman/origin-key.pem /etc/defter/origin-key.pem
```
With the hostname proxied by the CDN, its limits apply to uploads through
defter (Cloudflare: 100 MB per request on the Free and Pro plans), so set
`DEFTER_MAX_UPLOAD_MB=100` there; downloads come from the bucket directly.
### Hetzner Object Storage
[Hetzner Object Storage](https://www.hetzner.com/storage/object-storage/)
is the intended storage, for a defter running at home as well as for one
later deployed on a Hetzner Cloud server.
1. In Hetzner Console, create a project, a **private** Bucket in one location
(`fsn1` Falkenstein, `nbg1` Nuremberg or `hel1` Helsinki), and S3
credentials for the project (shown once: save both keys). Keys are valid
for every Bucket of the project, so give defter a project of its own.
2. Start defter with the location as both endpoint and region:
```sh
defter start \
--s3-bucket <your-bucket> \
--s3-endpoint https://fsn1.your-objectstorage.com --s3-region fsn1 \
--s3-access-key-id-file ~/.config/defter/s3-access-key-id \
--s3-secret-access-key-file ~/.config/defter/s3-secret-access-key
```
defter addresses Hetzner Buckets by host name
(`https://<bucket>.fsn1.your-objectstorage.com`), which its pre-signed
download URLs need.
3. Check the round trip: `DEFTER_KEY=$(defter key) scripts/smoke-s3.sh`.
Costs (Hetzner's prices, November 2026, excl. VAT): €6.49 a month includes
1 TB of storage and 1 TB of outgoing traffic; uploads and S3 requests are
free. Downloads by pre-signed URL count as outgoing traffic. Objects under
64 KB are billed as 64 KB.
For a later deployment on a Hetzner Cloud server, put the server in an EU
location: it is then in the Bucket's network zone (`eu-central`), and traffic
between them is free. With `--proxy-downloads`, files then reach clients
through the server, whose traffic allowance is separate from the Bucket's.
### Connecting Zotero
defter requires **Zotero 10** (desktop). Earlier versions are not supported.
defter syncs **My Library** only. Group libraries stay on zotero.org: a
Zotero that syncs with defter no longer syncs its groups, and asks whether to
keep or remove the local copy of each.
> [!WARNING]
> Connect defter only to a **new, empty data directory**. Zotero treats
> logging in to defter as switching accounts: if the data directory was
> synced with another account (zotero.org) before, Zotero **deletes its local
> libraries** and restarts; it does not combine them with the defter library.
> Moving an existing zotero.org library to defter is not supported yet.
Use a separate Zotero profile with its own data directory, so the zotero.org
library stays untouched in the default profile:
1. Close Zotero, back up the existing data directory (Settings → Advanced →
Files and Folders → Show Data Directory) and profile, then start the
profile manager with `zotero -P` (macOS:
`/Applications/Zotero.app/Contents/MacOS/zotero -P`) and create a profile,
e.g. `defter`.
2. In the new profile, Settings → Advanced → Files and Folders → Data
Directory Location → Custom: a new, empty directory (e.g.
`~/Zotero-defter`; check the path, a typo creates another empty one), then
restart Zotero. Do not log in to zotero.org in this profile.
3. Settings → Advanced → **Config Editor** (accept the warning), then set:
```
extensions.zotero.api.url = https://defter.example.org/api/
extensions.zotero.streaming.enabled = false
```
Keep the trailing slash: Zotero appends paths such as `keys/sessions`
directly. defter has no streaming server, hence the second setting.
4. Restart Zotero.
5. Settings → Sync → **Log In**. Zotero opens defter's login page in the
browser; enter an API key that may write and click Approve. Zotero picks
up the key within seconds; the login request expires after 10 minutes.
6. Keep file syncing set to **Zotero** (Settings → Sync → File Syncing):
attachment files then go to defter, and through it to the bucket.
Run one profile at a time; start the one you need with `zotero -P <name>`,
or from the profile manager.
To point a profile back at zotero.org, reset both settings in the Config
Editor (right-click → Reset) and restart Zotero; logging in then switches
accounts again, which deletes the local defter library (it stays on the
server).
### commonmeta sync
`commonmeta sync` enhances the items of a Zotero library with what the
commonmeta store knows about them. defter serves the API it uses under
`/api`; [`docs/commonmeta-rs-changes.md`](docs/commonmeta-rs-changes.md)
lists the commonmeta-rs changes needed for it to target a remote server (it
currently only talks to `http://localhost:23119/api`).
Until then, `commonmeta sync` works through Zotero desktop, which syncs the
changes to defter. This also fills a new, empty defter library for testing,
for instance with the works of an ORCID record:
1. Start Zotero in the profile connected to defter. Only that profile may be
running: the local API belongs to whichever Zotero is open, so with the
default profile open, commonmeta writes to the zotero.org library.
2. In that profile, enable Settings → Advanced → **Allow other applications
on this computer to communicate with Zotero**. The setting is per profile,
so a new profile has it off (commonmeta then reports "Zotero's local API is
off"). Check with
`curl -s -o /dev/null -w "%{http_code}\n" http://localhost:23119/api/`,
which prints `200`.
3. Look first, then write a few works, with up to 5 PDFs:
```sh
commonmeta sync --orcid <your-orcid> --dry-run
commonmeta sync --orcid <your-orcid> --limit 25 --pdf --max-pdfs 5
```
The first write asks for permission in a Zotero dialog: choose **Always
Allow**. Each Zotero profile asks once, even after a key was granted in
another.
4. Zotero syncs the new items and PDFs to defter (Settings → Sync → Sync
Now, or the sync button). defter's log then shows `POST
/api/users/<id>/items`, then the file uploads under `/file` and
`/uploads/`; its dashboard counts the items and files.
## Development
```sh
cargo test # unit and API tests
cargo run -- start # http://127.0.0.1:8189, files on local disk
```
Building needs only Rust. Pages are askama templates in `templates/`,
compiled into the binary, so a template change needs a rebuild (`cargo run`
does it). Their stylesheet, `ui/dist/app.css`, is built from `ui/app.css` by
the standalone [Tailwind CLI](https://tailwindcss.com/docs/installation/tailwind-cli)
and committed. To change styling, install it (`brew install tailwindcss`, or
a release binary with `TAILWINDCSS=/path/to/tailwindcss`); the build then
regenerates the stylesheet, and warns when it can't. The script is the
vendored htmx (`ui/vendor/`, see its README for updating) plus `ui/main.js`.
To run against real S3 storage locally, use [RustFS](https://rustfs.com):
```sh
docker run -d --rm --name rustfs -p 9000:9000 \
-e RUSTFS_ACCESS_KEY=defter -e RUSTFS_SECRET_KEY=defter-secret rustfs/rustfs
curl --aws-sigv4 "aws:amz:us-east-1:s3" --user defter:defter-secret \
-X PUT http://127.0.0.1:9000/zotero # create the bucket
DEFTER_S3_BUCKET=zotero DEFTER_S3_ENDPOINT=http://127.0.0.1:9000 \
DEFTER_S3_REGION=us-east-1 DEFTER_S3_ACCESS_KEY_ID=defter \
DEFTER_S3_SECRET_ACCESS_KEY=defter-secret cargo run -- start
DEFTER_KEY=$(cargo run -q -- key) scripts/smoke-s3.sh # file round trip
```
### Layout
| `src/main.rs` | CLI and startup |
| `src/app.rs` | state, router, gzip and logging middleware |
| `src/zotero.rs` | Zotero API handlers |
| `src/auth.rs` | keys, login sessions, expiring tokens |
| `src/store.rs`, `migrations/` | library storage in SQLite |
| `src/query.rs` | item listing parameters |
| `src/db.rs` | connections and migrations |
| `src/storage.rs` | attachment files via `object_store` |
| `src/ui.rs`, `templates/` | web UI pages (askama) |
| `ui/` | the pages' stylesheet (Tailwind) and script (vendored htmx) |
## License
MIT. Includes code adapted from zhost (MIT), see `NOTICE`.