defter
A self-hosted Zotero sync and file storage server: keep one Zotero library in
sync across your machines — metadata and attachment files such as PDFs —
without Zotero Storage. Metadata lives in SQLite, files in S3-compatible object
storage (Hetzner Object Storage, AWS S3, Cloudflare R2, RustFS, …). Stock Zotero 10 clients sync against
it, and commonmeta's sync
command can enrich the library through the same API.
Status: early development (0.3). The sync protocol, file storage and query API are implemented and tested; interfaces may still change.
defter builds on zhost (MIT, see
NOTICE), which worked out the subset of the Zotero Web API v3 a client
needs. defter swaps PostgreSQL for SQLite (rusqlite), rust-s3 for the
object_store crate, and adds a web UI and the endpoints commonmeta sync
uses.
How it works
- Client redirect, no patching. Zotero reads its sync server from the
hidden pref
extensions.zotero.api.url; pointing it at defter redirects sync and login. - Server. A Rust/axum service. Library
objects are stored as opaque JSON keyed by
(kind, key), with the columns queries need derived on every write. One library version counter, bumped by each write, keeps the client'ssincereads andIf-Unmodified-Since-Versionwrites coherent. - Files. Uploads stream through defter, which checks the md5 while writing them to the bucket; downloads redirect to a short-lived pre-signed URL, so the bytes come straight from the bucket.
- Auth. One user, one library. API keys are configured, each read/write or read-only. The Zotero client gets a read/write key through its browser login, which you approve on defter's login page by entering that key.
- Web UI. A dashboard (library statistics, storage, connection instructions), shown after signing in with an API key, and the login consent page for Zotero, rendered on the server from askama templates and styled with Tailwind CSS. htmx refreshes the dashboard in place; every page works without JavaScript and is served under a strict Content-Security-Policy.
The full contract is in docs/api.md.
Running
Every flag has a DEFTER_* environment variable; see env.example and
defter start --help. S3 credentials also fall back to the standard AWS_*
variables. Without --s3-bucket, files go to <data-dir>/files — fine for
development, not meant for production.
API keys
The simplest setup is one read/write key in the environment, e.g. in
/etc/defter/env:
DEFTER_API_KEY=<output When Zotero logs in, it opens defter's login page in the browser; you approve
the login by entering that key, and Zotero receives it. Anyone can start a
login, so the key is what protects the library: generate it, don't choose it
(defter warns about keys shorter than 32 characters), and keep the file
readable by root and the defter group only.
To give scripts or agents read-only access as well, provision keys as files instead, each read/write or read-only:
With no key configured at all, defter generates a read/write key in
<data-dir>/api-key; defter key prints it.
Deployment
defter listens on 127.0.0.1:8189; put it behind a reverse proxy with HTTPS
and set --public-url to the proxy's address. The data API is protected by
keys and safe to expose, and so is /login, which hands out a key only to
whoever enters one. The dashboard (/) shows a sign-in form until an API key
is entered (a read-only key is enough), and /ui/status takes the key as a
header. Behind an SSO proxy, add --login-authorized-user you@example.org to
also require its identity on login approval.
Clients download from pre-signed URLs on --s3-endpoint, so the bucket
endpoint must be reachable from them; otherwise add --proxy-downloads.
Debian with systemd
As root, with Rust installed through rustup:
# the binary, straight to /usr/local/bin (the service cannot run one from ~);
# chmod in case a restrictive umask (e.g. 027) made it unreadable to defter
# env.example and defter.service from this repository:
# the S3 keys, one line each, root:defter 0640:
# /etc/defter/s3-access-key-id, /etc/defter/s3-secret-access-key
&&
To upgrade: cargo install --locked --root /usr/local defter and
chmod 0755 /usr/local/bin/defter again, then systemctl restart defter.
A binary the defter user cannot execute makes systemd fail with status
203/EXEC (behind Cloudflare: error 525).
defter.service runs defter as the defter user with its
data in /var/lib/defter (systemd's StateDirectory, which defter picks up
without --data-dir) and the rest of the file system read-only.
Sharing port 443 with dragoman
On a machine where dragoman
already serves HTTPS on port 443 behind a CDN, defter needs no port, proxy or
certificate of its own: dragoman's --sni-route passes connections for
defter's hostname, still encrypted, to defter, which serves HTTPS on a
loopback port with the same CDN origin certificate (if it covers defter's
hostname, e.g. *.example.org).
# dragoman (/etc/default/commonmeta, which its service reads), then restart it:
DRAGOMAN_SNI_ROUTES=defter.example.org=127.0.0.1:8443
# defter (/etc/defter/env):
DEFTER_PORT=8443
DEFTER_PUBLIC_URL=https://defter.example.org
DEFTER_TLS_CERT=/etc/defter/origin.pem
DEFTER_TLS_KEY=/etc/defter/origin-key.pem
Give defter its own copy of the certificate and key, readable by its group:
With the hostname proxied by the CDN, its limits apply to uploads through
defter (Cloudflare: 100 MB per request on the Free and Pro plans), so set
DEFTER_MAX_UPLOAD_MB=100 there; downloads come from the bucket directly.
Without a CDN in front of defter's hostname, let defter get its own certificate
from Let's Encrypt, as dragoman does, instead of giving it DEFTER_TLS_CERT and
DEFTER_TLS_KEY:
# defter (/etc/defter/env):
DEFTER_PORT=8443
DEFTER_PUBLIC_URL=https://defter.example.org
DEFTER_DOMAIN=defter.example.org
DEFTER_EMAIL=you@example.org
The TLS-ALPN-01 challenge needs no port 80: Let's Encrypt connects to port 443
with defter's hostname, and dragoman passes that through like any other connection.
So port 443 must be open to everyone, not only to a CDN, and the hostname's DNS
must point straight at the server. Certificates renew by themselves and are kept
in <data-dir>/acme (DEFTER_ACME_CACHE). Set DEFTER_ACME_STAGING=true to try
the setup without Let's Encrypt's rate limits; browsers don't trust those
certificates, so delete the cache directory when switching back.
The server must also reach Let's Encrypt. Its ACME client tries addresses in the
order the system returns them and has no timeout of its own, so where IPv6 is
configured but broken, an order stalls without an error while HTTPS fails with
access denied. defter checks this at startup and logs a warning, and warns again
if there is no certificate after five minutes. Fix IPv6, or prefer IPv4 by adding
precedence ::ffff:0:0/96 100 to /etc/gai.conf. To see the order's progress, set
RUST_LOG=defter=info,rustls_acme=info.
Hetzner Object Storage
Hetzner Object Storage is the intended storage, for a defter running at home as well as for one later deployed on a Hetzner Cloud server.
-
In Hetzner Console, create a project, a private Bucket in one location (
fsn1Falkenstein,nbg1Nuremberg orhel1Helsinki), and S3 credentials for the project (shown once: save both keys). Keys are valid for every Bucket of the project, so give defter a project of its own. -
Start defter with the location as both endpoint and region:
defter addresses Hetzner Buckets by host name (
https://<bucket>.fsn1.your-objectstorage.com), which its pre-signed download URLs need. -
Check the round trip:
DEFTER_KEY=$(defter key) scripts/smoke-s3.sh.
Costs (Hetzner's prices, November 2026, excl. VAT): €6.49 a month includes 1 TB of storage and 1 TB of outgoing traffic; uploads and S3 requests are free. Downloads by pre-signed URL count as outgoing traffic. Objects under 64 KB are billed as 64 KB.
For a later deployment on a Hetzner Cloud server, put the server in an EU
location: it is then in the Bucket's network zone (eu-central), and traffic
between them is free. With --proxy-downloads, files then reach clients
through the server, whose traffic allowance is separate from the Bucket's.
Signing in with Keycloak
API keys are always there, and an OpenID Connect provider such as Keycloak can be
added beside them. In a Keycloak realm, create a confidential client with standard
flow on and <public URL>/ui/oidc/callback as its valid redirect URI, and map the
account's ORCID iD to a token claim, orcid by default. Then:
DEFTER_OIDC_ISSUER=https://auth.example.org/realms/<realm>
DEFTER_OIDC_CLIENT_ID=defter
DEFTER_OIDC_CLIENT_SECRET_FILE=/etc/defter/oidc-client-secret
DEFTER_OIDC_ALLOWED_ORCIDS=0000-0003-1419-2405
The claim decides who gets in, so accounts must not be able to set it themselves.
Let it come from ORCID's own login, and in Keycloak's realm settings make the orcid
attribute editable by admins only, with unmanaged attributes off or admin-only. Test
it with a throwaway account if the realm allows registration.
With a provider configured, an API key can only be typed into the web pages if
DEFTER_API_KEY is set. Leave it out to make Keycloak the only way in. Keys from
DEFTER_API_KEY_FILE, DEFTER_KEYS or defter key keep working for the data API, such
as commonmeta sync. Without a provider, typing a key always works.
Only accounts with an ORCID iD on that list get in. They can use the dashboard, hide
the public profile, and approve Zotero logins with one click, which hands Zotero the
read/write key defter key prints.
Without DEFTER_ORCID, the first allowed account to sign in owns the public profile.
Its iD is saved in the database, and the profile stops being shown if the iD leaves
the list. DEFTER_ORCID takes precedence.
Signing out also ends the session at the provider, and so in other applications using
it. Add <public URL>/ to the client's valid post logout redirect URIs, or Keycloak
refuses. Signing out of Keycloak elsewhere, or disabling an account there, does not reach
defter: its session lasts until it expires after seven days, or defter restarts.
Taking an iD off the list needs a restart, which also ends every session. If that account approved a Zotero login, Zotero holds the read/write key: change it as well.
Connecting Zotero
defter requires Zotero 10 (desktop). Earlier versions are not supported.
defter syncs My Library only. Group libraries stay on zotero.org: a Zotero that syncs with defter no longer syncs its groups, and asks whether to keep or remove the local copy of each.
[!WARNING] Connect defter only to a new, empty data directory. Zotero treats logging in to defter as switching accounts: if the data directory was synced with another account (zotero.org) before, Zotero deletes its local libraries and restarts; it does not combine them with the defter library. Moving an existing zotero.org library to defter is not supported yet.
Use a separate Zotero profile with its own data directory, so the zotero.org library stays untouched in the default profile:
-
Close Zotero, back up the existing data directory (Settings → Advanced → Files and Folders → Show Data Directory) and profile, then start the profile manager with
zotero -P(macOS:/Applications/Zotero.app/Contents/MacOS/zotero -P) and create a profile, e.g.defter. -
In the new profile, Settings → Advanced → Files and Folders → Data Directory Location → Custom: a new, empty directory (e.g.
~/Zotero-defter; check the path, a typo creates another empty one), then restart Zotero. Do not log in to zotero.org in this profile. -
Settings → Advanced → Config Editor (accept the warning), then set:
extensions.zotero.api.url = https://defter.example.org/api/ extensions.zotero.streaming.enabled = falseKeep the trailing slash: Zotero appends paths such as
keys/sessionsdirectly. defter has no streaming server, hence the second setting. -
Restart Zotero.
-
Settings → Sync → Log In. Zotero opens defter's login page in the browser; enter an API key that may write and click Approve. Zotero picks up the key within seconds; the login request expires after 10 minutes.
-
Keep file syncing set to Zotero (Settings → Sync → File Syncing): attachment files then go to defter, and through it to the bucket.
Run one profile at a time; start the one you need with zotero -P <name>,
or from the profile manager.
To point a profile back at zotero.org, reset both settings in the Config Editor (right-click → Reset) and restart Zotero; logging in then switches accounts again, which deletes the local defter library (it stays on the server).
commonmeta sync
commonmeta sync enhances the items of a Zotero library with what the
commonmeta store knows about them. defter serves the API it uses under
/api; docs/commonmeta-rs-changes.md
lists the commonmeta-rs changes needed for it to target a remote server (it
currently only talks to http://localhost:23119/api).
Until then, commonmeta sync works through Zotero desktop, which syncs the
changes to defter. This also fills a new, empty defter library for testing,
for instance with the works of an ORCID record:
-
Start Zotero in the profile connected to defter. Only that profile may be running: the local API belongs to whichever Zotero is open, so with the default profile open, commonmeta writes to the zotero.org library.
-
In that profile, enable Settings → Advanced → Allow other applications on this computer to communicate with Zotero. The setting is per profile, so a new profile has it off (commonmeta then reports "Zotero's local API is off"). Check with
curl -s -o /dev/null -w "%{http_code}\n" http://localhost:23119/api/, which prints200. -
Look first, then write a few works, with up to 5 PDFs:
The first write asks for permission in a Zotero dialog: choose Always Allow. Each Zotero profile asks once, even after a key was granted in another.
-
Zotero syncs the new items and PDFs to defter (Settings → Sync → Sync Now, or the sync button). defter's log then shows
POST /api/users/<id>/items, then the file uploads under/fileand/uploads/; its dashboard counts the items and files.
Public profile
With --orcid <your-orcid> (DEFTER_ORCID), defter publishes a profile at
/<your-orcid>, without a key: your ORCID record, read by commonmeta and
fetched at most hourly, and your works, which can be searched and narrowed by
year and type, or exported as BibTeX, RIS, CSL-JSON and more. It is available in
English and German.
Every work in My Publications is shown, as long as it is at the top level
and not in the trash, with its title, creators, date, abstract, publication,
language, rights and DOI. Its tags, notes and files are not, nor are they exported. The rest of the
library stays private. The profile opens on up to five works with the manual tag
featured, newest first, if there are any; unticking "Featured only" shows them all.
A work with a PDF attached links to where that PDF was
published, never to the file stored in defter or Zotero; a PDF uploaded from disk
has no such place and is not linked.
The dashboard links to the profile and, signed in with a read/write key, can
hide it; the address then answers 404. Unlike defter's other pages, the
profile may be indexed by search engines.
Development
Building needs only Rust. Pages are askama templates in templates/,
compiled into the binary, so a template change needs a rebuild (cargo run
does it). Their stylesheet, ui/dist/app.css, is built from ui/app.css by
the standalone Tailwind CLI
and committed. To change styling, install it (brew install tailwindcss, or
a release binary with TAILWINDCSS=/path/to/tailwindcss); the build then
regenerates the stylesheet, and warns when it can't. The script is the
vendored htmx (ui/vendor/, see its README for updating) plus ui/main.js.
To run against real S3 storage locally, use RustFS:
DEFTER_S3_BUCKET=zotero DEFTER_S3_ENDPOINT=http://127.0.0.1:9000 \
DEFTER_S3_REGION=us-east-1 DEFTER_S3_ACCESS_KEY_ID=defter \
DEFTER_S3_SECRET_ACCESS_KEY=defter-secret
DEFTER_KEY= Layout
| Path | |
|---|---|
src/main.rs |
CLI and startup |
src/app.rs |
state, router, gzip and logging middleware |
src/zotero.rs |
Zotero API handlers |
src/auth.rs |
keys, login sessions, expiring tokens |
src/store.rs, migrations/ |
library storage in SQLite |
src/query.rs |
item listing parameters |
src/db.rs |
connections and migrations |
src/storage.rs |
attachment files via object_store |
src/ui.rs, templates/ |
web UI pages (askama) |
src/profile.rs |
public profile: ORCID record and what the page shows |
ui/ |
the pages' stylesheet (Tailwind) and script (vendored htmx) |
License
MIT. Includes code adapted from zhost (MIT), see NOTICE.