defter 0.5.2

Self-hosted Zotero sync and file storage server, with metadata in SQLite and attachments in S3.
defter-0.5.2 is not a library.

defter

A self-hosted Zotero sync and file storage server: keep one Zotero library in sync across your machines — metadata and attachment files such as PDFs — complementing Zotero Storage as a distributed option, e.g. for universities. Metadata lives in SQLite, files in S3-compatible object storage (Hetzner Object Storage, AWS S3, Cloudflare R2, RustFS, …). Stock Zotero 10 clients sync against it, and commonmeta's sync command can enrich the library through the same API.

Status: early development (0.5.x). The sync protocol, file storage and query API are implemented and tested; interfaces may still change.

defter builds on zhost (MIT, see NOTICE), which worked out the subset of the Zotero Web API v3 a client needs. defter swaps PostgreSQL for SQLite (rusqlite), rust-s3 for the object_store crate, and adds a web UI and the endpoints commonmeta sync uses.

How it works

  • Client redirect, no patching. Zotero reads its sync server from the hidden pref extensions.zotero.api.url; pointing it at defter redirects sync and login.
  • Server. A Rust/axum service. Each library is a SQLite database of its own (libraries/u<id>.sqlite3), and control.sqlite3 holds the users and what is known about each library. Library objects are stored as opaque JSON keyed by (kind, key), with the columns queries need derived on every write. One version counter per library, bumped by each write, keeps the client's since reads and If-Unmodified-Since-Version writes coherent.
  • Files. Uploads stream through defter, which checks the md5 while writing them to the bucket; downloads redirect to a short-lived pre-signed URL, so the bytes come straight from the bucket.
  • People. Each person has a library of their own, signs in through an OpenID Connect provider such as Keycloak with their ORCID iD, and gets a key of their own for each device when they approve its Zotero login. Admins invite people by ORCID iD and set storage quotas. Keys configured on the server are the owner's.
  • Web UI. A dashboard (library statistics, storage, connection instructions), shown after signing in with an API key, and the login consent page for Zotero, rendered on the server from askama templates and styled with Tailwind CSS. htmx refreshes the dashboard in place; every page works without JavaScript and is served under a strict Content-Security-Policy.

The full contract is in docs/api.md.

Running

cargo install --path .          # or: cargo install defter
defter start --public-url https://zotero.example.org \
  --s3-bucket <your-bucket> \
  --s3-endpoint https://fsn1.your-objectstorage.com --s3-region fsn1 \
  --s3-access-key-id-file /etc/defter/s3-access-key-id \
  --s3-secret-access-key-file /etc/defter/s3-secret-access-key

Every flag has a DEFTER_* environment variable; see env.example and defter start --help. S3 credentials also fall back to the standard AWS_* variables. Without --s3-bucket, files go to <data-dir>/files — fine for development, not meant for production.

API keys

The simplest setup is one read/write key in the environment, e.g. in /etc/defter/env:

DEFTER_API_KEY=<output of: openssl rand -hex 32>

When Zotero logs in, it opens defter's login page in the browser; you approve the login by entering that key, and Zotero receives it. Anyone can start a login, so the key is what protects the library: generate it, don't choose it (defter warns about keys shorter than 32 characters), and keep the file readable by root and the defter group only.

To give scripts or agents read-only access as well, provision keys as files instead, each read/write or read-only:

openssl rand -hex 32 > app-key   # read/write, for Zotero
openssl rand -hex 32 > cli-key   # read-only
defter start --keys rw:/path/app-key,ro:/path/cli-key …

With no key configured at all, defter generates a read/write key in <data-dir>/api-key; defter key prints it.

Deployment

defter listens on 127.0.0.1:8189; put it behind a reverse proxy with HTTPS and set --public-url to the proxy's address. The data API is protected by keys and safe to expose, and so is /login, which hands out a key only to whoever enters one. The dashboard (/) shows a sign-in form until an API key is entered (a read-only key is enough), and /ui/status takes the key as a header. Behind an SSO proxy, add --login-authorized-user you@example.org to also require its identity on login approval.

Clients download from pre-signed URLs on --s3-endpoint, so the bucket endpoint must be reachable from them; otherwise add --proxy-downloads.

Debian with systemd

As root, with Rust installed through rustup:

# the binary, straight to /usr/local/bin (the service cannot run one from ~);
# chmod in case a restrictive umask (e.g. 027) made it unreadable to defter
cargo install --locked --root /usr/local defter
chmod 0755 /usr/local/bin/defter

useradd --system --no-create-home --shell /usr/sbin/nologin defter
install -d -m 0750 -o root -g defter /etc/defter
# env.example and defter.service from this repository:
install -m 0640 -o root -g defter env.example /etc/defter/env   # then edit it
# the S3 keys, one line each, root:defter 0640:
#   /etc/defter/s3-access-key-id, /etc/defter/s3-secret-access-key
install -m 0644 defter.service /etc/systemd/system/defter.service
systemctl daemon-reload && systemctl enable --now defter
journalctl -u defter -f

To upgrade: cargo install --locked --root /usr/local defter and chmod 0755 /usr/local/bin/defter again, then systemctl restart defter. A binary the defter user cannot execute makes systemd fail with status 203/EXEC (behind Cloudflare: error 525). Upgrading from 0.5.0 takes one more step, below.

Upgrading from 0.5.0

defter 0.5.0 kept the library in defter.sqlite3. Later versions keep each library in libraries/, beside control.sqlite3, and refuse to start until defter migrate has moved it. Nothing changes for Zotero clients: the library keeps its versions and its server ID, and its attachment files stay where they are. As root, with the defter-admin alias from "Backups and restore":

systemctl stop defter
cargo install --locked --root /usr/local defter && chmod 0755 /usr/local/bin/defter
defter-admin migrate --dry-run    # what it will do
defter-admin migrate              # backs up first, then moves the library
systemctl start defter

The library database moves to libraries/u<DEFTER_USER_ID>.sqlite3 unchanged. control.sqlite3 gets the server ID, the cached Zotero schema and the public profile's settings; ORCID iDs in DEFTER_OIDC_ALLOWED_ORCIDS other than the owner's are kept as invitations for a later release. An interrupted migration finishes when run again. If DEFTER_DB was set, leave it while migrating; it is not needed afterwards.

To go back to 0.5.0: stop the server, move libraries/u<id>.sqlite3 back to defter.sqlite3, together with its -wal and -shm files if there are any (they hold its last writes after an unclean stop), move control.sqlite3 aside, and install 0.5.0. Profile settings changed since the migration do not carry back. Never start 0.5.0 without moving the library back first: it would start an empty library, and clients would see their library go back to nothing.

defter.service runs defter as the defter user with its data in /var/lib/defter (systemd's StateDirectory, which defter picks up without --data-dir) and the rest of the file system read-only.

Sharing port 443 with dragoman

On a machine where dragoman already serves HTTPS on port 443 behind a CDN, defter needs no port, proxy or certificate of its own: dragoman's --sni-route passes connections for defter's hostname, still encrypted, to defter, which serves HTTPS on a loopback port with the same CDN origin certificate (if it covers defter's hostname, e.g. *.example.org).

# dragoman (/etc/default/commonmeta, which its service reads), then restart it:
DRAGOMAN_SNI_ROUTES=defter.example.org=127.0.0.1:8443

# defter (/etc/defter/env):
DEFTER_PORT=8443
DEFTER_PUBLIC_URL=https://defter.example.org
DEFTER_TLS_CERT=/etc/defter/origin.pem
DEFTER_TLS_KEY=/etc/defter/origin-key.pem

Give defter its own copy of the certificate and key, readable by its group:

install -m 0644 -o root -g defter /etc/dragoman/origin.pem /etc/defter/origin.pem
install -m 0640 -o root -g defter /etc/dragoman/origin-key.pem /etc/defter/origin-key.pem

With the hostname proxied by the CDN, its limits apply to uploads through defter (Cloudflare: 100 MB per request on the Free and Pro plans), so set DEFTER_MAX_UPLOAD_MB=100 there; downloads come from the bucket directly.

Without a CDN in front of defter's hostname, let defter get its own certificate from Let's Encrypt, as dragoman does, instead of giving it DEFTER_TLS_CERT and DEFTER_TLS_KEY:

# defter (/etc/defter/env):
DEFTER_PORT=8443
DEFTER_PUBLIC_URL=https://defter.example.org
DEFTER_DOMAIN=defter.example.org
DEFTER_EMAIL=you@example.org

The TLS-ALPN-01 challenge needs no port 80: Let's Encrypt connects to port 443 with defter's hostname, and dragoman passes that through like any other connection. So port 443 must be open to everyone, not only to a CDN, and the hostname's DNS must point straight at the server. Certificates renew by themselves and are kept in <data-dir>/acme (DEFTER_ACME_CACHE). Set DEFTER_ACME_STAGING=true to try the setup without Let's Encrypt's rate limits; browsers don't trust those certificates, so delete the cache directory when switching back.

The server must also reach Let's Encrypt. Its ACME client tries addresses in the order the system returns them and has no timeout of its own, so where IPv6 is configured but broken, an order stalls without an error while HTTPS fails with access denied. defter checks this at startup and logs a warning, and warns again if there is no certificate after five minutes. Fix IPv6, or prefer IPv4 by adding precedence ::ffff:0:0/96 100 to /etc/gai.conf. To see the order's progress, set RUST_LOG=defter=info,rustls_acme=info.

Hetzner Object Storage

Hetzner Object Storage is the intended storage, for a defter running at home as well as for one later deployed on a Hetzner Cloud server.

  1. In Hetzner Console, create a project, a private Bucket in one location (fsn1 Falkenstein, nbg1 Nuremberg or hel1 Helsinki), and S3 credentials for the project (shown once: save both keys). Keys are valid for every Bucket of the project, so give defter a project of its own.

  2. Start defter with the location as both endpoint and region:

    defter start \
  --s3-bucket <your-bucket> \
  --s3-endpoint https://fsn1.your-objectstorage.com --s3-region fsn1 \
  --s3-access-key-id-file ~/.config/defter/s3-access-key-id \
  --s3-secret-access-key-file ~/.config/defter/s3-secret-access-key

    defter addresses Hetzner Buckets by host name (https://<bucket>.fsn1.your-objectstorage.com), which its pre-signed download URLs need.

  3. Check the round trip: DEFTER_KEY=$(defter key) scripts/smoke-s3.sh.

Costs (Hetzner's prices, November 2026, excl. VAT): €6.49 a month includes 1 TB of storage and 1 TB of outgoing traffic; uploads and S3 requests are free. Downloads by pre-signed URL count as outgoing traffic. Objects under 64 KB are billed as 64 KB.

For a later deployment on a Hetzner Cloud server, put the server in an EU location: it is then in the Bucket's network zone (eu-central), and traffic between them is free. With --proxy-downloads, files then reach clients through the server, whose traffic allowance is separate from the Bucket's.

Backups and restore

defter backs up its databases (control.sqlite3 and every library's) to object storage every 6 hours, under backup/ in the files bucket, and keeps 30 days of backups. Each run takes a consistent snapshot while the server keeps running, checks it, and uploads it gzipped, together with a manifest naming every database at that point. A database that has not changed since the last backup is not uploaded again. The dashboard shows the last backup or its error, and so does /ui/status.

DEFTER_BACKUP_INTERVAL_HOURS=6     # 0 turns scheduled backups off
DEFTER_BACKUP_KEEP_DAYS=30
DEFTER_BACKUP_S3_BUCKET=defter-backups   # optional: a second bucket, same endpoint and keys

A second bucket is worth it: backups in the files bucket do not survive losing that bucket, or credentials that can delete from it. Without S3, backups go to <files-dir>/backup/ on the same disk, which is for development only.

Attachment files are not copied, as they are in object storage already. While scheduled backups run, a deleted or replaced attachment file is moved to deleted/<date>/ rather than removed, and purged after the same 30 days, so a restored library gets back its files as they were. With DEFTER_BACKUP_INTERVAL_HOURS=0 files are removed for good, so backups taken another way (defter backup from cron) restore the library without them.

Run these commands as root, through an alias that runs defter as the defter user with the server's environment and data directory, which systemd otherwise sets (runuser comes with Debian, unlike sudo):

alias defter-admin="runuser -u defter -- env DEFTER_DATA_DIR=/var/lib/defter \
  bash -c 'set -a; . /etc/defter/env; exec defter \"\$@\"' defter"

To back up now, or see what there is (both safe while the server runs):

defter-admin backup
defter-admin backup --list

To restore, stop the server, look at what would be done, then restore:

systemctl stop defter
defter-admin restore                 # the newest backup; --manifest <time> for another
defter-admin restore --yes
systemctl start defter

This restores the owner's library; --library u<id> restores another, and --control the control database. The database it replaces is kept next to it as <name>.sqlite3.before-restore-<time>. A library's new versions are set above the replaced database's and above the highest version the server recorded serving. If neither is known, restore stops and asks for --min-version: a version at least as high as any client has seen, such as the last Last-Modified-Version in the server's log.

After losing the data directory, restore the control database first, then each library: defter-admin restore --control --yes, then defter-admin restore --yes (and --library u<id> --yes for others). The control database knows where each library's attachment files are, and the server refuses to start with libraries but without it. With the library's database gone, its restore may ask for --min-version, as above.

Zotero clients refuse a library whose version goes down, so a restore sets every version in the library above any it can have served: clients see a newer library rather than an error. They still hold whatever they synced after the backup. Afterwards, in Zotero's Settings → Sync → Reset, pick one:

  • Replace Online Library on the client with the most complete library, when the restore was to recover lost data: its newer changes go back up.
  • Restore from Online Library on every client, when the restore was to undo bad changes: they take the restored library as it is.

Practise it once on a copy, with a throwaway Zotero profile, before you need it.

Signing in with Keycloak

API keys are always there, and an OpenID Connect provider such as Keycloak can be added beside them. In a Keycloak realm, create a confidential client with standard flow on and <public URL>/ui/oidc/callback as its valid redirect URI, and map the account's ORCID iD to a token claim, orcid by default. Then:

DEFTER_OIDC_ISSUER=https://auth.example.org/realms/<realm>
DEFTER_OIDC_CLIENT_ID=defter
DEFTER_OIDC_CLIENT_SECRET_FILE=/etc/defter/oidc-client-secret
DEFTER_ADMIN_ORCIDS=0000-0003-1419-2405

The claim decides who gets in, so accounts must not be able to set it themselves. Let it come from ORCID's own login, and in Keycloak's realm settings make the orcid attribute editable by admins only, with unmanaged attributes off or admin-only. Test it with a throwaway account if the realm allows registration.

With a provider configured, an API key can only be typed into the web pages if DEFTER_API_KEY is set. Leave it out to make Keycloak the only way in. Keys from DEFTER_API_KEY_FILE, DEFTER_KEYS or defter key keep working for the data API, such as commonmeta sync. Without a provider, typing a key always works.

Who gets in: users, who keep their ORCID iD; people an admin invited, who become users with a library of their own when they first sign in; and the admins in DEFTER_ADMIN_ORCIDS, who need no invitation. Everyone else is turned away. The owner of the server (DEFTER_USER_ID) is the user with DEFTER_ORCID, or the first admin to sign in while the owner has no ORCID iD yet. Without DEFTER_ADMIN_ORCIDS, the owner is the admin. iDs in DEFTER_OIDC_ALLOWED_ORCIDS, from before there were invitations, are invited on the first start, and the setting can go afterwards; an invitation withdrawn later stays withdrawn. Without DEFTER_ORCID or admins, a single iD there is the owner's, as it was.

Signed in, everyone has a dashboard of their own: their library, the storage it takes, their public profile and their devices. Approving a Zotero login with one click mints a key for that device, which the dashboard lists and can revoke; defter keeps only its hash. Unlinking the account in Zotero revokes it too.

Signing out also ends the session at the provider, and so in other applications using it. Add <public URL>/ to the client's valid post logout redirect URIs, or Keycloak refuses. Signing out of Keycloak elsewhere, or disabling an account there, does not reach defter: its session lasts until it expires after seven days, or defter restarts.

Disabling a user on the admin page ends their sessions, revokes their devices' keys and hides their profile at once. Their library stays. The owner cannot be disabled: their keys are configured on the server.

People and storage

Admins see an Admin link on the dashboard, to /ui/admin:

  • Invitations: invite an ORCID iD, with a note for yourself. The person signs in through Keycloak and has an empty library of their own, to sync with a fresh Zotero profile pointed at defter (see "Connecting Zotero"). An unused invitation can be withdrawn.
  • Users: each one's storage use, devices and quota, which can be changed, and disabling or enabling them.
  • Backups: how the last one went.

Each user's attachment files count against a quota, 2 GB unless an admin sets theirs (DEFTER_DEFAULT_QUOTA_MB); the owner has none unless set. An upload past it is refused as zotero.org refuses it, and Zotero says the storage is full. A new user's files go to users/<id>/files/ in the bucket; the owner's stay where they are.

Connecting Zotero

defter requires Zotero 10 (desktop). Earlier versions are not supported.

defter syncs My Library only. Group libraries stay on zotero.org: a Zotero that syncs with defter no longer syncs its groups, and asks whether to keep or remove the local copy of each.

[!WARNING] Connect defter only to a new, empty data directory. Zotero treats logging in to defter as switching accounts: if the data directory was synced with another account (zotero.org) before, Zotero deletes its local libraries and restarts; it does not combine them with the defter library. Moving an existing zotero.org library to defter is not supported yet.

Use a separate Zotero profile with its own data directory, so the zotero.org library stays untouched in the default profile:

  1. Close Zotero, back up the existing data directory (Settings → Advanced → Files and Folders → Show Data Directory) and profile, then start the profile manager with zotero -P (macOS: /Applications/Zotero.app/Contents/MacOS/zotero -P) and create a profile, e.g. defter.

  2. In the new profile, Settings → Advanced → Files and Folders → Data Directory Location → Custom: a new, empty directory (e.g. ~/Zotero-defter; check the path, a typo creates another empty one), then restart Zotero. Do not log in to zotero.org in this profile.

  3. Settings → Advanced → Config Editor (accept the warning), then set:

    extensions.zotero.api.url = https://defter.example.org/api/
extensions.zotero.streaming.enabled = false

    Keep the trailing slash: Zotero appends paths such as keys/sessions directly. defter has no streaming server, hence the second setting.

  4. Restart Zotero.

  5. Settings → Sync → Log In. Zotero opens defter's login page in the browser; enter an API key that may write and click Approve. Zotero picks up the key within seconds; the login request expires after 10 minutes.

  6. Keep file syncing set to Zotero (Settings → Sync → File Syncing): attachment files then go to defter, and through it to the bucket.

Run one profile at a time; start the one you need with zotero -P <name>, or from the profile manager.

To point a profile back at zotero.org, reset both settings in the Config Editor (right-click → Reset) and restart Zotero; logging in then switches accounts again, which deletes the local defter library (it stays on the server).

commonmeta sync

commonmeta sync enhances the items of a Zotero library with what the commonmeta store knows about them. defter serves the API it uses under /api; docs/commonmeta-rs-changes.md lists the commonmeta-rs changes needed for it to target a remote server (it currently only talks to http://localhost:23119/api).

Until then, commonmeta sync works through Zotero desktop, which syncs the changes to defter. This also fills a new, empty defter library for testing, for instance with the works of an ORCID record:

  1. Start Zotero in the profile connected to defter. Only that profile may be running: the local API belongs to whichever Zotero is open, so with the default profile open, commonmeta writes to the zotero.org library.

  2. In that profile, enable Settings → Advanced → Allow other applications on this computer to communicate with Zotero. The setting is per profile, so a new profile has it off (commonmeta then reports "Zotero's local API is off"). Check with curl -s -o /dev/null -w "%{http_code}\n" http://localhost:23119/api/, which prints 200.

  3. Look first, then write a few works, with up to 5 PDFs:

    commonmeta sync --orcid <your-orcid> --dry-run
commonmeta sync --orcid <your-orcid> --limit 25 --pdf --max-pdfs 5

    The first write asks for permission in a Zotero dialog: choose Always Allow. Each Zotero profile asks once, even after a key was granted in another.

  4. Zotero syncs the new items and PDFs to defter (Settings → Sync → Sync Now, or the sync button). defter's log then shows POST /api/users/<id>/items, then the file uploads under /file and /uploads/; its dashboard counts the items and files.

Public profile

Every user with an ORCID iD can publish a profile at /<their-orcid>, without a key. The owner's is shown unless hidden; everyone else's starts hidden until they show it on their dashboard. A profile has their ORCID record, read by commonmeta and fetched at most hourly, and your works, which can be searched and narrowed by year and type, or exported as BibTeX, RIS, CSL-JSON and more. It is available in English and German.

Every work in My Publications is shown, as long as it is at the top level and not in the trash, with its title, creators, date, abstract, publication, language, rights and DOI. Its tags, notes and files are not, nor are they exported. The rest of the library stays private. The profile opens on up to five works with the manual tag featured, newest first, if there are any; unticking "Featured only" shows them all. A work with a PDF attached links to where that PDF was published, never to the file stored in defter or Zotero; a PDF uploaded from disk has no such place and is not linked. The dashboard links to the profile and, signed in with a read/write key, can hide or show it; while hidden, or for a disabled user, the address answers 404. Unlike defter's other pages, the profile may be indexed by search engines.

Development

cargo test                 # unit and API tests
cargo run -- start         # http://127.0.0.1:8189, files on local disk

Building needs only Rust. Pages are askama templates in templates/, compiled into the binary, so a template change needs a rebuild (cargo run does it). Their stylesheet, ui/dist/app.css, is built from ui/app.css by the standalone Tailwind CLI and committed. To change styling, install it (brew install tailwindcss, or a release binary with TAILWINDCSS=/path/to/tailwindcss); the build then regenerates the stylesheet, and warns when it can't. The script is the vendored htmx (ui/vendor/, see its README for updating) plus ui/main.js.

To run against real S3 storage locally, use RustFS:

docker run -d --rm --name rustfs -p 9000:9000 \
  -e RUSTFS_ACCESS_KEY=defter -e RUSTFS_SECRET_KEY=defter-secret rustfs/rustfs
curl --aws-sigv4 "aws:amz:us-east-1:s3" --user defter:defter-secret \
  -X PUT http://127.0.0.1:9000/zotero          # create the bucket
DEFTER_S3_BUCKET=zotero DEFTER_S3_ENDPOINT=http://127.0.0.1:9000 \
DEFTER_S3_REGION=us-east-1 DEFTER_S3_ACCESS_KEY_ID=defter \
DEFTER_S3_SECRET_ACCESS_KEY=defter-secret cargo run -- start
DEFTER_KEY=$(cargo run -q -- key) scripts/smoke-s3.sh   # file round trip

Layout

Path
src/main.rs CLI and startup
src/app.rs state, router, gzip and logging middleware
src/zotero.rs Zotero API handlers
src/auth.rs keys, the access decision, login sessions, expiring tokens
src/control.rs, migrations/control/ the control database: users, library state
src/registry.rs opening each library's database on demand
src/migrate.rs defter migrate, from 0.5.0's single database
src/store.rs, migrations/ library storage in SQLite
src/query.rs item listing parameters
src/db.rs connections and migrations
src/storage.rs attachment files via object_store
src/backup.rs backups of the databases, and restoring them
src/ui.rs, templates/ web UI pages (askama)
src/profile.rs public profile: ORCID record and what the page shows
ui/ the pages' stylesheet (Tailwind) and script (vendored htmx)

License

MIT. Includes code adapted from zhost (MIT), see NOTICE.