# defter
A self-hosted Zotero sync and file storage server: keep one Zotero library in
sync across your machines — metadata **and** attachment files such as PDFs —
complementing [Zotero Storage](https://www.zotero.org/storage) as a distributed option, e.g. for universities. Metadata lives in SQLite, files in S3-compatible object
storage (Hetzner Object Storage, AWS S3, Cloudflare R2, RustFS, …). Stock Zotero 10 clients sync against
it, and [commonmeta](https://codeberg.org/front-matter/commonmeta-rs)'s `sync`
can enrich each library on the server, every day, for the users who opt in.
defter works **in parallel with zotero.org and Zotero Storage**, never
replacing them: with the "defter for Zotero" plugin, Zotero keeps syncing with
zotero.org, and the plugin also syncs the same library with defter. Signing in
to defter therefore needs a zotero.org account, whose user ID becomes yours
here.
**Status: early development (0.7.x).** The sync protocol, file storage and query
API are implemented and tested; interfaces may still change. defter builds on [zhost](https://github.com/mulatta/zhost) (MIT, see
`NOTICE`), which worked out the subset of the Zotero Web API v3 a client
needs. defter swaps PostgreSQL for SQLite (rusqlite), `rust-s3` for the
`object_store` crate, and adds a web UI, the endpoints `commonmeta sync` uses and a server-side commonmeta sync.
## How it works
- **Client redirect, no patching.** Zotero reads its sync server from the
hidden pref `extensions.zotero.api.url`; pointing it at defter redirects
sync and login.
- **Server.** A Rust/[axum](https://github.com/tokio-rs/axum) service. Each
library is a SQLite database of its own (`libraries/u<id>.sqlite3`), and
`control.sqlite3` holds the users and what is known about each library.
Library objects are stored as opaque JSON keyed by `(kind, key)`, with the
columns queries need derived on every write. One version counter per library,
bumped by each write, keeps the client's `since` reads and
`If-Unmodified-Since-Version` writes coherent.
- **Files.** Uploads stream through defter, which checks the md5 while writing
them to the bucket; downloads redirect to a short-lived pre-signed URL, so
the bytes come straight from the bucket.
- **People.** Each person has a library of their own, signs in through an
OpenID Connect provider such as Keycloak with their ORCID iD, and gets a key
of their own for each device when they approve its Zotero login. Their user
ID is their zotero.org user ID, so one Zotero can sync the same library with
both. Admins
invite people by ORCID iD, or a Keycloak group or role decides who gets in,
and set storage quotas. Keys configured on the server are the owner's.
- **Web UI.** A dashboard (library statistics, storage, connection
instructions), shown after signing in with an API key, and the login
consent page for Zotero, rendered on the server from
[askama](https://askama.rs) templates and styled with Tailwind CSS.
[htmx](https://htmx.org) refreshes the dashboard in place; every page works
without JavaScript and is served under a strict Content-Security-Policy.
The full contract is in [`docs/api.md`](docs/api.md).
## Running
```sh
cargo install --path . # or: cargo install defter
defter start --public-url https://zotero.example.org \
--user-id <your zotero.org user ID> \
--s3-bucket <your-bucket> \
--s3-endpoint https://fsn1.your-objectstorage.com --s3-region fsn1 \
--s3-access-key-id-file /etc/defter/s3-access-key-id \
--s3-secret-access-key-file /etc/defter/s3-secret-access-key
```
Every flag has a `DEFTER_*` environment variable; see `env.example` and
`defter start --help`. S3 credentials also fall back to the standard `AWS_*`
variables. Without `--s3-bucket`, files go to `<data-dir>/files` — fine for
development, not meant for production.
### API keys
The simplest setup is one read/write key in the environment, e.g. in
`/etc/defter/env`:
```sh
DEFTER_API_KEY=<output of: openssl rand -hex 32>
```
When Zotero logs in, it opens defter's login page in the browser; you approve
the login by entering that key, and Zotero receives it. Anyone can start a
login, so the key is what protects the library: generate it, don't choose it
(defter warns about keys shorter than 32 characters), and keep the file
readable by root and the `defter` group only.
To give scripts or agents read-only access as well, provision keys as files
instead, each read/write or read-only:
```sh
openssl rand -hex 32 > app-key # read/write, for Zotero
openssl rand -hex 32 > cli-key # read-only
defter start --keys rw:/path/app-key,ro:/path/cli-key …
```
With no key configured at all, defter generates a read/write key in
`<data-dir>/api-key`; `defter key` prints it.
### Deployment
defter listens on `127.0.0.1:8189`; put it behind a reverse proxy with HTTPS
and set `--public-url` to the proxy's address. The data API is protected by
keys and safe to expose, and so is `/login`, which hands out a key only to
whoever enters one. The dashboard (`/`) shows a sign-in form until an API key
is entered (a read-only key is enough), and `/ui/status` takes the key as a
header. Behind an SSO proxy, add `--login-authorized-user you@example.org` to
also require its identity on login approval.
Clients download from pre-signed URLs on `--s3-endpoint`, so the bucket
endpoint must be reachable from them; otherwise add `--proxy-downloads`.
### Debian with systemd
As root, with Rust installed through [rustup](https://rustup.rs):
```sh
# the binary, straight to /usr/local/bin (the service cannot run one from ~);
# chmod in case a restrictive umask (e.g. 027) made it unreadable to defter
cargo install --locked --root /usr/local defter
chmod 0755 /usr/local/bin/defter
useradd --system --no-create-home --shell /usr/sbin/nologin defter
install -d -m 0750 -o root -g defter /etc/defter
# env.example and defter.service from this repository:
install -m 0640 -o root -g defter env.example /etc/defter/env # then edit it
# the S3 keys, one line each, root:defter 0640:
# /etc/defter/s3-access-key-id, /etc/defter/s3-secret-access-key
install -m 0644 defter.service /etc/systemd/system/defter.service
systemctl daemon-reload && systemctl enable --now defter
journalctl -u defter -f
```
To upgrade: `cargo install --locked --root /usr/local defter` and
`chmod 0755 /usr/local/bin/defter` again, then `systemctl restart defter`.
A binary the `defter` user cannot execute makes systemd fail with status
203/EXEC (behind Cloudflare: error 525). Upgrading to 0.7.0 and 0.6.0, and
from 0.5.0, takes more, below.
### Upgrading to 0.7.0
From 0.7.0, a user's ID is their zotero.org user ID: a Zotero, or the
"defter for Zotero" plugin, then syncs the same library with zotero.org and
with defter. Everyone signing in needs a zotero.org account, which they
connect once (see "Signing in with Keycloak"). Users of a server set up
earlier have other IDs, which are not renumbered: **such a server starts
over.**
1. Take a backup (`defter-admin backup`) and stop defter.
2. Set `DEFTER_USER_ID` to the owner's zotero.org user ID (zotero.org →
Settings → Security: "Your user ID for use in API calls"),
and `DEFTER_DATA_DIR` to a new, empty directory (or move the old one
aside). With S3, use a new bucket or `DEFTER_S3_PREFIX`, so that old and
new attachment files stay apart.
3. Start defter. Admins invite people again; everyone signs in again and
connects their zotero.org account.
4. Each Zotero uploads its library to the new server. A Zotero profile
connected to the old server sees an account with another ID, and Zotero
deletes its local copy when it logs in again: connect a profile that syncs
with zotero.org through the plugin instead (see "Connecting Zotero").
### Upgrading to 0.6.0
0.6.0 adds the server-side commonmeta sync. The upgrade itself needs nothing
more: on its first start defter moves the control database to schema 2, and
without `DEFTER_COMMONMETA_DB` it serves as before. There is no going back past
schema 2, as an older defter refuses to start on it, so take a backup first
(`defter-admin backup`). defter now logs to stderr, which the journal keeps as
before.
To turn the sync on, set `DEFTER_COMMONMETA_DB` and add the `defter` job with
commonmeta 0.9.134's `commonmeta jobs`; on a server shared with dragoman,
dragoman 0.6.2 hands its cache to the same jobs. See
[`docs/commonmeta-sync.md`](docs/commonmeta-sync.md).
### Upgrading from 0.5.0
defter 0.5.0 kept the library in `defter.sqlite3`. Later versions keep each
library in `libraries/`, beside `control.sqlite3`, and refuse to start until
`defter migrate` has moved it. Nothing changes for Zotero clients: the library
keeps its versions and its server ID, and its attachment files stay where they
are. As root, with the `defter-admin` alias from "Backups and restore":
```sh
systemctl stop defter
cargo install --locked --root /usr/local defter && chmod 0755 /usr/local/bin/defter
defter-admin migrate --dry-run # what it will do
defter-admin migrate # backs up first, then moves the library
systemctl start defter
```
The library database moves to `libraries/u<DEFTER_USER_ID>.sqlite3` unchanged.
`control.sqlite3` gets the server ID, the cached Zotero schema and the public
profile's settings. People who were allowed to sign in before need an
invitation on the admin page afterwards (see "People and storage"). An
interrupted migration
finishes when run again. If `DEFTER_DB` was set, leave it while migrating; it
is not needed afterwards.
To go back to 0.5.0: stop the server, move `libraries/u<id>.sqlite3` back to
`defter.sqlite3`, together with its `-wal` and `-shm` files if there are any
(they hold its last writes after an unclean stop), move `control.sqlite3`
aside, and install 0.5.0. Profile settings changed since the migration do not
carry back. Never start 0.5.0 without moving the library back first: it would
start an empty library, and clients would see their library go back to nothing.
[`defter.service`](defter.service) runs defter as the `defter` user with its
data in `/var/lib/defter` (systemd's `StateDirectory`, which defter picks up
without `--data-dir`) and the rest of the file system read-only.
### Sharing port 443 with dragoman
On a machine where [dragoman](https://codeberg.org/front-matter/dragoman)
already serves HTTPS on port 443 behind a CDN, defter needs no port, proxy or
certificate of its own: dragoman's `--sni-route` passes connections for
defter's hostname, still encrypted, to defter, which serves HTTPS on a
loopback port with the same CDN origin certificate (if it covers defter's
hostname, e.g. `*.example.org`).
```sh
# dragoman (/etc/default/commonmeta, which its service reads), then restart it:
DRAGOMAN_SNI_ROUTES=defter.example.org=127.0.0.1:8443
# defter (/etc/defter/env):
DEFTER_PORT=8443
DEFTER_PUBLIC_URL=https://defter.example.org
DEFTER_TLS_CERT=/etc/defter/origin.pem
DEFTER_TLS_KEY=/etc/defter/origin-key.pem
```
Give defter its own copy of the certificate and key, readable by its group:
```sh
install -m 0644 -o root -g defter /etc/dragoman/origin.pem /etc/defter/origin.pem
install -m 0640 -o root -g defter /etc/dragoman/origin-key.pem /etc/defter/origin-key.pem
```
With the hostname proxied by the CDN, its limits apply to uploads through
defter (Cloudflare: 100 MB per request on the Free and Pro plans), so set
`DEFTER_MAX_UPLOAD_MB=100` there; downloads come from the bucket directly.
Without a CDN in front of defter's hostname, let defter get its own certificate
from Let's Encrypt, as dragoman does, instead of giving it `DEFTER_TLS_CERT` and
`DEFTER_TLS_KEY`:
```sh
# defter (/etc/defter/env):
DEFTER_PORT=8443
DEFTER_PUBLIC_URL=https://defter.example.org
DEFTER_DOMAIN=defter.example.org
DEFTER_EMAIL=you@example.org
```
The TLS-ALPN-01 challenge needs no port 80: Let's Encrypt connects to port 443
with defter's hostname, and dragoman passes that through like any other connection.
So port 443 must be open to everyone, not only to a CDN, and the hostname's DNS
must point straight at the server. Certificates renew by themselves and are kept
in `<data-dir>/acme` (`DEFTER_ACME_CACHE`). Set `DEFTER_ACME_STAGING=true` to try
the setup without Let's Encrypt's rate limits; browsers don't trust those
certificates, so delete the cache directory when switching back.
The server must also reach Let's Encrypt. Its ACME client tries addresses in the
order the system returns them and has no timeout of its own, so where IPv6 is
configured but broken, an order stalls without an error while HTTPS fails with
`access denied`. defter checks this at startup and logs a warning, and warns again
if there is no certificate after five minutes. Fix IPv6, or prefer IPv4 by adding
`precedence ::ffff:0:0/96 100` to `/etc/gai.conf`. To see the order's progress, set
`RUST_LOG=defter=info,rustls_acme=info`.
### Hetzner Object Storage
[Hetzner Object Storage](https://www.hetzner.com/storage/object-storage/)
is the intended storage, for a defter running at home as well as for one
later deployed on a Hetzner Cloud server.
1. In Hetzner Console, create a project, a **private** Bucket in one location
(`fsn1` Falkenstein, `nbg1` Nuremberg or `hel1` Helsinki), and S3
credentials for the project (shown once: save both keys). Keys are valid
for every Bucket of the project, so give defter a project of its own.
2. Start defter with the location as both endpoint and region:
```sh
defter start --user-id <your zotero.org user ID> \
--s3-bucket <your-bucket> \
--s3-endpoint https://fsn1.your-objectstorage.com --s3-region fsn1 \
--s3-access-key-id-file ~/.config/defter/s3-access-key-id \
--s3-secret-access-key-file ~/.config/defter/s3-secret-access-key
```
defter addresses Hetzner Buckets by host name
(`https://<bucket>.fsn1.your-objectstorage.com`), which its pre-signed
download URLs need.
3. Check the round trip: `DEFTER_KEY=$(defter key) scripts/smoke-s3.sh`.
Costs (Hetzner's prices, November 2026, excl. VAT): €6.49 a month includes
1 TB of storage and 1 TB of outgoing traffic; uploads and S3 requests are
free. Downloads by pre-signed URL count as outgoing traffic. Objects under
64 KB are billed as 64 KB.
For a later deployment on a Hetzner Cloud server, put the server in an EU
location: it is then in the Bucket's network zone (`eu-central`), and traffic
between them is free. With `--proxy-downloads`, files then reach clients
through the server, whose traffic allowance is separate from the Bucket's.
### Backups and restore
defter backs up its databases (`control.sqlite3` and every library's) to object
storage every 6 hours, under `backup/` in the files bucket, and keeps 30 days of
backups. Each run takes a consistent snapshot while the server keeps running,
checks it, and uploads it gzipped, together with a manifest naming every
database at that point. A database that has not changed since the last backup
is not uploaded again. The dashboard shows the last backup or its error, and so
does `/ui/status`.
```sh
DEFTER_BACKUP_INTERVAL_HOURS=6 # 0 turns scheduled backups off
DEFTER_BACKUP_KEEP_DAYS=30
DEFTER_BACKUP_S3_BUCKET=defter-backups # optional: a second bucket, same endpoint and keys
```
A second bucket is worth it: backups in the files bucket do not survive losing
that bucket, or credentials that can delete from it. Without S3, backups go to
`<files-dir>/backup/` on the same disk, which is for development only.
Attachment files are not copied, as they are in object storage already. While
scheduled backups run, a deleted or replaced attachment file is moved to
`deleted/<date>/` rather than removed, and purged after the same 30 days, so a
restored library gets back its files as they were. With
`DEFTER_BACKUP_INTERVAL_HOURS=0` files are removed for good, so backups taken
another way (`defter backup` from cron) restore the library without them.
Run these commands as root, through an alias that runs defter as the defter user
with the server's environment and data directory, which systemd otherwise sets
(`runuser` comes with Debian, unlike `sudo`):
```sh
alias defter-admin="runuser -u defter -- env DEFTER_DATA_DIR=/var/lib/defter \
bash -c 'set -a; . /etc/defter/env; exec defter \"\$@\"' defter"
```
To back up now, or see what there is (both safe while the server runs):
```sh
defter-admin backup
defter-admin backup --list
```
To restore, stop the server, look at what would be done, then restore:
```sh
systemctl stop defter
defter-admin restore # the newest backup; --manifest <time> for another
defter-admin restore --yes
systemctl start defter
```
This restores the owner's library; `--library u<id>` restores another user's,
`--library g<id>` a group's (deleted ones too), and `--control` the control
database, which also moves every group's version on, so clients pick up its
changes. The database it replaces is kept next to it
as `<name>.sqlite3.before-restore-<time>`. A library's new versions are set
above the replaced database's and above the highest version the server recorded
serving. If neither is known, restore stops and asks for `--min-version`: a
version at least as high as any client has seen, such as the last
`Last-Modified-Version` in the server's log.
After losing the data directory, restore the control database first, then
each library: `defter-admin restore --control --yes`, then
`defter-admin restore --yes` (and `--library u<id> --yes` for other users, and
`--library g<id> --yes` for each group; `defter-admin backup --list` names them). The
control database knows where each library's attachment files are, and the
server refuses to start with libraries but without it. With the library's
database gone, its restore may ask for `--min-version`, as above.
Zotero clients refuse a library whose version goes down, so a restore sets every
version in the library above any it can have served: clients see a newer library
rather than an error. They still hold whatever they synced after the backup.
Afterwards, in Zotero's Settings → Sync → Reset, pick one:
- **Replace Online Library** on the client with the most complete library, when
the restore was to recover lost data: its newer changes go back up.
- **Restore from Online Library** on every client, when the restore was to undo bad
changes: they take the restored library as it is.
Practise it once on a copy, with a throwaway Zotero profile, before you need it.
### Signing in with Keycloak
API keys are always there, and an OpenID Connect provider such as Keycloak can be
added beside them. In a Keycloak realm, create a confidential client with standard
flow on and `<public URL>/ui/oidc/callback` as its valid redirect URI, and map the
account's ORCID iD to a token claim, `orcid` by default. Then:
```sh
DEFTER_OIDC_ISSUER=https://auth.example.org/realms/<realm>
DEFTER_OIDC_CLIENT_ID=defter
DEFTER_OIDC_CLIENT_SECRET_FILE=/etc/defter/oidc-client-secret
DEFTER_ADMIN_ORCIDS=0000-0003-1419-2405
```
**The claim decides who gets in, so accounts must not be able to set it themselves.**
Let it come from ORCID's own login, and in Keycloak's realm settings make the `orcid`
attribute editable by admins only, with unmanaged attributes off or admin-only. Test
it with a throwaway account if the realm allows registration.
With a provider configured, an API key can only be typed into the web pages if
`DEFTER_API_KEY` is set. Leave it out to make Keycloak the only way in. Keys from
`DEFTER_API_KEY_FILE`, `DEFTER_KEYS` or `defter key` keep working for the data API, such
as `commonmeta sync`. Without a provider, typing a key always works.
Who gets in: users, who keep their ORCID iD; people an admin invited; and the
admins in `DEFTER_ADMIN_ORCIDS`, who need no invitation. Everyone else is turned
away. Someone invited, or an admin, who is not a user yet connects their
zotero.org account at the first sign-in: they paste a zotero.org API key
(zotero.org → Settings → Security → Create new private key; read-only is
enough), defter asks zotero.org's `/keys/current` whose it is, and they become
a user with a library of their own, with that zotero.org user ID and user
name. The key is not stored, and can be deleted on zotero.org afterwards. A
zotero.org account that is another user's here is refused. The sign-in waits
15 minutes for the key. `DEFTER_ZOTERO_API_URL` changes where defter asks,
`https://api.zotero.org/` by default.
The owner of the server (`DEFTER_USER_ID`, which must be the owner's
zotero.org user ID; defter does not start without it) is the
user with `DEFTER_ORCID`, or, while the owner has no ORCID iD yet, the admin
whose zotero.org account, connected at their first sign-in, has that user ID.
Another admin signing in first gets a library of their own. Without
`DEFTER_ADMIN_ORCIDS`, the owner is the admin. If the owner has no ORCID iD
yet, set `DEFTER_ORCID` or `DEFTER_ADMIN_ORCIDS` before signing in: otherwise
nobody is let in as the owner.
A zotero.org account counts as whoever gives its key, so anyone holding
someone's key, even a read-only one given to another tool, can take that
zotero.org user ID here first. The admin page lists each user's ID with their
ORCID iD, so such a claim shows, and the person refused is told to ask an
admin, who can disable that user; the ID itself cannot be freed yet.
Signed in, everyone has a dashboard of their own: their library, the storage it
takes, their public profile and their devices. Approving a Zotero login with one
click mints a key for that device, which the dashboard lists and can revoke;
defter keeps only its hash. Unlinking the account in Zotero revokes it too.
Signing out also ends the session at the provider, and so in other applications using
it. Add `<public URL>/` to the client's valid post logout redirect URIs, or Keycloak
refuses. Signing out of Keycloak elsewhere, or disabling an account there, does not reach
defter: its session lasts until it expires after seven days, or defter restarts.
Disabling a user on the admin page ends their sessions, revokes their devices'
keys and hides their profile at once. Their library stays. The owner cannot be
disabled: their keys are configured on the server.
#### Limiting sign-in to a Keycloak group or role
On a realm other applications use too, a group or role can decide who gets in,
instead of invitations:
```sh
DEFTER_OIDC_USER_VALUES=/defter
DEFTER_OIDC_ADMIN_VALUES=/defter/admins
# The claim they are looked for in, a dotted path; groups by default.
# DEFTER_OIDC_ACCESS_CLAIM=groups
```
In Keycloak, create the group `defter` with a subgroup `admins`, and put people
in them. Then add a mapper, either to the client itself (Clients → the client
→ Client scopes → `<client>-dedicated` → Add mapper → By configuration), or to a
client scope of its own (Client scopes → Create, type Default, then add it to
the client's client scopes as Default):
- **Groups:** a *Group Membership* mapper with token claim name `groups`, *Full
group path* on, and *Add to ID token* and *Add to userinfo* on. Values are
group paths, `/defter` and `/defter/admins`. Keycloak lists only the groups
someone is directly in: a member of `/defter/admins` alone carries just that,
which is why admin values grant access too.
- **Or roles:** a *User Realm Role* mapper with token claim name
`realm_access.roles`, or a *User Client Role* mapper for the client with
`resource_access.<client>.roles`, multivalued, with *Add to ID token* on.
Set `DEFTER_OIDC_ACCESS_CLAIM` to that claim name and the values to role
names, such as `defter` and `defter-admin`.
defter reads the claim from the ID token, or, when it is not there, from
userinfo. Its value is a string or an array of strings; a dotted path may also
name a key with dots in it, such as a client ID. On a client other applications
share, the mapper adds the claim to their tokens too.
What changes:
- Only someone carrying one of the values can get an account; invitations no
longer let anyone in, and the admin page offers none. Someone refused is told
to ask for access: by email to `DEFTER_ADMIN_EMAIL` (e.g. the admins' or the
institution's IT address) when it is set, else to ask the admins of this
server. The admins in
`DEFTER_ADMIN_ORCIDS` and the owner are let in without a value.
- The values are checked at every sign-in through Keycloak. A user signing in
without one is refused and disabled, as if an admin had disabled them: their
sessions end and their devices' keys are revoked, their library stays.
Signing in with one again enables them (their devices then log in to Zotero
again); someone an admin disabled stays disabled. The owner is never disabled
this way, nor are admins in `DEFTER_ADMIN_ORCIDS`.
- A claim missing from both the ID token and userinfo, as when a mapper on a
shared client is removed or a client scope detached, refuses everyone but the
owner and the admins in `DEFTER_ADMIN_ORCIDS`, and changes nobody: no account
is disabled, no key revoked, no admin right taken. The log says `the access
claim is in neither the ID token nor userinfo`, at error level. Keycloak may
leave out a claim with nothing in it: someone in no group at all, or with no
role of the client (`resource_access.<client>` is then missing), is refused
but not disabled. Realm roles are always there (every user has the default
roles), so with `realm_access.roles` losing the role disables. Test a mapper
with a throwaway account before relying on it.
- Someone carrying an admin value is an admin, beside `DEFTER_ADMIN_ORCIDS`,
until a sign-in without it. Started without `DEFTER_OIDC_ADMIN_VALUES`,
defter forgets who was: set again, they count from the next sign-in. With either set, the owner is no admin unless
named.
- Taking someone out of the group does not reach defter until they sign in
again: their devices keep syncing with their keys, and a session lasts up to
seven days. defter holds no Keycloak credentials to ask with. To stop them at
once, disable them on the admin page too.
Without `DEFTER_OIDC_USER_VALUES`, invitations decide, and
`DEFTER_OIDC_ADMIN_VALUES` alone is refused at start.
### People and storage
Admins see an Admin link on the dashboard, to `/ui/admin`:
- **Invitations:** invite an ORCID iD, with a note for yourself. The person signs
in through Keycloak, which needs a zotero.org account too (see "Signing in
with Keycloak"), and has an empty library of their own, which their Zotero
then uploads (see "Connecting Zotero"). An unused invitation can be
withdrawn. With `DEFTER_OIDC_USER_VALUES`, the page says which values let
people in instead.
- **Users:** each one's storage use, devices and quota, which can be changed, and
disabling or enabling them; whom a sign-in disabled for lacking the group or
role is marked so.
- **Backups:** how the last one went.
- **commonmeta:** with the server-side sync, how many users opted in, each
one's setting and how their last run went (see
[`docs/commonmeta-sync.md`](docs/commonmeta-sync.md)).
Each user's attachment files count against a quota, 2 GB unless an admin sets
theirs (`DEFTER_DEFAULT_QUOTA_MB`); the owner has none unless set. An upload past
it is refused as zotero.org refuses it, and Zotero says the storage is full. A
new user's files go to `users/<id>/files/` in the bucket; the owner's stay where
they are.
### Groups
Anyone signed in can create a group on their dashboard and add other users of
the server to it by ORCID iD, as members or, by its owner, as admins. Its
members' Zotero clients sync the group's library beside their own, as they
would a zotero.org group. On the group's page:
- **The owner and admins** rename it, describe it, decide who may edit the
library (all members, or only admins) and who may add and change files (all
members, only admins, or nobody), and add or remove members. Only the owner
makes or unmakes admins, and deletes the group.
- **Members** see who is in it, and can leave. The owner cannot leave.
Taken out of a group, or when it is deleted, a member's Zotero is told it is
gone and offers to remove its copy. A deleted group's library is kept, and
backed up, but nobody can reach it. A group's files go to `groups/<id>/files/`
and count against its owner's quota, with their own.
### Connecting Zotero
defter requires **Zotero 10** (desktop). Earlier versions are not supported.
There are two ways to connect it: the plugin, beside zotero.org (the usual
one), or a Zotero profile of its own, for a library synced with defter alone.
#### With the plugin, beside zotero.org
The "defter for Zotero" plugin adds a second sync to Zotero: Zotero keeps
syncing with zotero.org as before, data and Zotero Storage, and the plugin
also syncs the same local library with defter, both ways. Changes made in
Zotero go to both; changes made on defter, such as the nightly commonmeta
sync's, come down into the local library, and from there Zotero's own sync
takes them to zotero.org. Attachment files go to Zotero Storage through
Zotero and to defter through the plugin. It syncs **My Library**.
1. Sign in to defter once (see "Signing in with Keycloak"), so that your user
here has your zotero.org user ID.
2. Download `defter-zotero-<version>.xpi` from defter's
[releases](https://codeberg.org/front-matter/defter/releases/latest) (the
plugin is in [`zotero-plugin/`](zotero-plugin) and shares defter's version)
and install it in Zotero (Tools → Plugins → the gear menu → Install Plugin
From File…). Zotero then updates it from there by itself.
3. Zotero → Settings → **defter**: enter defter's address and log in, which
approves a key for this device on defter's login page. The plugin checks
that the account on defter has the same user ID as the zotero.org account
Zotero syncs with, and stops otherwise.
4. Sync with defter. The first sync merges the two libraries and deletes
nothing: for a new account on defter, it uploads the library.
The plugin's README has the details: when it syncs, how it merges, and what
it leaves out.
#### A profile of its own, for a library synced with defter alone
defter syncs **My Library** and the groups on defter you are a member of.
Groups on zotero.org stay there: a Zotero that syncs with defter no longer
syncs them, and asks whether to keep or remove the local copy of each.
The Zotero app for iPad and iPhone **cannot** sync with defter: it only
syncs its library with zotero.org, whose address is built into the app, and
has no setting, config editor or plugins to change it (it can use WebDAV for
attachment files, but not for the library itself). Someone who uses Zotero on
an iPad keeps that library on zotero.org; a library synced with defter is only
on the desktops that sync with it. With the plugin, the library on zotero.org
is the same one, and gets defter's changes through the desktop's Zotero.
> [!WARNING]
> Connect defter only to a **new, empty data directory**. Zotero treats
> logging in to defter as switching accounts: if the data directory was
> synced with another account (zotero.org) before, Zotero **deletes its local
> libraries** and restarts; it does not combine them with the defter library.
> To bring an existing zotero.org library to defter, use the plugin instead.
Use a separate Zotero profile with its own data directory, so the zotero.org
library stays untouched in the default profile:
1. Close Zotero, back up the existing data directory (Settings → Advanced →
Files and Folders → Show Data Directory) and profile, then start the
profile manager with `zotero -P` (macOS:
`/Applications/Zotero.app/Contents/MacOS/zotero -P`) and create a profile,
e.g. `defter`.
2. In the new profile, Settings → Advanced → Files and Folders → Data
Directory Location → Custom: a new, empty directory (e.g.
`~/Zotero-defter`; check the path, a typo creates another empty one), then
restart Zotero. Do not log in to zotero.org in this profile.
3. Settings → Advanced → **Config Editor** (accept the warning), then set:
```
extensions.zotero.api.url = https://defter.example.org/api/
extensions.zotero.streaming.enabled = false
```
Keep the trailing slash: Zotero appends paths such as `keys/sessions`
directly. defter has no streaming server, hence the second setting.
4. Restart Zotero.
5. Settings → Sync → **Log In**. Zotero opens defter's login page in the
browser; enter an API key that may write and click Approve. Zotero picks
up the key within seconds; the login request expires after 10 minutes.
6. Keep file syncing set to **Zotero** (Settings → Sync → File Syncing):
attachment files then go to defter, and through it to the bucket.
Run one profile at a time; start the one you need with `zotero -P <name>`,
or from the profile manager.
To point a profile back at zotero.org, reset both settings in the Config
Editor (right-click → Reset) and restart Zotero; logging in then switches
accounts again, which deletes the local defter library (it stays on the
server).
### commonmeta sync
`commonmeta sync` enhances the items of a Zotero library with what the
commonmeta store knows about them. On a server that holds the store
(`commonmeta.sqlite3`), defter runs it itself: set `DEFTER_COMMONMETA_DB`, and
users who opt in on their dashboard can preview the changes and sync their
library from there. `defter sync` syncs everyone who opted in and exits;
commonmeta's crontab runs it daily after its imports, with defter's
environment, which cron does not have (`commonmeta jobs add defter --command
"DEFTER_COMMONMETA_DB=/path/to/commonmeta.sqlite3 /usr/local/bin/defter
sync"`, adding `DEFTER_DATA_DIR` and the rest the server is given). Items
are only filled in, with PDFs and their ORCID works in My Publications if
users choose, unless they choose to have commonmeta's values replace theirs.
The store is only read. Setup, permissions, scheduling, monitoring and
rollout are in [`docs/commonmeta-sync.md`](docs/commonmeta-sync.md).
#### Running commonmeta sync yourself
Without the server-side sync, `commonmeta sync` works through Zotero desktop,
which syncs the changes to defter. This also fills a new, empty defter library
for testing, for instance with the works of an ORCID record:
1. Start Zotero in the profile connected to defter. Only that profile may be
running: the local API belongs to whichever Zotero is open, so with the
default profile open, commonmeta writes to the zotero.org library.
2. In that profile, enable Settings → Advanced → **Allow other applications
on this computer to communicate with Zotero**. The setting is per profile,
so a new profile has it off (commonmeta then reports "Zotero's local API is
off"). Check with
`curl -s -o /dev/null -w "%{http_code}\n" http://localhost:23119/api/`,
which prints `200`.
3. Look first, then write a few works, with up to 5 PDFs:
```sh
commonmeta sync --orcid <your-orcid> --dry-run
commonmeta sync --orcid <your-orcid> --limit 25 --pdf --max-pdfs 5
```
The first write asks for permission in a Zotero dialog: choose **Always
Allow**. Each Zotero profile asks once, even after a key was granted in
another.
4. Zotero syncs the new items and PDFs to defter (Settings → Sync → Sync
Now, or the sync button). defter's log then shows `POST
/api/users/<id>/items`, then the file uploads under `/file` and
`/uploads/`; its dashboard counts the items and files.
### Public profile
Every user with an ORCID iD can publish a profile at `/<their-orcid>`, without a
key. The owner's is shown unless hidden; everyone else's starts hidden until they
show it on their dashboard. A profile has their ORCID record, read by commonmeta and
fetched at most hourly, and your works, which can be searched and narrowed by
year, language, type and license, as on dragoman, or exported as BibTeX, RIS,
CSL-JSON and more. Languages and licenses are read from Zotero's Language and
Rights fields as ISO 639-1 codes and SPDX IDs: `en-US`, `eng`, `English` and
`Englisch` are all English, and `CC BY 4.0` and its creativecommons.org URL are
both `CC-BY-4.0`. A work whose field names neither is not counted there. It is available in
English and German.
**Every work in My Publications is shown**, as long as it is at the top level
and not in the trash, with its title, creators, date, abstract, publication,
language, rights and DOI. Its tags, notes and files are not, nor are they exported. The rest of the
library stays private. The profile opens on up to five works with the manual tag
`featured`, newest first, if there are any; unticking "Featured only" shows them all.
A work with a PDF attached links to where that PDF was
published, never to the file stored in defter or Zotero; a PDF uploaded from disk
has no such place and is not linked.
The dashboard links to the profile and, signed in with a read/write key, can
hide or show it; while hidden, or for a disabled user, the address answers `404`. Unlike defter's other pages, the
profile may be indexed by search engines.
## Development
```sh
cargo test # unit and API tests
cargo run -- start --user-id 1 # http://127.0.0.1:8189, files on local disk
```
Building needs only Rust. Pages are askama templates in `templates/`,
compiled into the binary, so a template change needs a rebuild (`cargo run`
does it). Their stylesheet, `ui/dist/app.css`, is built from `ui/app.css` by
the standalone [Tailwind CLI](https://tailwindcss.com/docs/installation/tailwind-cli)
and committed. To change styling, install it (`brew install tailwindcss`, or
a release binary with `TAILWINDCSS=/path/to/tailwindcss`); the build then
regenerates the stylesheet, and warns when it can't. The script is the
vendored htmx (`ui/vendor/`, see its README for updating) plus `ui/main.js`.
To run against real S3 storage locally, use [RustFS](https://rustfs.com):
```sh
docker run -d --rm --name rustfs -p 9000:9000 \
-e RUSTFS_ACCESS_KEY=defter -e RUSTFS_SECRET_KEY=defter-secret rustfs/rustfs
curl --aws-sigv4 "aws:amz:us-east-1:s3" --user defter:defter-secret \
-X PUT http://127.0.0.1:9000/zotero # create the bucket
DEFTER_S3_BUCKET=zotero DEFTER_S3_ENDPOINT=http://127.0.0.1:9000 \
DEFTER_S3_REGION=us-east-1 DEFTER_S3_ACCESS_KEY_ID=defter \
DEFTER_S3_SECRET_ACCESS_KEY=defter-secret cargo run -- start --user-id 1
DEFTER_KEY=$(cargo run -q -- key) scripts/smoke-s3.sh # file round trip
```
### Layout
| `src/main.rs` | CLI and startup |
| `src/app.rs` | state, router, gzip and logging middleware |
| `src/zotero.rs` | Zotero API handlers |
| `src/auth.rs` | keys, the access decision, login sessions, expiring tokens |
| `src/control.rs`, `migrations/control/` | the control database: users, library state |
| `src/registry.rs` | opening each library's database on demand |
| `src/migrate.rs` | `defter migrate`, from 0.5.0's single database |
| `src/store.rs`, `migrations/` | library storage in SQLite |
| `src/query.rs` | item listing parameters |
| `src/db.rs` | connections and migrations |
| `src/storage.rs` | attachment files via `object_store` |
| `src/backup.rs` | backups of the databases, and restoring them |
| `src/sync.rs` | server-side commonmeta sync: `defter sync`'s run, the dashboard's, a library as commonmeta sees it |
| `src/ui.rs`, `templates/` | web UI pages (askama) |
| `src/profile.rs` | public profile: ORCID record and what the page shows |
| `src/vocab.rs` | a work's language and license as ISO 639-1 and SPDX |
| `ui/` | the pages' stylesheet (Tailwind) and script (vendored htmx) |
## License
MIT. Includes code adapted from zhost (MIT) and the PDF icon of
[Catppuccin Icons](https://github.com/catppuccin/vscode-icons) (MIT), see
`NOTICE`.