# defter API
defter serves a subset of the [Zotero Web API v3](https://www.zotero.org/support/dev/web_api/v3/start)
— what a stock Zotero client needs to sync, plus the read/query API a CLI
uses — and the parts of Zotero's local API that `commonmeta sync` relies on.
Every Zotero endpoint is served twice: at the root (the Zotero client's
`extensions.zotero.api.url`) and under `/api` (where commonmeta expects
Zotero's local API, `http://localhost:23119/api`). `<lib>` is `users/<id>`, the
key's own user's library: any other answers `403`. `users/0` is accepted as an
alias for it, as in the local API. `<lib>` is also `groups/<id>`, the library of
a group the key's user is a member of: anyone else gets `403`. In a group, a
member may write when the group lets members edit (`libraryEditing`), admins and
the owner always, and may upload files as `fileEditing` says; otherwise `403`.
## Supported client
defter requires the **Zotero 10** desktop client; earlier versions are not
supported. Zotero 10 has no setting for a sync server in its interface: it
reads `extensions.zotero.api.url` (Config Editor) and falls back to
`https://api.zotero.org/`. It appends paths such as `keys/sessions` to that
URL directly, so the URL must end in `/`. With
`extensions.zotero.streaming.enabled` off, Zotero polls instead of using a
streaming server, which defter does not have.
Zotero 10's login (checked against the source of 10.0.7:
`syncAPIClient.js`, `syncRunner.js`, `preferences_account.js`):
- `POST keys/sessions`, with a JSON body `{userID}` when the profile was
linked to an account before (defter ignores it); Zotero opens `loginURL`
in the browser.
- It polls `GET keys/sessions/<token>` for up to 10 minutes, treating `404`
as "session not found" and `410` as "expired".
- A completed session must carry `apiKey`, `userID` and `username`; if
storing the key fails, Zotero deletes it with `DELETE keys/current`.
The sync contract follows zhost's `server/SPEC.md`, which records what live
testing against Zotero 9 showed; the differences are noted below.
## Authentication
Data endpoints need a key in `Zotero-API-Key` (or `Authorization: Bearer`);
an unknown key is `403`. Read-only keys get `403` on `POST`/`PUT`/`PATCH`/`DELETE`.
Keys configured on the server are the owner's. Every other key is minted when a
user signed in through the OpenID Connect provider approves a Zotero login, and
only its sha256 is stored (see the README).
| Endpoint | Auth | Notes |
|---|---|---|
| `GET /api/` | — | `Zotero-Server-ID` (stable per installation) and `X-Defter-Version` headers; `/` carries them too |
| `GET /keys/current` | key | `{key, userID, username, displayName, access}` of the key's user; `access.user.write` reflects the key, and `access.groups.all` covers every group they are in, as their role there allows |
| `DELETE /keys/current` | key | `204`; Zotero sends it when the account is unlinked. A minted key is revoked; one configured on the server stays |
| `POST /keys/sessions` | — | `201 {sessionToken, loginURL}` |
| `GET /keys/sessions/<token>` | — | `{status: "pending"}` until approved, then once `{status: "completed", apiKey, userID, username}`: the configured key the session was approved with, or the key minted for the user who approved it |
| `DELETE /keys/sessions/<token>` | — | cancel |
| `GET /login?session=<token>` | — | server-rendered consent page with a form for an API key, no JavaScript needed; never authorizes by itself. `400` without a session, `404` when unknown or expired. Not cacheable or frameable, strict CSP; its URL (with the token) goes as a referrer to defter only (`Referrer-Policy: same-origin`) |
| `POST /login` (form `session`, `key`) | — | approve with an API key that may write; a wrong or read-only key gets the form again with `403`, after a second's delay. Also rejects a foreign `Origin`, and checks the SSO identity with `--login-authorized-user` (`403` page). `303` to `/login?approved=1` |
Anyone can create a session; only a key that may write approves one, so
`/login` is safe to expose. The web UI needs a key too (see below).
## Versions and preconditions
A single library version is bumped by every write and returned in
`Last-Modified-Version`; each object carries the version it last changed at.
Every response about a library also carries `X-Defter-Library-Epoch`, an ID
that stays the same while the library goes on and changes when `defter
restore` brings it back from a backup. A restore stamps versions above every
one served, so versions alone cannot tell a restored library from one that
moved on; defter for Zotero stops when the epoch changes.
- `?since=<v>` reads return objects with a greater version, always `200`
(never `304`: the client's version reads loop on it). Only `GET /settings`
answers `304` to `If-Modified-Since-Version`.
- Multi-object writes: `If-Unmodified-Since-Version` is the **library**
version → `412` when stale. It is optional on `POST`/`PATCH` (as in the Web
API; commonmeta sends none when creating); without it, an object whose
`version` property differs from the stored one fails on its own with `412`
in `failed`. Deletes require it (`428` without).
- Single-object writes (`PUT`/`PATCH`/`DELETE …/<key>`): the precondition is the
**object's** version, from the header or the body's `version` (`428` when
neither is given). `PUT` with version `0` creates.
Write bodies may be gzip-compressed (`Content-Encoding: gzip`).
An item field written as `""` is cleared, not stored, as on zotero.org: a
Zotero client clears the fields an item's new type lacks that way, and a client
that later downloads the item refuses a field its type does not have, even an
empty one.
Item writes stamp the dates as zotero.org does: `dateAdded` is the one sent,
else the stored one, else now; `dateModified` is now when the write sends
none, or when a `PUT` sends the stored value back unchanged, and otherwise the
one sent. Zotero sends its own, so its writes keep them; a change made on the
server, by the commonmeta sync, say, gets a new `dateModified`.
## Library data
| Endpoint | Notes |
|---|---|
| `GET <lib>/{items,collections,searches}?format=versions&since=<v>` | `{key: version}` |
| `GET <lib>/{items,collections,searches}?{item,collection,search}Key=a,b` | `[{key, version, meta?, data}]` |
| `POST`/`PATCH <lib>/{items,collections,searches}` | ≤ 50 objects; merge semantics; `{successful, success, unchanged, failed}` keyed by batch index; `successful` holds full objects |
| `DELETE <lib>/{items,collections,searches}?<kind>Key=a,b` | `204` |
| `GET`/`PUT`/`PATCH`/`DELETE <lib>/{items,collections,searches}/<key>` | single object; `GET` carries the object's version in `Last-Modified-Version`; writes `204` |
| `GET <lib>/items/<key>/children` | attachments and notes (a listing, see below) |
| `GET <lib>/items/top`, `/items/trash` | listings; `items/top` also answers the `format=versions` and `itemKey` reads |
| `GET <lib>/collections/<key>/items[/top]` | listings |
| `GET <lib>/tags` | `[{tag, meta: {type, numItems}}]`, trash excluded |
| `DELETE <lib>/tags?tag=a \|\| b` | strips the tags from every item and logs them |
| `GET`/`POST`/`DELETE <lib>/settings` | `{key: {value, version}}`; delete by `settingKey=a,b`. A `POST` of an unchanged value makes no new version, as on zotero.org |
| `GET <lib>/deleted?since=<v>` | `{collections, searches, items, settings, tags}` |
| `GET <lib>/fulltext?since=<v>` | `{itemKey: version}` |
| `GET <lib>/items/<key>/fulltext` | `{content, indexedChars, …}` with the row's own version |
| `POST <lib>/fulltext` | batch of `{key, content, …}` |
| `GET users/<id>/groups` | the groups the user is a member of: with `format=versions` `{groupID: version}`, which a client compares with its own (a group missing from it is one the user was taken out of, or deleted), otherwise their JSON as below. The `Last-Modified-Version` is the user's library's |
| `GET groups/<id>` | the group's JSON, as zotero.org's: `{id, version, links, meta: {created, lastModified, numItems}, data: {id, version, name, owner, type: "Private", description, url, libraryEditing, libraryReading: "members", fileEditing, admins, members}}`, with the owner, admins and members as user IDs (the owner and admins are not repeated among the members), and the group's version in `Last-Modified-Version`: every change to the group or its members makes a new one |
| `GET /schema` | Zotero's item schema, fetched from `api.zotero.org/schema`, stored, revalidated daily |
Object keys are 8 characters from `23456789ABCDEFGHIJKLMNPQRSTUVWXYZ`;
others fail with `400`. A keyless object gets a fresh key. Item data is
stored verbatim; attachments and annotations come back with `linkMode` /
`annotationType` first. Top-level items carry `meta.numChildren`. In a group
library, objects also carry `meta.createdByUser` and `meta.lastModifiedByUser`,
each `{id, username, name}`: the member who added the object and the one who
last changed it, as zotero.org's do.
### Listings
Item listings accept:
| Parameter | Meaning |
|---|---|
| `q`, `qmode` | case-insensitive substring; `titleCreatorYear` (default) searches title, date and creators; `everything` searches every field and tag (DOI, `extra`, URL, …) and stored full text |
| `itemType` | `book \|\| journalArticle`, `-attachment` |
| `tag` | repeat for AND, `\|\|` for OR |
| `sort`, `direction` | `dateModified` (default), `dateAdded`, `title`, `creator`, `date`, `itemType`; `asc`/`desc` |
| `limit`, `start` | page of ≤ 100 (default 25); `Total-Results` and `Link: <…>; rel="next"` on the public URL |
| `includeTrashed` | include items in the trash |
| `format=keys` | every matching key, unpaged, as plain text |
## Files
Attachment bytes live in object storage, under `users/<id>/files/<item key>`,
`groups/<id>/files/<item key>` for a group's, which count against its owner's
quota, or
`files/<item key>` for the library of a server from before there were several;
the database keeps their md5, size and mtime.
1. **Authorize** — `POST <lib>/items/<key>/file`, form `md5, filename,
filesize, mtime` (ms), with `If-None-Match: *` (new) or `If-Match: <md5>`
(replace); `428` without either, `404` for an unknown item. Answers
`{exists: 1}` when the same file is stored, `412` when another is (or when
`If-Match` names a different md5), `413` past `--max-upload-mb`, `413` with
`Zotero-Storage-Usage` and `Zotero-Storage-Quota` (MB) and
`Zotero-Storage-UserID` when the file would take the user past their quota (a
file it replaces does not count), otherwise
`{url, uploadKey, contentType, prefix: "", suffix: ""}` with an absolute
`url`. As on zotero.org, `exists` and `412` carry the item's version in
`Last-Modified-Version`, except a `412` for `If-Match` on an item with no
stored file, which carries none: Zotero then retries with
`If-None-Match: *`. Files added through Zotero's local API (as
`commonmeta sync --pdf` does) arrive that way.
Snapshots (`text/*` attachments) and attachments with several files come
as a ZIP of the attachment directory, with `zipMD5` and `zipFilename` added:
`md5` is then the main file's, `filesize` the ZIP's, and the upload is
checked against `zipMD5`. defter stores the ZIP as it is.
2. **Upload** — `POST <url>` with the raw bytes (no key: the token in the URL
is the capability). Streamed to `pending/<token>` while hashing; a md5 or
size mismatch is `400` and the bytes are dropped. `201`.
3. **Register** — `POST <lib>/items/<key>/file`, form `upload=<uploadKey>`:
moves the file to `files/<key>`, records it and stamps `md5`/`mtime` into the
item. `204`.
**Download** — `GET <lib>/items/<key>/file` answers `302` with
`Zotero-File-MD5`, `Zotero-File-Modification-Time`, `Zotero-File-Compressed`
(`Yes` for a stored ZIP, which Zotero unpacks) and `Location`: a pre-signed
bucket URL valid for `--presign-ttl` seconds, or, without signing (local
storage, `--proxy-downloads`), a `/downloads/<token>` link valid for five
minutes that streams the file through defter.
Deleting an attachment item deletes its file.
## Web UI
| Endpoint | Notes |
|---|---|
| `GET /?q=&page=` | the homepage, for everyone: the first two paragraphs of the README about defter, and the public profiles (users with an ORCID iD whose profile is shown, not disabled), by name, searchable by name or ORCID iD (`q`), ten a page (`page`), each as dragoman lists people: the names, biography and country of their ORCID record (as cached; one not cached yet is fetched in the background and the person shows by name meanwhile), how many public works they have, and their iD. Not cacheable |
| `GET /ui/settings` | signed in: the user's dashboard (their library, storage, profile and devices; with `--commonmeta-db` their commonmeta sync's settings and last run, and with `?sync=busy` a note that another run was going; for admins also where files are stored, backups and a link to `/ui/admin`), and with `HX-Request: true` (htmx's 30-second poll) only its status block (`Vary: HX-Request`). Otherwise `303` to `/ui/login` (with `Zotero-Server-ID`), or for htmx's poll an empty `401` with `HX-Refresh: true`, which reloads the page. Not cacheable. `/ui/groups/{id}` and `/ui/admin` send whoever is not signed in to `/ui/login` too |
| `GET /ui/login` | the sign-in page: a form for an API key (with key login) and the OpenID Connect provider's button, when configured; `303` to `/` for whoever is signed in already. Not cacheable |
| `POST /ui/session` (form `key`) | sign in as the owner with an API key configured on the server, read-only included: sets the `defter_session` cookie (HttpOnly, SameSite=Lax, Secure with an HTTPS public URL, 24 hours) and `303` to `/`. A wrong key gets the form again with `403` after a second's delay; a foreign `Origin` is rejected |
| `GET /ui/oidc/login?session=` | with `--oidc-issuer`: sends the browser to the provider, with a state tied to it by the `defter_oidc` cookie. `session` is a Zotero login to return to. `404` without `--oidc-issuer`, `502` if the provider cannot be read |
| `GET /ui/oidc/callback?state=&code=` | the provider's redirect back: for a user's ORCID iD sets the `defter_session` cookie and `303` to `/` or the Zotero login. An invited iD, or an admin's (`--admin-orcids`), that is no user's yet gets the `defter_account` cookie instead (with `--oidc-user-values`, an iD carrying one of its or `--oidc-admin-values`' values in the `--oidc-access-claim` claim of the ID token, else of userinfo, or an admin's; invitations then do not count) (HttpOnly, SameSite=Lax, `Path=/ui/account`, 15 minutes), naming a pending sign-in held in memory, and `303` to `/ui/account`. `403` with the sign-in form for anyone else or a disabled user, telling them to ask for access: by email to `--admin-email` when it is set, else the admins of this server. With `--oidc-user-values`, every sign-in also checks the values: a user without one gets `403` and is disabled, with their keys revoked and sessions ended (not the owner, nor an admin on `--admin-orcids`); when the claim is in neither the ID token nor userinfo, anyone but those gets `403` and nobody is changed; one a sign-in disabled so is enabled again by a sign-in with one, one an admin disabled is not; and whether they carry an admin value is kept as their admin right until their next sign-in |
| `GET /ui/account` | with a pending sign-in: the form to connect a zotero.org account (`Cache-Control: no-store`). `403` with the sign-in form, saying to sign in again, without one or when it has expired |
| `POST /ui/account` (form `key`, a zotero.org API key) | defter asks `<--zotero-api-url>/keys/current` with the key whose it is, then forgets it (never stored or logged), and makes the pending sign-in a user with that `userID` and `username` (their iD when another user has that name), or the owner when the `userID` is `--user-id`, the owner has no ORCID iD yet and the pending sign-in is an admin's (`--admin-orcids`; anyone else gets `409`), using up the invitation: the `defter_session` cookie, the `defter_account` cookie cleared, and `303` to `/` or the Zotero login. A key zotero.org rejects gets the form again with `403` after a second's delay, a `userID` another user has `409`, zotero.org unreachable `502`; no or an expired pending sign-in, or one no longer invited, `403` with the sign-in form; a foreign `Origin` `403` |
| `POST /ui/logout` | ends the session, clears the cookie, `303` to `/`. A session from an OpenID Connect provider gets a page that goes on to the provider's logout instead, which returns to `/` |
| `GET /<orcid>?q=&year=&type=&featured=&content=&lang=&page=` | the public profile of the user with that ORCID iD, without a key. `q` searches title, creators and date; `year`, `language`, `type` and `license` can repeat: values of one are ORed, different ones ANDed. `language` is an ISO 639-1 code and `license` an SPDX ID, read from Zotero's Language and Rights fields (`en-US`, `English`, `Deutsch` and `ger` are all `en` or `de`; `CC BY 4.0` and its URL are `CC-BY-4.0`); a work whose field names neither is left out of those facets. With none of these, or with `featured=1`, it shows up to ten works with the manual tag `featured` (`featured=0` for all works; a bare address shows all of them if none is featured). `content=pdf` to works with a PDF link. `lang` is `en` or `de`, otherwise `Accept-Language` decides. With `HX-Request: true` only the list of works (`Vary: HX-Request, Accept-Language`). `404` for any other path, while hidden, and for a disabled user |
| `GET /<orcid>/export?format=&style=&…` | the works the list shows, under the same filters and without paging, as a file, or with `key` the one public work with that key (`404` for any other): `bibtex`, `commonmeta` (default), `crossref`, `crossref_xml`, `csl`, `datacite`, `datacite_xml`, `citation`, `inveniordm`, `ris` or `schemaorg`. `style` is for `citation`: `apa` (default), `chicago-author-date`, `harvard-cite-them-right`, `ieee`, `modern-language-association` or `vancouver`. `400` for anything else |
| `POST /ui/profile` (form `hidden`, `true` or `false`) | hide or show the signed-in user's public profile: `303` to `/`. Needs a session signed in with a read/write key (`401`, `403`); a foreign `Origin` is rejected, as on every `POST` below |
| `POST /ui/sync` (form `metadata`, `publications`, each on when present and not `false`) | the signed-in user's choices for the server-side commonmeta sync of their own library, each on its own (see [`commonmeta-sync.md`](commonmeta-sync.md)): `303` to `/`. `metadata` updates the works' metadata, PDFs included; `publications` adds the works of their ORCID record to My Publications, and is saved only for a user with an ORCID iD. commonmeta only fills what an item lacks; nothing is replaced, and where an item says otherwise than commonmeta, a duplicate with commonmeta's metadata is created for Zotero to merge. `404` without `--commonmeta-db`; `401`, `403` as above |
| `POST /ui/sync/run` (form `dry_run`, `true` or `false`) | start a run of the signed-in user's library now, in the background, with their saved choices: `303` to `/`, or to `/?sync=busy` while another run is going, in the server or in `defter sync` (one at a time, held by `<data-dir>/sync.lock`). `dry_run=true` previews, writing nothing, and needs no choice made (with none, it previews both); a real run needs one (`400` otherwise); a `dry_run` that is neither is a `400`. `404` without `--commonmeta-db`; `401`, `403` as above |
| `GET /ui/sync/log.json` | the signed-in user's last run or preview as a JSON download (`Content-Disposition: attachment`, `Cache-Control: no-store`): `{startedAt, finishedAt, status, dryRun, commonmetaVersion, summary, error}` as in `/ui/status`, with `results`, every per-item result kept (up to 1,000, those with changed fields first when there were more: `{key, doi, status, fields, pdf, message, changes}`, each change as `{"from", "to"}` in a preview), and `resultsTotal`, how many there were, more than kept when the log was cut short. `401` without a session, `404` without `--commonmeta-db` or before the first run |
| `POST /ui/groups` (form `name`) | create a group the signed-in user owns: `303` to its page |
| `GET /ui/groups/<id>` | a group's settings and members, for its members: `404` for others |
| `POST /ui/groups/<id>/settings` (form `name`, `description`, `library_editing`, `file_editing`) | for the owner and admins |
| `POST /ui/groups/<id>/members` (form `orcid`, or `user` for a member already, and `role`, `member` or `admin`) | add a user of the server by their ORCID iD, or change a member's role: the owner and admins add members, only the owner makes or unmakes admins. `?error=orcid` or `?error=user` for no iD or no user |
| `POST /ui/groups/<id>/remove` (form `user`, the signed-in user's own without) | take a member out, or leave: admins take out members, the owner anyone; the owner cannot leave |
| `POST /ui/groups/<id>/delete` | for the owner: members' clients are told the group is gone; its library is kept |
| `POST /ui/keys/revoke` (form `id`) | revoke one of the signed-in user's devices' keys: `303` to `/`, `404` for a key that is not theirs |
| `GET /ui/admin` | for admins: invitations (with `--oidc-user-values` instead the claim and values that let people in), users with their storage, devices and quota, and backups; with `--commonmeta-db` also how many users opted in to the commonmeta sync and when the newest sync ended, each user's setting and last run, and how many last runs failed. `404` for others, `303` to `/` when not signed in |
| `POST /ui/admin/invite` (form `orcid`, `note`) | invite an ORCID iD (or ORCID URL): `303` to `/ui/admin`, with `?error=orcid` for no iD and `?error=user` for a user's; `400` with `--oidc-user-values`. Admins with a read/write session only |
| `POST /ui/admin/uninvite` (form `orcid`) | withdraw an invitation not used yet |
| `POST /ui/admin/user` (form `id`, and `quota_mb`, empty for the default, or `disabled`, `true` or `false`) | set a user's quota (`?error=quota` for no number), or disable or enable them. Disabling revokes their keys and ends their sessions, and signing in does not enable them again; enabling hands someone a sign-in disabled back to `--oidc-user-values`; admins cannot disable themselves, nor the owner, whose keys are configured on the server (`400`) |
| `GET /assets/<name>` | the stylesheet and script (htmx), named by content hash and cached for a year; public |
| `GET /ui/status` | the dashboard's data as JSON, for monitoring, about the user of the API key (`Zotero-API-Key` or `Authorization: Bearer`) or the session cookie, `401` otherwise: their library, `quota` (bytes, `null` for none) and `storage.maxUpload`. For admins also `storage.location`, `storage.signedDownloads` and `backup`, which holds `enabled`, and the last run's `lastAt`, `manifest`, `error`, and `warning` when it succeeded with a problem, such as old backups it could not remove. For everyone, `sync` is `{enabled: false}` without `--commonmeta-db`, otherwise `{enabled: true, setting, running, lastRun}`: `setting` is `{metadata, publications}`, `running` whether the server is running a sync of their library (not `defter sync`), and `lastRun` their newest run or preview (`null` for none) as `{startedAt, finishedAt, status, dryRun, commonmetaVersion, summary, error}`, with `status` `ok`, `partial`, `failed` or `aborted` and `summary` commonmeta's counts (camelCase: `items`, `withDoi`, `updated`, `pdfs`, `conflicts`, `failed`, …). For admins `sync` also holds `users`, `[{userId, setting, lastRun}]` for every user with a setting or a run |
Sessions are kept in memory, so a restart signs everyone out.
Pages are in English or German: `?lang=en` or `?lang=de`, which the `defter_lang` cookie
then remembers for a year, otherwise the browser's `Accept-Language`, otherwise English.
With `--oidc-issuer` and without `--api-key` (`DEFTER_API_KEY`), a key can't be typed in:
the forms are not offered, and `POST /ui/session` and a `POST /login` from a session not
signed in through the provider answer `403`. Keys keep working for the data API and
`GET /ui/status`.
Pages are rendered on the server from the askama templates in `templates/`
and work without JavaScript. They are sent with a Content-Security-Policy
that allows scripts and styles only from defter (plus a per-response nonce
for the inline theme script) and forbids framing; htmx is configured not to
use `eval` or inline styles, so it runs under that policy.
## Changes in 0.7.0
A user's ID is their zotero.org user ID, read from zotero.org with a key they
give at their first sign-in (`/ui/account`); new users no longer get a random
ID. `DEFTER_USER_ID` must be the owner's zotero.org user ID, and is required: it no
longer defaults to 1. Group IDs are
still defter's own. Users of a server set up earlier keep their old IDs, which
are not renumbered: such a server starts over with a new data directory (see
"Upgrading to 0.7.0" in the README).
## Out of scope
Group libraries, the streaming (websocket) server, binary-diff uploads
(`PATCH …/file`), and styled bibliographies.