# defter API
defter serves a subset of the [Zotero Web API v3](https://www.zotero.org/support/dev/web_api/v3/start)
— what a stock Zotero client needs to sync, plus the read/query API a CLI
uses — and the parts of Zotero's local API that `commonmeta sync` relies on.
Every Zotero endpoint is served twice: at the root (the Zotero client's
`extensions.zotero.api.url`) and under `/api` (where commonmeta expects
Zotero's local API, `http://localhost:23119/api`). `<lib>` is `users/<id>`, the
key's own user's library: any other answers `403`. `users/0` is accepted as an
alias for it, as in the local API. `<lib>` is also `groups/<id>`, the library of
a group the key's user is a member of: anyone else gets `403`. In a group, a
member may write when the group lets members edit (`libraryEditing`), admins and
the owner always, and may upload files as `fileEditing` says; otherwise `403`.
## Supported client
defter requires the **Zotero 10** desktop client; earlier versions are not
supported. Zotero 10 has no setting for a sync server in its interface: it
reads `extensions.zotero.api.url` (Config Editor) and falls back to
`https://api.zotero.org/`. It appends paths such as `keys/sessions` to that
URL directly, so the URL must end in `/`. With
`extensions.zotero.streaming.enabled` off, Zotero polls instead of using a
streaming server, which defter does not have.
Zotero 10's login (checked against the source of 10.0.7:
`syncAPIClient.js`, `syncRunner.js`, `preferences_account.js`):
- `POST keys/sessions`, with a JSON body `{userID}` when the profile was
linked to an account before (defter ignores it); Zotero opens `loginURL`
in the browser.
- It polls `GET keys/sessions/<token>` for up to 10 minutes, treating `404`
as "session not found" and `410` as "expired".
- A completed session must carry `apiKey`, `userID` and `username`; if
storing the key fails, Zotero deletes it with `DELETE keys/current`.
The sync contract follows zhost's `server/SPEC.md`, which records what live
testing against Zotero 9 showed; the differences are noted below.
## Authentication
Data endpoints need a key in `Zotero-API-Key` (or `Authorization: Bearer`);
an unknown key is `403`. Read-only keys get `403` on `POST`/`PUT`/`PATCH`/`DELETE`.
Keys configured on the server are the owner's. Every other key is minted when a
user signed in through the OpenID Connect provider approves a Zotero login, and
only its sha256 is stored (see the README).
| Endpoint | Auth | Notes |
|---|---|---|
| `GET /api/` | — | `Zotero-Server-ID` (stable per installation) and `X-Defter-Version` headers; `/` carries them too |
| `GET /keys/current` | key | `{key, userID, username, displayName, access}` of the key's user; `access.user.write` reflects the key, and `access.groups.all` covers every group they are in, as their role there allows |
| `DELETE /keys/current` | key | `204`; Zotero sends it when the account is unlinked. A minted key is revoked; one configured on the server stays |
| `POST /keys/sessions` | — | `201 {sessionToken, loginURL}` |
| `GET /keys/sessions/<token>` | — | `{status: "pending"}` until approved, then once `{status: "completed", apiKey, userID, username}`: the configured key the session was approved with, or the key minted for the user who approved it |
| `DELETE /keys/sessions/<token>` | — | cancel |
| `GET /login?session=<token>` | — | server-rendered consent page with a form for an API key, no JavaScript needed; never authorizes by itself. `400` without a session, `404` when unknown or expired. Not cacheable or frameable, strict CSP; its URL (with the token) goes as a referrer to defter only (`Referrer-Policy: same-origin`) |
| `POST /login` (form `session`, `key`) | — | approve with an API key that may write; a wrong or read-only key gets the form again with `403`, after a second's delay. Also rejects a foreign `Origin`, and checks the SSO identity with `--login-authorized-user` (`403` page). `303` to `/login?approved=1` |
Anyone can create a session; only a key that may write approves one, so
`/login` is safe to expose. The web UI needs a key too (see below).
## Versions and preconditions
A single library version is bumped by every write and returned in
`Last-Modified-Version`; each object carries the version it last changed at.
Every response about a library also carries `X-Defter-Library-Epoch`, an ID
that stays the same while the library goes on and changes when `defter
restore` brings it back from a backup. A restore stamps versions above every
one served, so versions alone cannot tell a restored library from one that
moved on; defter for Zotero stops when the epoch changes.
- `?since=<v>` reads return objects with a greater version, always `200`
(never `304`: the client's version reads loop on it). Only `GET /settings`
answers `304` to `If-Modified-Since-Version`.
- Multi-object writes: `If-Unmodified-Since-Version` is the **library**
version → `412` when stale. It is optional on `POST`/`PATCH` (as in the Web
API; commonmeta sends none when creating); without it, an object whose
`version` property differs from the stored one fails on its own with `412`
in `failed`. Deletes require it (`428` without).
- Single-object writes (`PUT`/`PATCH`/`DELETE …/<key>`): the precondition is the
**object's** version, from the header or the body's `version` (`428` when
neither is given). `PUT` with version `0` creates.
Write bodies may be gzip-compressed (`Content-Encoding: gzip`).
An item field written as `""` is cleared, not stored, as on zotero.org: a
Zotero client clears the fields an item's new type lacks that way, and a client
that later downloads the item refuses a field its type does not have, even an
empty one.
Item writes stamp the dates as zotero.org does: `dateAdded` is the one sent,
else the stored one, else now; `dateModified` is now when the write sends
none, or when a `PUT` sends the stored value back unchanged, and otherwise the
one sent. Zotero sends its own, so its writes keep them; a change made on the
server, by the commonmeta sync, say, gets a new `dateModified`.
## Library data
| Endpoint | Notes |
|---|---|
| `GET <lib>/{items,collections,searches}?format=versions&since=<v>` | `{key: version}` |
| `GET <lib>/{items,collections,searches}?{item,collection,search}Key=a,b` | `[{key, version, meta?, data}]` |
| `POST`/`PATCH <lib>/{items,collections,searches}` | ≤ 50 objects; merge semantics; `{successful, success, unchanged, failed}` keyed by batch index; `successful` holds full objects |
| `DELETE <lib>/{items,collections,searches}?<kind>Key=a,b` | `204` |
| `GET`/`PUT`/`PATCH`/`DELETE <lib>/{items,collections,searches}/<key>` | single object; `GET` carries the object's version in `Last-Modified-Version`; writes `204` |
| `GET <lib>/items/<key>/children` | attachments and notes (a listing, see below) |
| `GET <lib>/items/top`, `/items/trash` | listings; `items/top` also answers the `format=versions` and `itemKey` reads |
| `GET <lib>/collections/<key>/items[/top]` | listings |
| `GET <lib>/tags` | `[{tag, meta: {type, numItems}}]`, trash excluded |
| `DELETE <lib>/tags?tag=a \|\| b` | strips the tags from every item and logs them |
| `GET`/`POST`/`DELETE <lib>/settings` | `{key: {value, version}}`; delete by `settingKey=a,b`. A `POST` of an unchanged value makes no new version, as on zotero.org |
| `GET <lib>/deleted?since=<v>` | `{collections, searches, items, settings, tags}` |
| `GET <lib>/fulltext?since=<v>` | `{itemKey: version}` |
| `GET <lib>/items/<key>/fulltext` | `{content, indexedChars, …}` with the row's own version |
| `POST <lib>/fulltext` | batch of `{key, content, …}` |
| `GET users/<id>/groups` | the groups the user is a member of: with `format=versions` `{groupID: version}`, which a client compares with its own (a group missing from it is one the user was taken out of, or deleted), otherwise their JSON as below. The `Last-Modified-Version` is the user's library's |
| `GET groups/<id>` | the group's JSON, as zotero.org's: `{id, version, links, meta: {created, lastModified, numItems}, data: {id, version, name, owner, type: "Private", description, url, libraryEditing, libraryReading: "members", fileEditing, admins, members}}`, with the owner, admins and members as user IDs (the owner and admins are not repeated among the members), and the group's version in `Last-Modified-Version`: every change to the group or its members makes a new one |
| `GET /schema` | Zotero's item schema, fetched from `api.zotero.org/schema`, stored, revalidated daily |
Object keys are 8 characters from `23456789ABCDEFGHIJKLMNPQRSTUVWXYZ`;
others fail with `400`. A keyless object gets a fresh key. Item data is
stored verbatim; attachments and annotations come back with `linkMode` /
`annotationType` first. Top-level items carry `meta.numChildren`. In a group
library, objects also carry `meta.createdByUser` and `meta.lastModifiedByUser`,
each `{id, username, name}`: the member who added the object and the one who
last changed it, as zotero.org's do.
### Listings
Item listings accept:
| Parameter | Meaning |
|---|---|
| `q`, `qmode` | case-insensitive substring; `titleCreatorYear` (default) searches title, date and creators; `everything` searches every field and tag (DOI, `extra`, URL, …) and stored full text |
| `itemType` | `book \|\| journalArticle`, `-attachment` |
| `tag` | repeat for AND, `\|\|` for OR |
| `sort`, `direction` | `dateModified` (default), `dateAdded`, `title`, `creator`, `date`, `itemType`; `asc`/`desc` |
| `limit`, `start` | page of ≤ 100 (default 25); `Total-Results` and `Link: <…>; rel="next"` on the public URL |
| `includeTrashed` | include items in the trash |
| `format=keys` | every matching key, unpaged, as plain text |
## Files
Attachment bytes live in object storage, under `users/<id>/files/<item key>`,
`groups/<id>/files/<item key>` for a group's, which count against its owner's
quota, or
`files/<item key>` for the library of a server from before there were several;
the database keeps their md5, size and mtime.
1. **Authorize** — `POST <lib>/items/<key>/file`, form `md5, filename,
filesize, mtime` (ms), with `If-None-Match: *` (new) or `If-Match: <md5>`
(replace); `428` without either, `404` for an unknown item. Answers
`{exists: 1}` when the same file is stored, `412` when another is (or when
`If-Match` names a different md5), `413` past `--max-upload-mb`, `413` with
`Zotero-Storage-Usage` and `Zotero-Storage-Quota` (MB) and
`Zotero-Storage-UserID` when the file would take the user past their quota (a
file it replaces does not count), otherwise
`{url, uploadKey, contentType, prefix: "", suffix: ""}` with an absolute
`url`. As on zotero.org, `exists` and `412` carry the item's version in
`Last-Modified-Version`, except a `412` for `If-Match` on an item with no
stored file, which carries none: Zotero then retries with
`If-None-Match: *`. Files added through Zotero's local API (as
`commonmeta sync --pdf` does) arrive that way.
Snapshots (`text/*` attachments) and attachments with several files come
as a ZIP of the attachment directory, with `zipMD5` and `zipFilename` added:
`md5` is then the main file's, `filesize` the ZIP's, and the upload is
checked against `zipMD5`. defter stores the ZIP as it is.
2. **Upload** — `POST <url>` with the raw bytes (no key: the token in the URL
is the capability). Streamed to `pending/<token>` while hashing; a md5 or
size mismatch is `400` and the bytes are dropped. `201`.
3. **Register** — `POST <lib>/items/<key>/file`, form `upload=<uploadKey>`:
moves the file to `files/<key>`, records it and stamps `md5`/`mtime` into the
item. `204`.
**Download** — `GET <lib>/items/<key>/file` answers `302` with
`Zotero-File-MD5`, `Zotero-File-Modification-Time`, `Zotero-File-Compressed`
(`Yes` for a stored ZIP, which Zotero unpacks) and `Location`: a pre-signed
bucket URL valid for `--presign-ttl` seconds, or, without signing (local
storage, `--proxy-downloads`), a `/downloads/<token>` link valid for five
minutes that streams the file through defter.
Deleting an attachment item deletes its file.
## Web UI
| Endpoint | Notes |
|---|---|
| `GET /` | signed in: the user's dashboard (their library, storage, profile and devices; with `--commonmeta-db` their commonmeta sync's settings and last run, and with `?sync=busy` a note that another run was going; for admins also where files are stored, backups and a link to `/ui/admin`), and with `HX-Request: true` (htmx's 30-second poll) only its status block (`Vary: HX-Request`). Otherwise `401` with a sign-in form, or for htmx's poll an empty `401` with `HX-Refresh: true`, which reloads the page. Not cacheable |
| `POST /ui/session` (form `key`) | sign in as the owner with an API key configured on the server, read-only included: sets the `defter_session` cookie (HttpOnly, SameSite=Lax, Secure with an HTTPS public URL, 7 days) and `303` to `/`. A wrong key gets the form again with `403` after a second's delay; a foreign `Origin` is rejected |
| `GET /ui/oidc/login?session=` | with `--oidc-issuer`: sends the browser to the provider, with a state tied to it by the `defter_oidc` cookie. `session` is a Zotero login to return to. `404` without `--oidc-issuer`, `502` if the provider cannot be read |
| `GET /ui/oidc/callback?state=&code=` | the provider's redirect back: for a user's ORCID iD sets the `defter_session` cookie and `303` to `/` or the Zotero login. An invited iD, or an admin's (`--admin-orcids`), that is no user's yet gets the `defter_account` cookie instead (with `--oidc-user-values`, an iD carrying one of its or `--oidc-admin-values`' values in the `--oidc-access-claim` claim of the ID token, else of userinfo, or an admin's; invitations then do not count) (HttpOnly, SameSite=Lax, `Path=/ui/account`, 15 minutes), naming a pending sign-in held in memory, and `303` to `/ui/account`. `403` with the sign-in form for anyone else or a disabled user, telling them to ask for access: by email to `--admin-email` when it is set, else the admins of this server. With `--oidc-user-values`, every sign-in also checks the values: a user without one gets `403` and is disabled, with their keys revoked and sessions ended (not the owner, nor an admin on `--admin-orcids`); when the claim is in neither the ID token nor userinfo, anyone but those gets `403` and nobody is changed; one a sign-in disabled so is enabled again by a sign-in with one, one an admin disabled is not; and whether they carry an admin value is kept as their admin right until their next sign-in |
| `GET /ui/account` | with a pending sign-in: the form to connect a zotero.org account (`Cache-Control: no-store`). `403` with the sign-in form, saying to sign in again, without one or when it has expired |
| `POST /ui/account` (form `key`, a zotero.org API key) | defter asks `<--zotero-api-url>/keys/current` with the key whose it is, then forgets it (never stored or logged), and makes the pending sign-in a user with that `userID` and `username` (their iD when another user has that name), or the owner when the `userID` is `--user-id`, the owner has no ORCID iD yet and the pending sign-in is an admin's (`--admin-orcids`; anyone else gets `409`), using up the invitation: the `defter_session` cookie, the `defter_account` cookie cleared, and `303` to `/` or the Zotero login. A key zotero.org rejects gets the form again with `403` after a second's delay, a `userID` another user has `409`, zotero.org unreachable `502`; no or an expired pending sign-in, or one no longer invited, `403` with the sign-in form; a foreign `Origin` `403` |
| `POST /ui/logout` | ends the session, clears the cookie, `303` to `/`. A session from an OpenID Connect provider gets a page that goes on to the provider's logout instead, which returns to `/` |
| `GET /<orcid>?q=&year=&type=&featured=&content=&lang=&page=` | the public profile of the user with that ORCID iD, without a key. `q` searches title, creators and date; `year`, `language`, `type` and `license` can repeat: values of one are ORed, different ones ANDed. `language` is an ISO 639-1 code and `license` an SPDX ID, read from Zotero's Language and Rights fields (`en-US`, `English`, `Deutsch` and `ger` are all `en` or `de`; `CC BY 4.0` and its URL are `CC-BY-4.0`); a work whose field names neither is left out of those facets. With none of these, or with `featured=1`, it shows up to five works with the manual tag `featured` (`featured=0` for all works; a bare address shows all of them if none is featured). `content=pdf` to works with a PDF link. `lang` is `en` or `de`, otherwise `Accept-Language` decides. With `HX-Request: true` only the list of works (`Vary: HX-Request, Accept-Language`). `404` for any other path, while hidden, and for a disabled user |
| `GET /<orcid>/export?format=&style=&…` | the works the list shows, under the same filters and without paging, as a file, or with `key` the one public work with that key (`404` for any other): `bibtex`, `commonmeta` (default), `crossref`, `crossref_xml`, `csl`, `datacite`, `datacite_xml`, `citation`, `inveniordm`, `ris` or `schemaorg`. `style` is for `citation`: `apa` (default), `chicago-author-date`, `harvard-cite-them-right`, `ieee`, `modern-language-association` or `vancouver`. `400` for anything else |
| `POST /ui/profile` (form `hidden`, `true` or `false`) | hide or show the signed-in user's public profile: `303` to `/`. Needs a session signed in with a read/write key (`401`, `403`); a foreign `Origin` is rejected, as on every `POST` below |
| `POST /ui/sync` (form `enabled`, `pdf`, `publications`, `deduplicate`, `overwrite`, each on when present and not `false`) | the signed-in user's choice for the server-side commonmeta sync of their own library (see [`commonmeta-sync.md`](commonmeta-sync.md)): `303` to `/`. `publications` is saved only for a user with an ORCID iD; `overwrite` lets commonmeta's container title, date, authors, license and DOI form replace the item's, not only fill what it lacks. `404` without `--commonmeta-db`; `401`, `403` as above |
| `POST /ui/sync/run` (form `dry_run`, `true` or `false`) | start a run of the signed-in user's library now, in the background, with their saved choices: `303` to `/`, or to `/?sync=busy` while another run is going, in the server or in `defter sync` (one at a time, held by `<data-dir>/sync.lock`). `dry_run=true` previews, writing nothing, and needs no opt-in; a real run needs `enabled` (`400` otherwise); a `dry_run` that is neither is a `400`. `404` without `--commonmeta-db`; `401`, `403` as above |
| `GET /ui/sync/log.json` | the signed-in user's last run or preview as a JSON download (`Content-Disposition: attachment`, `Cache-Control: no-store`): `{startedAt, finishedAt, status, dryRun, commonmetaVersion, summary, error}` as in `/ui/status`, with `results`, every per-item result kept (up to 1,000, those with changed fields first when there were more: `{key, doi, status, fields, pdf, message, changes}`, each change as `{"from", "to"}`, in a preview always and in a real run with `overwrite`), and `resultsTotal`, how many there were, more than kept when the log was cut short. `401` without a session, `404` without `--commonmeta-db` or before the first run |
| `POST /ui/groups` (form `name`) | create a group the signed-in user owns: `303` to its page |
| `GET /ui/groups/<id>` | a group's settings and members, for its members: `404` for others |
| `POST /ui/groups/<id>/settings` (form `name`, `description`, `library_editing`, `file_editing`) | for the owner and admins |
| `POST /ui/groups/<id>/members` (form `orcid`, or `user` for a member already, and `role`, `member` or `admin`) | add a user of the server by their ORCID iD, or change a member's role: the owner and admins add members, only the owner makes or unmakes admins. `?error=orcid` or `?error=user` for no iD or no user |
| `POST /ui/groups/<id>/remove` (form `user`, the signed-in user's own without) | take a member out, or leave: admins take out members, the owner anyone; the owner cannot leave |
| `POST /ui/groups/<id>/delete` | for the owner: members' clients are told the group is gone; its library is kept |
| `POST /ui/keys/revoke` (form `id`) | revoke one of the signed-in user's devices' keys: `303` to `/`, `404` for a key that is not theirs |
| `GET /ui/admin` | for admins: invitations (with `--oidc-user-values` instead the claim and values that let people in), users with their storage, devices and quota, and backups; with `--commonmeta-db` also how many users opted in to the commonmeta sync and when the newest sync ended, each user's setting and last run, and how many last runs failed. `404` for others, `303` to `/` when not signed in |
| `POST /ui/admin/invite` (form `orcid`, `note`) | invite an ORCID iD (or ORCID URL): `303` to `/ui/admin`, with `?error=orcid` for no iD and `?error=user` for a user's; `400` with `--oidc-user-values`. Admins with a read/write session only |
| `POST /ui/admin/uninvite` (form `orcid`) | withdraw an invitation not used yet |
| `POST /ui/admin/user` (form `id`, and `quota_mb`, empty for the default, or `disabled`, `true` or `false`) | set a user's quota (`?error=quota` for no number), or disable or enable them. Disabling revokes their keys and ends their sessions, and signing in does not enable them again; enabling hands someone a sign-in disabled back to `--oidc-user-values`; admins cannot disable themselves, nor the owner, whose keys are configured on the server (`400`) |
| `GET /assets/<name>` | the stylesheet and script (htmx), named by content hash and cached for a year; public |
| `GET /ui/status` | the dashboard's data as JSON, for monitoring, about the user of the API key (`Zotero-API-Key` or `Authorization: Bearer`) or the session cookie, `401` otherwise: their library, `quota` (bytes, `null` for none) and `storage.maxUpload`. For admins also `storage.location`, `storage.signedDownloads` and `backup`, which holds `enabled`, and the last run's `lastAt`, `manifest`, `error`, and `warning` when it succeeded with a problem, such as old backups it could not remove. For everyone, `sync` is `{enabled: false}` without `--commonmeta-db`, otherwise `{enabled: true, setting, running, lastRun}`: `setting` is `{enabled, pdf, publications, deduplicate, overwrite}`, `running` whether the server is running a sync of their library (not `defter sync`), and `lastRun` their newest run or preview (`null` for none) as `{startedAt, finishedAt, status, dryRun, commonmetaVersion, summary, error}`, with `status` `ok`, `partial`, `failed` or `aborted` and `summary` commonmeta's counts (camelCase: `items`, `withDoi`, `updated`, `pdfs`, `conflicts`, `failed`, …). For admins `sync` also holds `users`, `[{userId, setting, lastRun}]` for every user with a setting or a run |
Sessions are kept in memory, so a restart signs everyone out.
Pages are in English or German: `?lang=en` or `?lang=de`, which the `defter_lang` cookie
then remembers for a year, otherwise the browser's `Accept-Language`, otherwise English.
With `--oidc-issuer` and without `--api-key` (`DEFTER_API_KEY`), a key can't be typed in:
the forms are not offered, and `POST /ui/session` and a `POST /login` from a session not
signed in through the provider answer `403`. Keys keep working for the data API and
`GET /ui/status`.
Pages are rendered on the server from the askama templates in `templates/`
and work without JavaScript. They are sent with a Content-Security-Policy
that allows scripts and styles only from defter (plus a per-response nonce
for the inline theme script) and forbids framing; htmx is configured not to
use `eval` or inline styles, so it runs under that policy.
## Changes in 0.7.0
A user's ID is their zotero.org user ID, read from zotero.org with a key they
give at their first sign-in (`/ui/account`); new users no longer get a random
ID. `DEFTER_USER_ID` must be the owner's zotero.org user ID, and is required: it no
longer defaults to 1. Group IDs are
still defter's own. Users of a server set up earlier keep their old IDs, which
are not renumbered: such a server starts over with a new data directory (see
"Upgrading to 0.7.0" in the README).
## Out of scope
Group libraries, the streaming (websocket) server, binary-diff uploads
(`PATCH …/file`), and styled bibliographies.