# Changes in commonmeta-rs to sync with defter
`commonmeta sync` (commonmeta-rs 0.9.127, `src/cmd/sync.rs` and
`src/formats/zotero.rs`) talks only to the Zotero desktop app's **local API**
at `http://localhost:23119/api`. defter serves the Zotero **Web API v3**
(what the Zotero client syncs against) plus the parts of the local API that
commonmeta uses. This document lists what commonmeta-rs has to change so the
same `sync` can target a defter server as well as a local Zotero, and what
defter takes on itself so commonmeta-rs changes stay small.
References are to commonmeta-rs `main` at `fe44387` (v0.9.127).
## How `sync` uses the API today
| `LocalApi::connect` | `GET {base}/` — needs a `Zotero-Server-ID` header; `403` read as "local API off" | none |
| `schema` | `GET {base}/schema` | none |
| `top_items` | `GET {base}/{library}/items/top?format=json`, following `Link: rel="next"` | none |
| `children` (via `pdf_parents_of`, `scoped_items`) | `GET {base}/{library}/items/{key}/children?format=json` | none |
| `scoped_items` | `GET …/items?format=json&itemKey=a,b,…` and `GET …/items?format=json&qmode=everything&q=<doi suffix>` | none |
| `ensure_key` | `POST {base}/local/authorize` `{appName}` → `{key, remember}` (Zotero's "Always Allow" dialog), stored in `cache_dir("zotero")/local-api-key-<server-id>` | — |
| `patch_item` | `PATCH {base}/{library}/items/{key}` + `If-Unmodified-Since-Version: <item version>` → `200/204`, `412` | key |
| `put_item` | `PUT {base}/{library}/items/{key}` + `If-Unmodified-Since-Version: <item version>` | key |
| `create_item` (via `trash_copy`, `attach_pdf`) | `POST {base}/{library}/items` `[item]`, **no** `If-Unmodified-Since-Version` | key |
| `delete_item` (via `attach_pdf` cleanup) | `DELETE {base}/{library}/items/{key}` + `If-Unmodified-Since-Version` | key |
| `upload_file` (via `attach_pdf`) | `POST …/items/{key}/file` (form, `If-None-Match: *`) → `POST <url>` raw bytes → `201` → `POST …/file` `upload=<uploadKey>` → `204` | key (not on the byte upload) |
The library is `users/0` by default — the local API's alias for the logged-in user.
## Required changes in commonmeta-rs
### 1. Make the API base URL configurable
`cmd/sync.rs:224` hard-codes `LocalApi::connect(LOCAL_API)`.
- Add `--zotero-url <URL>` (env `ZOTERO_API_URL`), defaulting to `LOCAL_API`.
A defter base is `https://defter.example.org/api`.
- Optionally keep the last-used URL as a `settings` row (`zotero_url`), like
`ORCID_SETTING` (`zotero_orcid`) already is.
- Tell the two kinds apart: a target is **local** when its base is
`LOCAL_API` (or a loopback host on port 23119); anything else is **remote**.
Several of the points below depend on this.
### 2. Send the API key on reads, not only on writes
`LocalApi::get` sends no `Zotero-API-Key`: the local API's reads need none.
defter, like zotero.org, needs a key on **every** data request, and a
read-only key is enough for `--dry-run`.
- Remote target: send `Zotero-API-Key` on every request (`get` as well as `write`).
- Get the key *before* the first read, not lazily on the first write
(`ensure_key` runs only from `write` today).
### 3. Get a key the Web API way, not through `/local/authorize`
`ensure_key` falls back to `POST /local/authorize`, which only the Zotero
desktop app serves (it opens a dialog and blocks for up to 600 s). For a
remote target:
- Take the key from `--zotero-key-file <path>` or env `ZOTERO_API_KEY`
(keep `ZOTERO_LOCAL_API_KEY` for the local API, so the two never mix).
- Otherwise run the **login-session flow** the Zotero client uses against a
sync server (zhost verified it against Zotero 9): `POST {base}/keys/sessions` → `201 {sessionToken, loginURL}`; print
(or open) `loginURL`; poll `GET {base}/keys/sessions/{token}` until
`{status: "completed", apiKey, userID}`; `DELETE` the session on Ctrl-C or
timeout. The approval happens in the browser.
- Store the key as now (`local-api-key-<server-id>`, mode `0600`), but under a
name that covers remote servers too (e.g. `api-key-<server-id>`).
- Check the key with `GET {base}/keys/current` and stop early with a clear
message when `access.user.write` is `false` and the run is not `--dry-run`.
### 4. Treat `403` as a rejected key on remote targets
`write` drops a stored key and asks again only on `401`. zotero.org and defter
answer an unknown key with `403 Forbidden`. On a remote target, handle `403`
from a data request like `401` (drop the stored key, authenticate again, once).
The message after `connect` sees `403` ("Zotero's local API is off…") should
only appear for the local target.
### 5. Resolve `users/0` to the real user ID on remote targets
`users/0` means "the local user" only to the local API. On a remote target,
read `userID` from `GET {base}/keys/current` (or the login session's
`userID`) and use `users/<userID>` when `--library` is not given.
`validate_library` stays as it is. (defter also accepts `users/0` as an alias,
so this is defensive.)
### 6. Fall back to the published schema
`schema()` reads `GET {base}/schema`. Zotero's local API serves its bundled
schema; zotero.org serves it at `https://api.zotero.org/schema`; Zotero sync
servers usually don't serve it at all. If the target answers `404` (or is
remote and lacks it), fetch `https://api.zotero.org/schema` and cache it in
`cache_dir("zotero")`, revalidating with `If-None-Match`/`ETag`.
### 7. Honour `prefix`/`suffix` in file uploads
`upload_file` sends the raw bytes and ignores the `prefix`, `suffix` and
`params` fields of the authorization response. defter returns empty
`prefix`/`suffix`, so this works with defter as-is, but zotero.org returns an
S3 form upload that needs them. Send `prefix + bytes + suffix` with the given
`contentType`, as Zotero's own `storage/zfs.js` does. A remote `url` is
absolute; the existing "starts with `/`" branch can stay for the local API.
### 8. Page with `limit=100`
`read_items` follows `Link: rel="next"`; the Web API pages at 25 items by
default. Add `&limit=100` to the `top_items`, `children` and `scoped_items`
URLs to cut round trips by four on remote targets (the local API ignores it).
### 9. Respect `Backoff` and `Retry-After`
zotero.org (and later defter) may send `Backoff: <s>` or `429`/`503` with
`Retry-After: <s>`. Sleep for the given time before the next request, retry
`429`/`503` a few times, and stop rather than hammer the server.
### 10. Name and shape of the client
- Rename `LocalApi` to `ZoteroApi` (keep `pub type LocalApi = ZoteroApi;` for
one release) with a `kind: Local | Remote` field set in `connect`.
- `connect(base)` keeps requiring `Zotero-Server-ID` (defter sends one, stable
per installation), so the per-server key file still works.
- Print the target in the first line of output
(`zotero: defter 0.1.0 (<server-id>) at https://…, library users/1`);
`X-Zotero-Version` is empty for a remote target, so also read
`Server`/`X-Defter-Version` or fall back to the URL.
### 11. Tests
- Unit tests for the local/remote decision, key precedence
(`--zotero-key-file` > `ZOTERO_API_KEY` > stored > session flow), `403`
re-authentication, and the `limit`/schema-fallback URLs.
- An integration test against a mock server (or a defter binary, behind a
feature flag or `#[ignore]`) covering connect → read → patch → create →
file upload.
## Done on the defter side instead
These keep the commonmeta-rs changes above minimal. They are implemented in
defter 0.1 and covered by `src/api_tests.rs`
(`commonmeta_sync_reads_and_writes_single_items` and others):
- Serve the Zotero API under `/api` as well as `/`, so a base of
`https://host/api` works and `upload_file`'s `trim_end_matches("/api")` origin
logic stays correct.
- Answer `GET /api/` (and `GET /`) with `Zotero-Server-ID` (stable, stored in
the database) and a version header.
- Accept `users/0` as an alias for the configured user.
- Serve `GET /schema` (Zotero's `schema.json`, fetched from
`api.zotero.org`, stored and revalidated daily), without a key — so point 6
above matters only for other sync servers.
- Single-object endpoints `GET/PUT/PATCH/DELETE /users/{id}/items/{key}`,
where `If-Unmodified-Since-Version` is the **object's** version (Web API v3
semantics), unlike the multi-object endpoints where it is the library
version. zhost implements only the multi-object ones.
- `GET /users/{id}/items/{key}/children` (zhost lacks it), and `meta.numChildren`
on items.
- `POST /users/{id}/items` without `If-Unmodified-Since-Version` must succeed
(the header is optional in the Web API; zhost answers `428`, and
commonmeta's `create_item` sends none).
- `qmode=everything` must search **all fields** (DOI, `extra`, URL, …) as
well as full text: `scoped_items` finds items by DOI suffix that way. zhost's
`everything` adds only full-text content to title/creator/year.
- An absolute upload `url` (from `--public-url`), empty `prefix`/`suffix`,
`201` for the byte upload and `204` for registration (as in zhost).
- Read-only keys, so a `--dry-run` can use a key that cannot write.