# commonmeta sync on the server
defter can run [commonmeta](https://codeberg.org/front-matter/commonmeta-rs)'s
`sync` itself, every day, for the library of each user who opts in on their
dashboard. It fills in what the items with a DOI lack from the commonmeta
store (`commonmeta.sqlite3`), as `commonmeta sync` does against Zotero
desktop, and Zotero clients pick the changes up on their next sync. It is
meant for a server that runs commonmeta and dragoman as well, such as one a
university hosts for its researchers. Users need nothing on their machines.
## How it works
- **One server, no HTTP between services.** defter links commonmeta as a
library and calls its sync directly. commonmeta's sync reads and writes a
library through the `ZoteroLibrary` trait (`commonmeta::zotero`); for the
command line, commonmeta implements it over Zotero desktop's local API, and
defter (`src/sync.rs`) implements it over the user's library database and
file storage. Writes go through the same code as the API's, so they get new
versions and clients sync them like any other change.
- **Each database has one writer.** commonmeta writes the store, dragoman its
cache, defter its own databases (`control.sqlite3`, `libraries/`) and
attachment files. defter only ever reads the store: commonmeta opens it
read-only and defter runs the sync with `network` off, so a DOI the store
does not hold is skipped, not fetched from Crossref or DataCite (a fetched
record would be kept in the store).
- **`defter sync` runs it for everyone.** The daily run is a short-lived
command, `defter sync`, which commonmeta's crontab starts after its imports
(see [Scheduling](#scheduling)). It opens defter's databases beside the
running server, syncs every library whose user opted in and exits. SQLite's
locking keeps the two processes' writes apart: every write that reads first
takes the write lock at once (an `IMMEDIATE` transaction). The server has no
timer of its own.
- **The dashboard's runs stay in the server.** **Preview changes** and **Sync
now** run in the server process, in the background.
- **One run at a time.** A lock file in the data directory, `sync.lock`, is
held by every run, in the server or in `defter sync`. A run started from the
dashboard while another is going is refused; `defter sync` waits for one
started from the dashboard. `defter restore` and `defter migrate` hold it
too, and refuse to start while `defter sync` runs.
- **In order.** `defter sync` syncs one library after another: the owner
first, then whoever has waited longest since their last run. A run that
fails is recorded for that user and the next one goes on.
- **What still leaves the server.** Only PDFs, downloaded from where the store
says they were published, and only from public addresses: a URL, or a
redirect, to loopback, private, link-local or shared addresses is refused,
the address checked is the one connected to, and a download stops past
100 MB or `DEFTER_MAX_UPLOAD_MB`. An ORCID record the store lacks is not
fetched: My Publications is then left as it is.
- **Own libraries only.** The sync runs over each user's own library
(`users/<id>`), not over groups.
- **Opting out stops a run.** Before every write, defter checks that the user
is still opted in and not disabled. If not, the run stops at once with
status `aborted`; what it wrote before stays.
## What changes
commonmeta **fills in what an item lacks** — title, journal, publication
date, authors, language, license and more. A field the item already holds is
kept, so what users entered stays: defter never runs commonmeta's sync with
overwrite.
Where an item holds something else than commonmeta — another value in a field,
other authors or editors, or another type than the one its work calls for —
commonmeta **creates a duplicate** with its own metadata, of its type, in the
item's collections, related to the item (Zotero's Related) and not in My
Publications, and leaves merging to Zotero: Zotero's **Duplicate Items** finds
the two by their DOI, and the user merges them, picking each field's value.
Values are compared without case, accents, markup and punctuation, dates as far
as both give them, authors by last name and first initial.
An item gets one duplicate for each version of commonmeta's record of its work:
the duplicate's key is derived from the item's key, the DOI and a fingerprint
of commonmeta's metadata. While it waits in the library or the trash, none
other is made, and once merged (Zotero records `dc:replaces` on the item kept)
the sync leaves the item as it is. When commonmeta's record changes, a
duplicate still waiting to be merged goes to the trash, and the item, merged
or not, is compared again and gets a new one where it still differs. A
preview lists each difference as `{"from", "to"}`.
commonmeta merges nothing itself: items sharing a DOI, and items holding a
version of a work another item holds, are left as they are.
## Configuration
Setting `DEFTER_COMMONMETA_DB` turns the sync on, for the server (the
dashboard section) and for `defter sync`; without it, the server behaves as
before and the dashboard does not mention it.
| `DEFTER_COMMONMETA_DB` | `--commonmeta-db` | — | path to `commonmeta.sqlite3`. defter refuses to start when it is not a file it can see; `defter sync` needs it |
| `DEFTER_SYNC_LIMIT` | `--sync-limit` | `1000` | at most this many items changed or created per library and run; `0` for no limit. The rest are counted as left for the next sync, which reaches them once the items changed now need no change |
| `DEFTER_SYNC_MAX_PDFS` | `--sync-max-pdfs` | `1000` | at most this many PDFs attached per library and run |
The server and `defter sync` take the same settings, and `defter sync` also
those that place the databases and files (`DEFTER_DATA_DIR`, `DEFTER_USER_ID`,
the `DEFTER_S3_*` settings or `DEFTER_FILES_DIR`) and `DEFTER_MAX_UPLOAD_MB`,
`DEFTER_DEFAULT_QUOTA_MB` and `DEFTER_SCHEMA_URL`. Give it the server's
environment file.
At startup the server logs `commonmeta sync on: …` with the store, or
`commonmeta sync is off (no --commonmeta-db)`.
PDFs count against the user's storage quota and `DEFTER_MAX_UPLOAD_MB`, as
uploads do; one that does not fit is reported as failed and its attachment
removed again.
## `defter sync`
```sh
defter sync [--user <id>] [--dry-run] [--sync-limit <n>] [--sync-max-pdfs <n>]
```
Syncs the library of everyone who opted in, then exits:
| `0` | every library ran (`ok`, `partial` or `aborted`), or nobody opted in |
| `1` | it could not start: a command line or environment it refuses (such as no `DEFTER_COMMONMETA_DB`), no store, no control database in the data directory (start the server first), the users could not be read, or `--user` names a user who does not exist, is disabled or has not opted in |
| `2` | a library's run failed (status `failed`); the others ran |
- `--user <id>`: only that user's library, as when rolling the sync out. With
`--dry-run`, the user need not have opted in, as with the dashboard's
preview.
- `--dry-run`: writes nothing to the libraries and records each run as a
preview, which the user sees on their dashboard with what it would change.
It logs plain lines to stderr, for cron's mail or the journal:
`commonmeta sync starting` with how many libraries, one `commonmeta sync of a
library` per run with its status and counts, and `commonmeta sync finished`
with how many ended `ok`, `partial` and `failed`. Set `RUST_LOG` as for the
server.
## Scheduling
Recurring jobs on the server are all configured with commonmeta's command
line, `commonmeta jobs`, which keeps them in the crontab of the user who runs
it (only lines tagged `# commonmeta-job:<name>` are touched). Its `defter` job
runs `defter sync` daily, by default at noon, after the Crossref and DataCite
imports: the job waits for the imports' lock beside the store, and lets go of
it before `defter sync` starts, since defter only reads the store.
```sh
commonmeta jobs add defter --command \
"DEFTER_COMMONMETA_DB=/path/to/commonmeta.sqlite3 /usr/local/bin/defter sync"
commonmeta jobs list
commonmeta jobs run defter # now, as cron would
```
`--at HH:MM` (local time, as cron reads it) moves it. cron starts the command
with `sh -c`, a minimal `PATH` and none of defter's environment, so the
command must bring that along (see below for Debian with systemd). On a
single-user machine (macOS, say) running defter as that user, the defaults of
the data directory apply and the store's path is enough:
```sh
commonmeta jobs add defter --command \
"DEFTER_COMMONMETA_DB='$HOME/Library/Application Support/commonmeta/commonmeta.sqlite3' /usr/local/bin/defter sync"
```
## What users choose
The dashboard's commonmeta section (shown only when the sync is configured,
with buttons only for a session that may write) has two choices, each on its
own and off until the user changes it. A library with either is synced daily,
and can be synced now.
- **Add the works in my ORCID record to My Publications**: an item holding
one of its DOIs is put into My Publications, a work the library lacks is
created there. Rogue Scholar and Zenodo give a work a concept DOI and a DOI
per version, and only the versions carry the PDF: a concept DOI without a
PDF gives way to the work's version DOI, and a work the ORCID record lists
only by such a concept DOI is left out (counted on the dashboard). Needs an
ORCID iD on the account; without one it is not offered and not saved. On
its own, the works' metadata is left as it is.
- **Update the works metadata in my Zotero library**: what "What changes"
above describes, and a PDF commonmeta knows attached to an item that has
none, up to `DEFTER_SYNC_MAX_PDFS` per run.
Earlier there were three choices: keeping the library up to date
(needed for the others), attaching PDFs and My Publications. Keeping it up to
date became updating the metadata, PDFs included; My Publications stays on
where the library was kept up to date too.
Two buttons:
- **Preview changes** runs the sync without writing anything and lists what
it would change: the fields it fills, and `field: from → to` where it adds
to a value, such as authors, Extra or tags. It needs no choice made, so
users can look before they decide: it uses the choices as saved, or both
when none is made yet.
- **Sync now** runs a real sync of the user's library with their choices. It
is shown only once they made one. If another run is going, here or in
`defter sync`, the dashboard says so; try again in a few minutes.
Both run in the background; the dashboard shows that a sync of the library is
running (one the server runs), and afterwards how the last run (or preview)
went: when it ended, its status, its counts in words (with how many items
were left for the next sync), its error if it failed, and the first 100
per-item changes.
**Download the changes as JSON** (`GET /ui/sync/log.json`) gets the last run
or preview in full: its counts and every per-item result, each change as
`{"from", "to"}` in a preview; a real run only fills empty fields, so it lists
their names.
## Runs
A run ends as `ok`, `partial` (some items failed, or changed in Zotero
meanwhile and are left for the next run), `failed` (the sync stopped with an
error) or `aborted` (the user opted out or was disabled while it ran).
The control database keeps the 20 newest runs of each user, previews
included, with their counts. Only the newest keeps its per-item results, up
to 1,000 of them. A run changes at most 1,000 items by default, but PDFs,
duplicates, conflicts and failures add results of their own, so a run
can have more: then the items given new values (with their old and new
values) are kept first, the rest in order, and the log says how many there
were (`resultsTotal`). The dashboard lists the first 100 and says how many
more there are; the JSON log has all that were kept.
## Dates
To reach Zotero as a change, an item written on the server gets its dates as
zotero.org stamps them. On every item write through defter, from Zotero or
from the sync:
- `dateAdded` is kept: the one sent, else the stored one, else now.
- `dateModified` is now when the write sends none, or when a `PUT` sends back
the stored value unchanged. Otherwise the one sent is kept.
Zotero desktop sends its own `dateModified`, so syncing from it changes
nothing; commonmeta sends none, so its changes are dated when they were made.
## Monitoring
- **Admin page** (`/ui/admin`): a line with how many users opted in, when the
newest sync ended, and how many users' last sync failed; each user's
setting (off, metadata, My Publications, or both) and their last run. Admins can see but
cannot opt users in or out; disabling a user stops their sync.
- **`/ui/status`**: `sync.enabled`, and when configured the user's `setting`,
whether the server is running a sync of their library (`running`), and
their `lastRun` (`startedAt`, `finishedAt`, `status`, `dryRun`,
`commonmetaVersion`, `summary`, `error`). For admins also `sync.users`: for
every user with a setting or a run, `userId`, `setting` and `lastRun`. See
[`api.md`](api.md).
- **`defter sync`'s exit code and log**, in cron's mail or the journal, and
`commonmeta jobs`' own `started`, `finished` or `failed` line for the job.
Alert when, for a user with `setting.metadata` or `setting.publications`,
`lastRun.status` is `failed`,
or `lastRun.finishedAt` is older than 36 hours (the daily run did not
happen). A preview counts as the last run, so check `dryRun` too if users
preview often.
## Deployment on Debian with systemd
The server runs as the `defter` user under `ProtectSystem=strict`, which makes
the file system read-only apart from its own `StateDirectory`. To read the
store it needs:
- read access to `commonmeta.sqlite3` and search access to the directories
above it;
- as a reader of an SQLite database in WAL mode, write access to its `-shm`
and `-wal` files, which every reader updates. Without it, commonmeta falls
back to opening the store as immutable, which can read a half-written state
while commonmeta writes.
With commonmeta's files in a group `commonmeta`, for example, add to
`defter.service` (both lines are there, commented out):
```ini
SupplementaryGroups=commonmeta
ReadWritePaths=/var/lib/commonmeta
```
`ReadWritePaths=` only lifts systemd's read-only mount; the files' own
permissions still decide. The store's path and permissions differ between
servers, so check them on yours first:
```sh
namei -l /var/lib/commonmeta/commonmeta.sqlite3 # each directory: group x
ls -l /var/lib/commonmeta/ # .sqlite3 g+r, -shm and -wal g+rw
```
The `-shm` and `-wal` files must stay group-writable when commonmeta creates
them anew (its umask); if they may be missing while defter opens the store,
the directory needs group write as well. A store under `/home` is hidden by
`ProtectHome=true`. Then, in `/etc/defter/env`:
```sh
DEFTER_COMMONMETA_DB=/var/lib/commonmeta/commonmeta.sqlite3
```
and `systemctl daemon-reload && systemctl restart defter`.
`defter sync` must run as the `defter` user too, with the server's
environment, since it writes defter's databases and files. Outside systemd
there is no `$STATE_DIRECTORY`, so give the data directory explicitly. Add the
job to the `defter` user's crontab:
```sh
sudo -u defter commonmeta jobs add defter \
--file /var/lib/commonmeta/commonmeta.sqlite3 \
--command 'set -a; . /etc/defter/env; DEFTER_DATA_DIR=/var/lib/defter exec /usr/local/bin/defter sync'
sudo -u defter commonmeta jobs list
sudo -u defter commonmeta jobs run defter # once now, to check
```
`. /etc/defter/env` reads the file as shell, so its values must be plain
`KEY=value` without spaces (quote them otherwise). The job waits for the
imports' lock beside the store (`commonmeta.sqlite3.jobs.lock`), so the
`defter` user must be able to open it: the same group access as for the
`-shm` and `-wal` files. The import jobs stay in the crontab of the user that
owns the store.
### Rollout
1. The owner first: opt in on the dashboard, **Preview changes** and read the
list (or download it as JSON), then **Sync now** and check the dashboard's
last run.
2. Sync Zotero desktop once and check that it takes the changes without
conflicts.
3. Run `defter sync --user <id> --dry-run`, then `defter sync --user <id>`,
for a few users (with the environment as in the crontab line), and check
their dashboards.
4. Add the crontab job, and watch the admin page, `/ui/status` and cron's
mail for a few days before telling everyone.
### Release note
The control database moves to schema 2 (`migrations/control/0002_sync.sql`),
whether the sync is configured or not. An older defter refuses a control
database at a schema newer than it knows, so once this release has started,
there is no downgrade past it. `DEFTER_SYNC_AT` is gone: `start` refuses
`--sync-at`, and an environment file that still sets it is ignored.