dragoman
A web server for scholarly metadata with full DOI Resolution and DOI content negotiation. Send a DOI as the URL path; receive a redirect to the landing page or metadata in any supported format depending on the Accept header.
Installation
Prebuilt binary (recommended)
If you have cargo-binstall, install a prebuilt binary without compiling:
Prebuilt binaries are published for macOS arm64 (aarch64-apple-darwin) and Linux amd64 (x86_64-unknown-linux-gnu). On any other platform, cargo binstall automatically falls back to building from source. The binary is installed to ~/.cargo/bin/dragoman; make sure ~/.cargo/bin is on your PATH (the Rust installer adds this automatically).
Build from source
Requires Rust 1.75+ (rustup.rs):
This builds a release binary and installs it to ~/.cargo/bin/dragoman.
Local SQLite database
dragoman can serve metadata directly from a local SQLite database in the commonmeta format, bypassing the live Crossref/DataCite APIs. This dramatically reduces latency and API load for high-traffic deployments.
Database format
The database is a SQLite3 file in the commonmeta store layout:
pid_recordsholds each record as its provider delivered it — Crossref and DataCite works, ROR organizations, ORCID persons, and containers — keyed by bare pid (10.5281/zenodo.1234,0000-0002-1825-0097,02nr0ka47) and converted to commonmeta on read.pid_relationsholds the links between them thatcommonmeta enrichderives: references and typed relations between works, and a work's persons (hasPerson), organizations (hasOrganization) and container (hasContainer).pid_annotationsholds whatenrichand the imports add per pid: the validated commonmeta, identifier columns such aspmid,openalexandissn, and the stored work counts of long works lists.
You can build a database from any commonmeta-supported source using the commonmeta CLI.
Running the server
Start
# Default port 3456
# Custom port
# With a local database
# Write a PID file so the process can be stopped later
# All options together
TLS
The --domain flag controls whether and how dragoman serves HTTPS.
--domain value |
Behaviour |
|---|---|
| (not set) | Plain HTTP on 127.0.0.1:<port>. Use this behind a reverse proxy. |
localhost |
HTTPS on 127.0.0.1:<port> using a locally-trusted certificate from mkcert. |
| any public hostname | HTTPS on 0.0.0.0:<tls-port> with automatic certificate provisioning via Let's Encrypt. Requires --email. |
any public hostname, with --tls-cert and --tls-key |
HTTPS on 0.0.0.0:<tls-port> with the certificate in those PEM files, e.g. a CDN origin certificate. See Behind a CDN. |
Local development (mkcert)
Install mkcert once and add its CA to your trust store:
Then start dragoman with --domain localhost:
dragoman runs mkcert automatically on the first start; subsequent starts reuse the cached certificate. Certificates are stored in --acme-cache (default /var/lib/dragoman/acme).
Production (Let's Encrypt)
dragoman provisions and renews certificates automatically using the ACME TLS-ALPN-01 challenge. No separate HTTP challenge server or reverse proxy is needed.
- Binds HTTPS on
--tls-port(default 443) with automatic certificate provisioning. - Binds HTTP on
--portwith a permanent redirect tohttps://commonmeta.org/…. - Stores certificates in
--acme-cache(default/var/lib/dragoman/acme) so they survive restarts.
Use --acme-staging while testing to avoid Let's Encrypt rate limits:
Behind a CDN
A CDN that proxies traffic answers the TLS handshake itself, so Let's Encrypt's TLS-ALPN-01 challenge never reaches dragoman and renewal fails. Serve a certificate the CDN trusts instead, such as an origin certificate it issues, and tell dragoman which peers are the CDN's edges, so that request limits count the clients behind them rather than the edges:
--trusted-proxies takes the CDN's published edge networks and --client-ip-header the header the CDN puts the client's address in. scripts/cloudflare-trusted-proxies.sh prints Cloudflare's full list from its published ranges; --env prints it as a line for /etc/dragoman/env:
|
Cloudflare changes its ranges rarely; rerun the script now and then, and restart dragoman when the list changes. The certificate files are also read at startup only, so restart dragoman after replacing them. See Caching for the headers a CDN keys on.
Cloudflare Tunnel
With Cloudflare Tunnel, cloudflared on the server dials out to Cloudflare and hands requests to dragoman locally, so the server accepts no inbound HTTP at all and needs no certificate. Run dragoman without --domain, on plain HTTP at 127.0.0.1, point the tunnel's ingress at it (service: http://127.0.0.1:3456), and name the header Cloudflare sets:
cloudflared connects from this machine, which dragoman always trusts, so --trusted-proxies is not needed. --client-ip-header is: without it dragoman would take X-Real-IP from the local peer, and Cloudflare passes on whatever X-Real-IP the client sent.
Firewall
Behind a CDN, only the CDN should reach dragoman. The Hetzner Robot firewall of a dedicated server takes at most 10 rules, too few for Cloudflare's 22 ranges, so filter on the server instead — or use Cloudflare Tunnel and close the HTTP ports altogether. The Robot firewall then only needs SSH and, without a tunnel, 443.
With nftables, let SSH in from anywhere and 443 only from Cloudflare. --nft prints the ranges as nftables variables:
|
#!/usr/sbin/nft -f
flush ruleset
include "/etc/nftables.d/cloudflare.nft"
table inet filter {
chain input {
type filter hook input priority 0; policy drop;
iif lo accept
ct state established,related accept
meta l4proto { icmp, ipv6-icmp } accept
tcp dport 22 accept
ip saddr $cloudflare_v4 tcp dport 443 accept
ip6 saddr $cloudflare_v6 tcp dport 443 accept
}
}
Unlike the Robot firewall, nftables tracks connections, so the replies to dragoman's own requests to Crossref and DataCite need no rule of their own.
Options can also be supplied as environment variables (flags take precedence):
PORT=8080 COMMONMETA_DB=/data/commonmeta-2026-06-15.sqlite3 RUST_LOG=dragoman=debug
During development you can use cargo run in place of the installed binary:
# Run from the project root — the sqlite3 file in the root is loaded by filename
# Or with a full path
Error: port already in use
If the chosen port is already in use, the server logs an error and exits:
ERROR dragoman: failed to bind port=3456 error=Address already in use (os error 48)
Choose a different port with --port or stop the process that holds the port.
Database file not found
If the resolved database path does not exist, dragoman logs a warning and starts without a local database, falling back to the live API for all requests:
WARN dragoman: sqlite file not found, running without local database path=…
To use a local database, place the SQLite file at the platform default path or set COMMONMETA_DB to its location.
Stop
# Stop using the default PID file location (/tmp/dragoman.pid)
# Stop using a custom PID file
dragoman stop sends SIGTERM to the running process. The server handles the signal gracefully: it finishes in-flight requests and removes the PID file before exiting. Pressing Ctrl-C has the same effect.
CLI reference
dragoman <COMMAND>
Commands:
start Start the server (runs in the foreground)
stop Stop a running server by sending SIGTERM to its PID file
import Import scholarly metadata into the local commonmeta database
convert Convert a metadata record from one format to another
list List records stored in the local commonmeta database
stats Show record counts and other statistics for the local database
settings Show key/value settings stored in the local SQLite database
help Print help
dragoman start [OPTIONS]
-p, --port <PORT> TCP port to listen on [env: PORT] [default: 3456]
-d, --db <PATH> Local commonmeta SQLite3 database [env: COMMONMETA_DB]
--pid-file <PATH> Write PID to this file on startup [env: DRAGOMAN_PID_FILE]
--domain <DOMAIN> Domain for HTTPS. Use `localhost` for local dev (mkcert) or a public hostname for Let's Encrypt [env: DRAGOMAN_DOMAIN]
--email <EMAIL> ACME contact email for Let's Encrypt (required for public domains) [env: DRAGOMAN_EMAIL]
--acme-cache <PATH> Directory for ACME certificate storage [env: DRAGOMAN_ACME_CACHE] [default: /var/lib/dragoman/acme]
--acme-staging Use Let's Encrypt staging (testing only) [env: DRAGOMAN_ACME_STAGING]
--tls-port <PORT> HTTPS port when --domain is set [env: DRAGOMAN_TLS_PORT] [default: 443]
--rate-pages <R/B> Per-client limit on entity pages [env: DRAGOMAN_RATE_PAGES] [default: 20/60]
--rate-lists <R/B> Per-client limit on works lists, facets, counts [env: DRAGOMAN_RATE_LISTS] [default: 5/20]
--rate-expensive <R/B> Per-client limit on search, handles, exports, conversion [env: DRAGOMAN_RATE_EXPENSIVE] [default: 2/20]
--max-in-flight <N> Requests served at once, across all clients [env: DRAGOMAN_MAX_IN_FLIGHT] [default: 256]
--request-timeout <S> Seconds a response may take [env: DRAGOMAN_REQUEST_TIMEOUT] [default: 60]
--header-timeout <S> Seconds for the TLS handshake and request headers [env: DRAGOMAN_HEADER_TIMEOUT] [default: 10]
--max-conns-per-ip <N> Open connections per client [env: DRAGOMAN_MAX_CONNS_PER_IP] [default: 32]
--trusted-proxies <NETS> Proxies that may name the client, e.g. a CDN's edges [env: DRAGOMAN_TRUSTED_PROXIES]
--client-ip-header <NAME> Header a trusted proxy names the client in [env: DRAGOMAN_CLIENT_IP_HEADER]
--tls-cert <PATH> PEM certificate for --domain instead of Let's Encrypt [env: DRAGOMAN_TLS_CERT]
--tls-key <PATH> PEM private key for --tls-cert [env: DRAGOMAN_TLS_KEY]
--reuse-port Share the listening ports with other dragoman processes [env: DRAGOMAN_REUSE_PORT]
dragoman stop [OPTIONS]
--pid-file <PATH> PID file to read [env: DRAGOMAN_PID_FILE] [default: /tmp/dragoman.pid]
Beyond start and stop, dragoman embeds the full commonmeta CLI. import, convert, list, stats and settings are listed above for convenience, and any other commonmeta subcommand (build, enrich, validate, push, migrate, …) is forwarded verbatim — no need to install commonmeta separately. Each command's own flags are passed straight through:
Run dragoman <command> --help for a command's full options.
Environment variables
All flags can also be supplied as environment variables; flags take precedence. See env.example for an annotated template.
| Variable | Default | Description |
|---|---|---|
PORT |
3456 |
TCP port to listen on. |
COMMONMETA_DB |
(platform default) | Path to a local commonmeta SQLite3 database. Served before falling back to the live API. Platform defaults: macOS → ~/Library/Application Support/commonmeta/commonmeta.sqlite3; Linux → /var/lib/commonmeta/commonmeta.sqlite3. |
DRAGOMAN_CACHE_DB |
(platform default) | Path to the dragoman cache database for API fallback results. Platform defaults: macOS → ~/Library/Application Support/commonmeta/cache.sqlite3; Linux → /var/lib/commonmeta/cache.sqlite3. |
DRAGOMAN_PID_FILE |
(none) | Path for the PID file written by start and read by stop. |
DRAGOMAN_DOMAIN |
(none) | Domain for HTTPS. localhost → mkcert; public hostname → Let's Encrypt. Unset → plain HTTP on 127.0.0.1 (reverse proxy mode). |
DRAGOMAN_EMAIL |
(none) | ACME contact email. Required when DRAGOMAN_DOMAIN is a public hostname. |
DRAGOMAN_ACME_CACHE |
(platform default) | Directory for ACME/mkcert certificates. Platform defaults: macOS → ~/Library/Application Support/dragoman/acme; Linux → /var/lib/dragoman/acme. |
DRAGOMAN_ACME_STAGING |
(false) | Set to any non-empty value to use the Let's Encrypt staging environment. |
DRAGOMAN_TLS_PORT |
443 / 3456 |
HTTPS port when DRAGOMAN_DOMAIN is set. Defaults to 3456 for localhost, 443 for public domains. |
DRAGOMAN_VIKTORIALOGS_URL |
(none) | VictoriaLogs OTLP/HTTP endpoint. When unset, logs go to stdout only. |
DRAGOMAN_METRICS_ALLOW |
(none) | Comma-separated IPs allowed to scrape /metrics. When unset, the route is not registered. |
DRAGOMAN_RATE_PAGES |
20/60 |
Per-client limit on entity pages, as requests per second / burst. 0 turns it off. See Request limits. |
DRAGOMAN_RATE_LISTS |
5/20 |
Per-client limit on works lists, facets and counts. |
DRAGOMAN_RATE_EXPENSIVE |
2/20 |
Per-client limit on search, handle lookups, exports, bibliographies and format conversion. |
DRAGOMAN_MAX_IN_FLIGHT |
256 |
Requests served at once across all clients; past it, 503. 0 turns the cap off. |
DRAGOMAN_REQUEST_TIMEOUT |
60 |
Seconds a response may take before it is abandoned with 503. 0 turns it off. |
DRAGOMAN_HEADER_TIMEOUT |
10 |
Seconds a client has to finish the TLS handshake, send its first request and each request's headers. HTTPS with a public domain only. |
DRAGOMAN_MAX_CONNS_PER_IP |
32 |
Open connections per client; an IPv6 client is its /64. HTTPS with a public domain only. 0 turns the cap off. |
DRAGOMAN_TRUSTED_PROXIES |
(none) | Comma-separated addresses and CIDR networks of proxies that may name the client, such as a CDN's edges; this machine is always trusted. Their connections are exempt from DRAGOMAN_MAX_CONNS_PER_IP. |
DRAGOMAN_CLIENT_IP_HEADER |
(none) | Header a trusted proxy names the client in, e.g. CF-Connecting-IP. |
DRAGOMAN_TLS_CERT |
(none) | PEM certificate (chain) to serve for DRAGOMAN_DOMAIN instead of one from Let's Encrypt. Requires DRAGOMAN_TLS_KEY. |
DRAGOMAN_TLS_KEY |
(none) | PEM private key for DRAGOMAN_TLS_CERT. |
DRAGOMAN_REUSE_PORT |
(false) | Set to true to share the listening ports with other dragoman processes (SO_REUSEPORT). See Several processes on one port. |
RUST_LOG |
dragoman=info |
Log filter (see tracing-subscriber). Use dragoman=debug for per-request cache hits. |
DOI Resolution
dragoman implements the Handle System resolution protocol. Two entry points are available for each PID:
| Endpoint | Purpose |
|---|---|
/{path} |
Render the entity page (HTML · Svelte UI) or content-negotiate (API) |
/api/handles/{path} |
Handle resolver: redirect to URL of record, content negotiation, or ?noredirect |
Entity page — /{path}
Visiting a PID in a browser renders the dragoman entity page (Svelte UI with rich metadata). Sending an API Accept header to the same URL triggers content negotiation instead (see Content negotiation).
# Open entity page in a browser
# Works for ROR organisation IDs and ORCID person IDs too
Paths that are not recognised PIDs redirect to the search page:
# Non-PID path → 307 to /?q=<path>
Handle resolver — /api/handles/{path}
The /api/handles/ endpoint is the programmatic resolution interface. Three outcomes are possible depending on the request:
Redirect to URL of record (default)
Without a recognised Accept header, /api/handles/{path} redirects to the PID's URL of record:
| PID type | Redirect target |
|---|---|
| DOI | URL from metadata (data.url); fallback https://doi.org/{doi} |
| ROR | Organisation website (from database); fallback https://ror.org/{id} |
| ORCID | https://orcid.org/{id} |
# Follow the redirect to the landing page
# Inspect the redirect target without following
# https://zenodo.org/record/1089100
View entity page — ?noredirect
The ?noredirect parameter redirects the browser to /{path} to render the entity page:
# 307 → http://localhost:3456/10.5281/zenodo.1089100
Content negotiation at /api/handles/{path}
Send a recognised Accept header to receive metadata in a specific format.
DOI
See Supported formats for the full list. The ?format= override is also accepted.
ROR (organisations)
Accept header |
?format= value |
Output |
|---|---|---|
application/json |
ror |
ROR native JSON |
application/vnd.commonmeta+json |
commonmeta |
commonmeta JSON |
# ROR native JSON
# commonmeta JSON
# ?format= override
ORCID (people)
Accept header |
?format= value |
Output |
|---|---|---|
application/vnd.orcid+json |
orcid |
commonmeta JSON (native ORCID JSON planned) |
application/json |
orcid |
commonmeta JSON (native ORCID JSON planned) |
application/vnd.commonmeta+json |
commonmeta |
commonmeta JSON |
# ORCID JSON (currently returns commonmeta JSON; native ORCID JSON planned)
# commonmeta JSON
# ?format= override
Usage
Content negotiation
Send an Accept header to /{path} to receive metadata for a DOI without a browser redirect.
BibTeX
RIS
CSL (Citeproc) JSON
Crossref
Crossref XML
DataCite
Schema.org JSON-LD
InvenioRDM
Formatted citation
text/x-bibliography accepts optional style= and locale= parameters. Style names come from the CSL style repository; locale codes from the CSL locales repository.
# APA (default)
# Vancouver in French
Query parameter overrides
Use ?format= instead of an Accept header:
Format-specific URL prefixes
Two URL prefixes return a fixed format without requiring an Accept header or ?format= parameter:
| Prefix | Output format | Content-Type |
|---|---|---|
/dois/{doi} |
DataCite JSON | application/vnd.datacite.datacite+json |
/works/{doi} |
Crossref JSON | application/vnd.crossref+json |
These routes are equivalent to /{doi}?format=datacite and /{doi}?format=crossref respectively. The ?source= override is still accepted if you need to force a specific registration agency.
Bulk export — /api/export
One request returns every work of one entity in a single document, in any of
the formats below. The entity is named by id, spelled exactly as its page path
spells it — a ROR id, an ORCID iD, or a container's ISSN, OpenAlex source id or
DOI:
# Every work of an organization as BibTeX
# A person's works as RIS, a journal's as formatted citations
This is what the entity page's Export and Copy buttons call. The works list on
the page is paged (/api/{org,person}/{id}/works, /api/container/works), so
exporting from the client would have covered only the pages loaded so far, at
one request per work.
| Response header | Meaning |
|---|---|
X-Total-Count |
The entity's full works total |
X-Exported-Count |
How many the body holds |
X-Export-Truncated |
Present when the export hit the ceiling |
The two counts also differ for an ordinary reason: a work superseded by a newer
version is left out. X-Export-Truncated is what distinguishes truncation —
an export writes at most 5000 works, or 1000 for citation, which renders every
record through the CSL processor.
Supported formats
DOI formats (works)
Accept header |
?format= value |
Notes |
|---|---|---|
application/x-bibtex |
bibtex |
|
text/x-bibliography |
citation |
style= and locale= params |
application/vnd.commonmeta+json |
commonmeta |
|
application/vnd.crossref+json |
crossref |
|
application/vnd.crossref.unixref+xml |
crossref_xml |
|
application/vnd.crossref.unixsd+xml |
crossref_xml |
alias |
application/vnd.citationstyles.csl+json |
csl |
|
application/vnd.datacite.datacite+json |
datacite |
|
application/vnd.datacite.datacite+xml |
datacite_xml |
|
application/vnd.inveniordm.v1+json |
inveniordm |
|
application/x-research-info-systems |
ris |
|
application/vnd.schemaorg.ld+json |
schemaorg |
|
text/html / (absent) at /{path} |
— | Svelte entity page |
text/html / (absent) at /api/handles/{path} |
— | 307 redirect to URL of record |
ROR formats (organisations) — /api/handles/{id} only
Accept header |
?format= value |
Notes |
|---|---|---|
application/json |
ror |
ROR native JSON |
application/vnd.commonmeta+json |
commonmeta |
ORCID formats (people) — /api/handles/{id} only
Accept header |
?format= value |
Notes |
|---|---|---|
application/vnd.orcid+json |
orcid |
commonmeta JSON (native ORCID JSON planned) |
application/json |
orcid |
commonmeta JSON (native ORCID JSON planned) |
application/vnd.commonmeta+json |
commonmeta |
HTTP status codes
| Code | Meaning |
|---|---|
| 200 | Metadata returned |
| 307 | Redirect (entity page, URL of record, or search) |
| 404 | PID not found or path not recognised |
| 406 | Requested content type not supported |
| 429 | Client over its request limit; see Retry-After |
| 502 | Upstream API error |
| 503 | Server busy or response timed out; see Retry-After |
Deployment (macOS)
Place the database
The platform default path on macOS is ~/Library/Application Support/commonmeta/commonmeta.sqlite3. Place the database there and no further configuration is needed:
To use a different path, set COMMONMETA_DB in the launchd plist or pass --db on the command line.
Run as a background service (launchd)
# Start at login and keep alive
# Check status
# View logs
# Stop the service
brew services start installs a launchd plist in ~/Library/LaunchAgents/ and starts the service immediately. It restarts automatically on crash and at login.
To run as a system-level daemon (starts at boot, not tied to a user login), use sudo brew services start dragoman. This installs the plist in /Library/LaunchDaemons/ instead.
Configuration
To change the port or other settings, edit the service environment variables and restart:
# Open the generated plist for editing
Manual installation (without Homebrew)
# Install Rust if not already installed
|
Intel Macs: replace
/opt/homebrewwith/usr/localin all paths below.
Run as a launchd daemon
The bundled com.front-matter.dragoman.plist targets Apple Silicon paths.
Check logs:
Stop and disable:
Updating
With Homebrew
Manual
Deployment (Debian / systemd)
This section covers running dragoman as a persistent system service on a Debian 13 server.
1. Build the binary
On the server, install Rust and install the binary:
|
--root /usr/local writes the binary straight to /usr/local/bin/dragoman,
so there is no second copy to forget on an upgrade. It has to live outside a
home directory: the service runs as User=dragoman with ProtectHome=true, so
/root and /home are invisible to it and a binary under ~/.cargo/bin could
not be executed at all.
Or cross-compile locally and copy the binary:
# macOS → Linux x86-64 (requires cross)
2. Create system user and directories
3. Place the SQLite database
The platform default path on Linux is /var/lib/commonmeta/commonmeta.sqlite3. Place the database there and no further configuration is needed:
To use a different path, set COMMONMETA_DB in the environment file instead.
4. Create the environment file
dragoman reads /etc/default/commonmeta, shared with the commonmeta importer
service. Copy the bundled example and edit it for your deployment:
The cache database variable is DRAGOMAN_CACHE_DB — CACHE_DB is not read.
For a production server with Let's Encrypt, at minimum set:
DRAGOMAN_DOMAIN=commonmeta.org
DRAGOMAN_EMAIL=admin@example.org
The systemd unit grants write access only to /var/lib/commonmeta
(ReadWritePaths), where both databases live. The ACME certificate cache
defaults to /var/lib/dragoman/acme, which is outside that path — for TLS,
point it inside:
DRAGOMAN_ACME_CACHE=/var/lib/commonmeta/acme
See Environment variables for the full reference.
5. Install and enable the systemd unit
dragoman.service is included in the source repository but not installed by cargo install. Download it directly:
Or, if you have the source checked out locally:
Check the service is running:
Several processes on one port
With --reuse-port, several dragoman processes bind the same ports and the kernel hands each new connection to one of them, so one machine serves from as many processes as it has use for. Each still does its own TLS; no proxy sits in front. dragoman@.service is a template unit that starts one process per instance:
Restart them one at a time and the others keep serving: for i in 1 2 3 4; do sudo systemctl restart dragoman@$i; sleep 5; done.
The processes share nothing but the ports and the databases:
- Each keeps its own request limit buckets. A client's connections land on different processes, so with four processes a client can get up to four times its limit; divide the
DRAGOMAN_RATE_*values by the number of processes to keep the per-client budget. - Each keeps its own redirect cache and its own connections to the databases.
- With Let's Encrypt, each process orders and renews its certificate on its own, and the TLS-ALPN-01 challenge connects to whichever process the kernel picks — most often not the one that asked, so renewals fail at random. Use
--tls-certand--tls-key(a CDN origin certificate, or one renewed elsewhere) with--reuse-port; dragoman warns at startup otherwise.
Linux spreads new connections evenly across the processes. macOS accepts --reuse-port but does not balance connections the same way, so measure on Linux.
Updating the binary
--force is what lets cargo install overwrite the binary already there.
Restarting picks up the new server; the hourly import job takes it on its next
run, since both units point at the same path.
Updating the database
The database file can be replaced while the service is running without restarting. dragoman opens a new read-only SQLite connection per request, so an atomic rename takes effect immediately for new requests while in-flight requests finish against the old file.
Copy the new file to a staging path on the same filesystem, then rename it over the live file:
If the source database uses WAL mode (has accompanying -wal/-shm files), produce a clean single-file snapshot first with VACUUM INTO before copying:
Hourly cache import (cron)
As it serves requests, dragoman writes newly fetched records into its cache
database. Run dragoman build --from cache periodically to fold them into the
main commonmeta.sqlite3: it copies the cached works, people, and organizations
into the main database and then empties the cache. This keeps later lookups fully
local and bounds the cache size. dragoman opens a fresh read-only connection per
request, so the job can write the live database with no restart — new requests
pick up the merged records immediately.
build is an import followed by an enrich pass over what it just imported,
which is what this job wants. The enrich derives pid_relations and
pid_annotations, and dragoman reads both: relations drive the references and
citations panels, and pid_annotations supplies the navigable container id —
ISSN, DOI, OpenAlex source id or DataCite client id — without which a container
is dropped from search results. A plain import --from cache folds the records
in but leaves them with no relation graph and no validated commonmeta overlay
until something else enriches them. --from cache enriches only the records it
imported, not the whole store; --full forces a whole-store rescan.
No extra binary is needed: dragoman embeds the commonmeta CLI and forwards
build to it verbatim, so the dragoman already installed at
/usr/local/bin/dragoman runs the job. It must run as a user that can
write both the main database and the cache — typically root, since
commonmeta.sqlite3 is root-owned.
Set COMMONMETA_DB and DRAGOMAN_CACHE_DB in /etc/default/commonmeta
before enabling the job. build takes the destination from COMMONMETA_DB
only; left unset it falls back to the per-user data directory
(/root/.local/share/commonmeta/commonmeta.sqlite3 under root), so the job
would succeed while quietly filling a database nothing serves.
# /etc/cron.d/dragoman-cache-import
SHELL=/bin/sh
# Fold dragoman's freshly-cached records into the main database every hour.
# `flock -n` skips a run while the previous one is still going, so a slow run
# never overlaps the next. This matters more for `build` than for a bare
# `import`: the enrich pass writes too, and an interrupted `build --from cache`
# is what left a 166 GB WAL on the production store, which then served
# `database is locked` for plain SELECTs.
# `export` matters: the env file assigns without exporting, so sourcing it alone
# leaves COMMONMETA_DB a shell variable the job would never see.
0 * * * * root . /etc/default/commonmeta && export COMMONMETA_DB && \
flock -n /run/dragoman-import.lock \
/usr/local/bin/dragoman build --from cache \
--cache-db "$DRAGOMAN_CACHE_DB" \
>> /var/log/dragoman-import.log 2>&1
The paths come from /etc/default/commonmeta — the same env file dragoman uses.
build reads the cache path from CACHE_DB, but the env file defines
DRAGOMAN_CACHE_DB, so the job passes it explicitly with --cache-db and the
file works unmodified.
On a systemd host a timer is the more idiomatic alternative: it reuses the same
EnvironmentFile, logs to the journal (journalctl -u dragoman-import),
serialises runs on its own, and catches up a missed run with Persistent=true.
# /etc/systemd/system/dragoman-import.service
[Unit]
Description=Import and enrich dragoman's cached records into the main commonmeta database
RequiresMountsFor=/var/lib/commonmeta
[Service]
Type=oneshot
EnvironmentFile=/etc/default/commonmeta
ExecStart=/usr/local/bin/dragoman build --from cache --cache-db ${DRAGOMAN_CACHE_DB}
# /etc/systemd/system/dragoman-import.timer
[Unit]
Description=Hourly dragoman cache import
[Timer]
OnCalendar=hourly
Persistent=true
[Install]
WantedBy=timers.target
Both unit files are bundled in the repository (dragoman-import.service and
dragoman-import.timer). Copy them into /etc/systemd/system/ and enable the
timer:
These units were named commonmeta-import.* before, and the cron file
commonmeta-cache-import. On a host that still has the old ones, remove them
before enabling the new ones — two hourly importers writing the same database is
the overlap flock exists to prevent, and it can balloon the WAL:
Reverse proxy
When running behind a reverse proxy, do not set --domain. Let the proxy handle TLS termination; dragoman listens on plain HTTP at 127.0.0.1:3456 (or whatever --port you set). The proxy must set X-Real-IP (or append to X-Forwarded-For) so that request limits count each client rather than the proxy; connection caps and header timeouts are then the proxy's to set.
Caching
The pid routes tell browsers and CDNs how long to keep their answers:
| Answer | Cache-Control |
|---|---|
a record in a format (200) |
public, max-age=3600, s-maxage=86400, stale-while-revalidate=86400: an hour in a browser, a day in a CDN |
| an entity page | no-cache: kept, but revalidated on every use |
a 302 or 307, e.g. a DOI to its landing page |
public, max-age=0, s-maxage=3600: an hour in a CDN, revalidated by a browser every time |
404, 406 |
public, max-age=300 |
5xx, e.g. Crossref or DataCite failing |
no-store |
429, 503 from the request limits |
no-store |
Every 200 carries a weak ETag of its body, and a request whose If-None-Match names it gets 304 Not Modified with no body. An entity page still renders on the server; the 304 saves sending it.
Every answer from /{pid} and /api/handles/{pid} — pages, formats, the 302 to a landing page, the 404 — carries Vary: Accept, since each depends on the Accept header. That is the safety net. Many CDNs either ignore Vary or split their cache on every distinct Accept header, so prefer URLs that need no Accept header: /{pid}?format=bibtex, /dois/{doi} and /works/{doi} are cached by URL alone and carry no Vary: Accept. A format chosen by the Accept header names its own such URL in Content-Location, e.g. Content-Location: /10.1038/171737a0?format=bibtex.
Request limits
dragoman limits what one client can ask of it, so it can face the internet without a proxy in front. A client is its IP address, an IPv6 client its /64. The peer address is used as is, unless the peer is a trusted proxy — this machine, or a network in --trusted-proxies — which names the client:
- in
--client-ip-header, if set (a CDN sets its own, overwriting the client's); - in
X-Real-IP, from this machine only (a CDN passes on whatever the client sent); - in
X-Forwarded-For, read from the right past the trusted proxies that appended themselves.
A trusted edge that names no client counts as one; a local request that names none is not limited.
Each client has a token bucket per tier. RATE/BURST means a client may send BURST requests at once and RATE per second after that; a request over the limit gets 429 Too Many Requests with Retry-After.
| Tier | Requests | Default |
|---|---|---|
| — | /, /assets/*, /og-image.png, /commonmeta_v1.0.json, /api/status, /api/metrics |
not limited |
| Pages | entity pages and redirects (/{pid}, /orcid/*, /ror/*), and ?local_only=true lookups — the records the UI shows on a page |
20/60 |
| Lists | /api/person/*/works, /api/org/*/works, /api/container/works, /api/facets, /api/count, /api/random |
5/20 |
| Expensive | /api/search, /api/handles/*, /api/export, /api/bibliography, /dois/*, /works/*, and a pid asked for in a format (?format= or an Accept header other than HTML) |
2/20 |
Across all clients, at most --max-in-flight requests are served at once and each response has --request-timeout seconds; past either, dragoman answers 503 with Retry-After.
With --domain set to a public hostname, the TLS listener also caps each client at --max-conns-per-ip open connections and gives it --header-timeout seconds to finish the TLS handshake, send its first request, and send each request's headers — so a client that connects and stays silent cannot hold a connection open.
Refusals are counted in the metrics: http_rate_limited_total{tier}, http_overloaded_total, http_timed_out_total and http_connections_refused_total.
Releasing prebuilt binaries
cargo binstall dragoman downloads target-named tarballs from Codeberg Releases. The lookup URL is defined by [package.metadata.binstall] in Cargo.toml; each release tagged v<version> must carry these assets:
| Target | Asset |
|---|---|
macOS arm64 (aarch64-apple-darwin) |
dragoman-aarch64-apple-darwin.tar.gz |
Linux amd64 (x86_64-unknown-linux-gnu) |
dragoman-x86_64-unknown-linux-gnu.tar.gz |
Each tarball holds the bare dragoman binary at its root. Build and package both with:
# macOS arm64 builds natively; Linux amd64 cross-compiles via cargo-zigbuild
&&
The tarballs and a SHA256SUMS file land in dist/. Create the v<version> release on Codeberg and upload both tarballs as assets.
Telemetry
Logs — VictoriaLogs
Set --viktorialogs-url (or DRAGOMAN_VIKTORIALOGS_URL) to the VictoriaLogs OTLP/HTTP endpoint. All tracing events are shipped in batches; the exporter is flushed on graceful shutdown.
Without the flag, logs are written to stdout only (the default).
Metrics — VictoriaMetrics / Grafana
Set --metrics-allow (or DRAGOMAN_METRICS_ALLOW) to a comma-separated list of IP addresses permitted to scrape the /api/metrics endpoint. The route is only registered when this flag is set; all other callers receive 403 Forbidden.
# curl http://localhost:3456/api/metrics # works from 127.0.0.1
The /api/metrics endpoint is served on the same port as the main application. The client IP is the peer address; only when the peer is on this machine — a reverse proxy such as nginx, Caddy or Traefik — is it taken from X-Real-IP, then the last address in X-Forwarded-For, the one the proxy appended.
Metrics recorded:
| Metric | Labels | Description |
|---|---|---|
http_requests_total |
method, route, status |
Request count by route and HTTP status code |
http_request_duration_seconds |
method, route |
Request latency histogram |
Routes are normalised to templates (/dois/:doi, /works/:doi, /orcid/:id, /{pid}) to keep cardinality low.
Benchmarking
scripts/bench.sh samples DOIs from the store's pid_records and runs oha against a running dragoman at concurrency 1 to 64, so a release can be compared with the one before. Run it on the server, where requests from the machine itself are not rate limited:
Each level's full oha report is kept as JSON under bench/, and bench/results.tsv gains one line per level: requests per second, p50 and p99 latency, success rate and status codes. -k picks the URLs: redirect (/{doi}, the default), handles, page or bibtex. -h lists the other options.
Keyboard shortcuts
| Shortcut | Action |
|---|---|
⌘K / Ctrl-K |
Focus search bar; press again to cycle the entity type |
⌘Y / Ctrl-Y |
Toggle History & Bookmarks panel (History tab) |
⌘B / Ctrl-B |
Toggle History & Bookmarks panel (Bookmarks tab) |
⌥↑ / ⌥↓, Alt-↑ / Alt-↓ |
Move the selected bookmark up or down |
Escape |
Close panel or cancel search |
↑ / ↓ |
Navigate search suggestions |
Enter |
Navigate to selected search result |
License
MIT