grobid-rs
A native Rust client for the GROBID REST API, with a strongly typed parser for GROBID TEI documents.
GROBID is a machine learning system for extracting, parsing and re-structuring raw documents such as PDFs into structured TEI-XML. This crate provides:
- an asynchronous HTTP client (
GrobidClient, built onreqwest) for the GROBID web services, with automatic retries with exponential backoff when the server is busy (HTTP 503/429), - a TEI parser that turns GROBID responses into strongly typed Rust structures, exposing the full document hierarchy (sections, paragraphs, figures, formulas, footnotes, lists), bibliographic metadata, citations, references and optional PDF coordinates,
- optional second-tier reference completion against OpenAlex (the
openalexfeature), filling in missing fields of parsed references by DOI or title lookup.
The client behaviour follows the official
Python client; the TEI
parser is modelled after the
Go client, which is itself
modelled after the grobid-tei-xml Python package.
Usage
use ;
// Inside a `#[tokio::main]` function:
async
PDFs can also be passed as bytes, avoiding a round-trip through the filesystem:
use ;
async
Services
| Method | GROBID service | Returns |
|---|---|---|
process_fulltext_document |
/api/processFulltextDocument |
Document |
process_header_document |
/api/processHeaderDocument |
Document |
process_references |
/api/processReferences |
Vec<Citation> |
process_citation |
/api/processCitation |
Option<Citation> |
process_citation_list |
/api/processCitationList |
Vec<Citation> |
process_pdf_raw |
any multipart service | raw TEI/response body |
ping |
/api/isalive |
bool |
version |
/api/version |
String |
Options
ProcessOptions controls the request parameters; its defaults follow the
GROBID server defaults and the official Python client (header consolidation
on, citation consolidation off):
use ;
async
Parsing TEI without a server
The parser works on TEI XML obtained elsewhere, e.g. written to disk by a batch process:
use parse_document;
The parser is namespace-agnostic: it accepts both full TEI documents in the
TEI namespace and bare <biblStruct> fragments as returned by
/api/processCitation.
Serialization
All parsed types implement serde::Serialize/Deserialize, so a document
can be converted to JSON (or any other serde format) with e.g. serde_json:
Reference completion against OpenAlex (optional)
With the openalex feature, parsed references can be completed against
OpenAlex: a reference is looked up by DOI when one
was parsed, otherwise by title, and missing fields — authors, journal, volume,
pages, DOI, PubMed ID, ... — are filled in from the matching work. Existing
requirement): a reference is looked up by DOI when one was parsed, otherwise by
title, and missing fields are filled in from the matching work. Existing
fields are never overwritten, and a search result is accepted only when
title, publication year and first author are compatible with the parsed
reference. Lookups require HTTPS access to api.openalex.org; they use the
async reqwest client this crate already depends on, so no external OpenAlex
client is required. Responses are parsed into a small, deliberately tolerant
in-house model: unknown, missing and null fields are ignored, so unrelated
OpenAlex schema changes do not break completion. Titles are sanitized for
OpenAlex's filter syntax before they are sent as search terms, so punctuation
such as commas, pipes or wildcards does not break the lookup either. The
matching heuristics are pinned by an extensive test suite, with known
weaknesses documented as ignored tests. Completer::with_timeout() (or
Completer::builder()) bounds each lookup, so an unreachable host cannot
stall a batch.
Enable the feature in Cargo.toml:
= { = "0.2", = ["openalex"] }
use Completer;
use Biblio;
async
Coordinates
When tei_coordinates is requested, GROBID adds a @coords attribute to the
selected structures and a <facsimile> section with the page dimensions.
Both are parsed:
Coords/BoundingBoxon sections, paragraphs, sentences, figures, formulas, notes, citations, authors, ...document.facsimile: Vec<Surface>with the width/height of every page.
The coordinate system is the native PDF one: origin at the upper-left corner of a page, the y-axis extending downward, values in abstract PDF units, and the first page numbered 1. See the GROBID documentation for details.
Examples
pdf2bibtex
The pdf2bibtex example processes all PDFs in a directory and writes the
extracted bibliographic metadata as a BibTeX file, one entry per document
(its header):
With -r/--rename, each PDF is renamed after its metadata has been
extracted to Author_Year_<first 10 title words>.pdf, e.g.
Kahle_2000_The_Barc_model_for_continuous_variables.pdf. Parts GROBID could
not extract (author, year or title) are dropped, non-ASCII characters are
removed, and colliding names get a -2, -3, ... suffix. The same policy
is available to library users as bibtex::suggest_file_name() and
bibtex::unique_path(); bibtex::FileStemStyle::Full selects
Author1, Author2, ... - Year - Full title with punctuation stripped and no
title truncation, and bibtex::unique_path_with_year() resolves collisions
by numbering the year (... - 2020-1 - Title).
With -l/--link, each entry records the path of its PDF in a file
field, so reference managers can open the document; combined with
-r/--rename, the field points at the renamed file. Paths are recorded
as passed on the command line, so relative input paths stay relative.
With --openalex (requires building the example with --features openalex),
each extracted header is completed against OpenAlex before entries are
written and PDFs are renamed: missing authors, journal, volume, pages, DOI,
... are filled in from the matching work. See Reference completion against
OpenAlex above.
refs2bibtex
The refs2bibtex example extracts the bibliographic references of all PDFs
in a directory (via /api/processReferences) and writes one BibTeX entry per
reference:
Both examples discover PDFs recursively and process them with a bounded
number of concurrent requests (-w, default 4); per-document failures are
reported on stderr and skipped, and an unresponsive server is detected by a
liveness probe with bounded retries and a clear error message. Entry types
(@article, @incollection, @techreport, @book, @misc) and keys
(first author surname + year, deduplicated) are derived from the parsed
metadata via the grobid::bibtex helpers, which render records as typed
BibLaTeX entries using the biblatex
crate (proper escaping, typed person lists and dates). refs2bibtex skips
empty parse results and drops references with a duplicate DOI, and supports
reference consolidation against CrossRef with -c/--consolidate. Both examples
accept --openalex when built with --features openalex: refs2bibtex
completes the collected references, pdf2bibtex each document header, before
the entries are written (see Reference completion against OpenAlex above):
Run either example with --help for all options.
The GROBID server
This crate is a client only: all PDF parsing happens in a running
GROBID server, which is reached over its
REST API. GROBID is a machine learning system that turns PDFs of scholarly
articles into structured TEI XML — header metadata, body structure,
references and optional PDF coordinates. The server preloads its models at
startup and listens on port 8070 by default.
The quickest way to get a server running is Docker, which is also the only
supported option on Windows (GROBID itself is developed and tested on Linux
and macOS). On arm64 hosts the images run emulated and are considerably
slower:
| Platform | Installation instructions |
|---|---|
| Docker (any platform, recommended) | Run with Docker |
| Linux | Build from source (JDK 21, then ./gradlew run) |
| macOS | Build from source (includes Homebrew JDK instructions) |
| Windows (native) | Not supported by GROBID; use Docker or WSL instead, see Windows related issues |
To start a server with Docker — the -crf image is small (about 500 MB)
and fast, the -full image (about 8 GB, GPU recommended) adds Deep Learning
models that improve reference parsing:
# CRF models only
# with Deep Learning models (add --gpus all on Linux to use a GPU)
Once the container is up, http://localhost:8070/api/isalive returns true
and the examples above work as shown. See the GROBID documentation for the
quick start, the
REST API reference
and the
troubleshooting guide.
Requirements
- Rust 1.85+
- A running GROBID server, see The GROBID server above
License
MIT, see LICENSE.