Skip to main content

Crate grobid

Crate grobid 

Source
Expand description

§grobid

A native Rust client for the GROBID REST API, with a strongly typed parser for GROBID TEI documents.

GROBID is a machine learning library for extracting, parsing and re-structuring raw documents such as PDFs into structured TEI-XML. This crate provides:

  • an asynchronous HTTP client (GrobidClient) for the GROBID web services, with busy-server retries,
  • a TEI parser that turns GROBID responses into strongly typed Rust structures (Document, Biblio, Citation, …), exposing the full document hierarchy, bibliographic metadata, references and optional PDF coordinates,
  • optional second-tier completion of parsed references against OpenAlex (openalex feature).

BibTeX/BibLaTeX rendering and bibliography tooling moved to the companion crate grobid-bibtex, together with the pdf2bibtex and refs2bibtex command-line tools.

§Example

use grobid::{GrobidClient, ProcessOptions};

let client = GrobidClient::new("http://localhost:8070")?;
assert!(client.ping().await?);

// Process a PDF and inspect the parsed TEI.
let document = client
    .process_fulltext_document("paper.pdf", &ProcessOptions::default())
    .await?;
println!("title: {:?}", document.header.title);
println!("authors: {}", document.header.authors.len());
println!("citations: {}", document.citations.len());

for section in &document.body {
    println!("## {}", section.head.as_deref().unwrap_or("(untitled)"));
    for paragraph in section.paragraphs() {
        println!("{}", paragraph.text);
    }
}

§Parsing TEI without a server

The parser works on TEI XML obtained elsewhere, e.g. written to disk by a batch process:

use grobid::tei::parse_document;

let xml = r#"<TEI xmlns="http://www.tei-c.org/ns/1.0">
    <teiHeader>
        <encodingDesc><appInfo>
            <application version="0.8.1" when="2024-01-01T00:00+0000"/>
        </appInfo></encodingDesc>
        <fileDesc>
            <titleStmt><title level="a" type="main">Example</title></titleStmt>
            <publicationStmt><publisher/></publicationStmt>
            <sourceDesc><biblStruct/></sourceDesc>
        </fileDesc>
    </teiHeader>
    <text/>
</TEI>"#;

let document = parse_document(xml)?;
assert_eq!(document.header.title.as_deref(), Some("Example"));

The full README is included below.

§grobid-rs

A native Rust client for the GROBID REST API, with a strongly typed parser for GROBID TEI documents.

GROBID is a machine learning system for extracting, parsing and re-structuring raw documents such as PDFs into structured TEI-XML. This crate provides:

  • an asynchronous HTTP client (GrobidClient, built on reqwest) for the GROBID web services, with automatic retries with exponential backoff when the server is busy (HTTP 503/429),
  • a TEI parser that turns GROBID responses into strongly typed Rust structures, exposing the full document hierarchy (sections, paragraphs, figures, formulas, footnotes, lists), bibliographic metadata, citations, references and optional PDF coordinates,
  • optional second-tier reference completion against OpenAlex (the openalex feature), filling in missing fields of parsed references by DOI or title lookup.

The client behaviour follows the official Python client; the TEI parser is modelled after the Go client, which is itself modelled after the grobid-tei-xml Python package.

§Usage

use grobid::{GrobidClient, ProcessOptions};

// Inside a `#[tokio::main]` function:
async fn example() -> Result<(), grobid::Error> {
    let client = GrobidClient::new("http://localhost:8070")?;
    assert!(client.ping().await?);

    // Process a PDF and inspect the parsed TEI.
    let document = client
        .process_fulltext_document("paper.pdf", &ProcessOptions::default())
        .await?;
    println!("title: {:?}", document.header.title);
    println!("authors: {}", document.header.authors.len());
    println!("citations: {}", document.citations.len());

    for section in &document.body {
        println!("## {}", section.head.as_deref().unwrap_or("(untitled)"));
        for paragraph in section.paragraphs() {
            println!("{}", paragraph.text);
        }
    }
    Ok(())
}

PDFs can also be passed as bytes, avoiding a round-trip through the filesystem:

use grobid::{GrobidClient, PdfInput, ProcessOptions};

async fn example(downloaded_bytes: Vec<u8>) -> Result<(), grobid::Error> {
    let client = GrobidClient::new("http://localhost:8070")?;
    let pdf = PdfInput::Data {
        filename: "paper.pdf".to_string(),
        data: downloaded_bytes,
    };
    let document = client
        .process_fulltext_document(pdf, &ProcessOptions::default())
        .await?;
    Ok(())
}

§Services

MethodGROBID serviceReturns
process_fulltext_document/api/processFulltextDocumentDocument
process_header_document/api/processHeaderDocumentDocument
process_references/api/processReferencesVec<Citation>
process_citation/api/processCitationOption<Citation>
process_citation_list/api/processCitationListVec<Citation>
process_pdf_rawany multipart serviceraw TEI/response body
ping/api/isalivebool
version/api/versionString

§Options

ProcessOptions controls the request parameters; its defaults follow the GROBID server defaults and the official Python client (header consolidation on, citation consolidation off):

use grobid::{CoordinateElement, GrobidClient, ProcessOptions};

async fn example() -> Result<(), grobid::Error> {
    let client = GrobidClient::new("http://localhost:8070")?;
    let options = ProcessOptions {
        generate_ids: true,
        include_raw_citations: true,
        tei_coordinates: CoordinateElement::COMMON.to_vec(),
        segment_sentences: true,
        ..Default::default()
    };
    let document = client
        .process_fulltext_document("paper.pdf", &options)
        .await?;
    // Coordinates of figures, references, ... in the original PDF:
    for figure in document.body[0].figures() {
        println!("{:?} at {:?}", figure.head, figure.coords);
    }
    Ok(())
}

§Parsing TEI without a server

The parser works on TEI XML obtained elsewhere, e.g. written to disk by a batch process:

use grobid::tei::parse_document;

fn example() -> Result<(), Box<dyn std::error::Error>> {
    let xml = std::fs::read_to_string("output.tei.xml")?;
    let document = parse_document(&xml)?;
    println!("{:?}", document.header.title);
    Ok(())
}

The parser is namespace-agnostic: it accepts both full TEI documents in the TEI namespace and bare <biblStruct> fragments as returned by /api/processCitation.

§Serialization

All parsed types implement serde::Serialize/Deserialize, so a document can be converted to JSON (or any other serde format) with e.g. serde_json:

fn example(xml: &str) -> Result<(), Box<dyn std::error::Error>> {
    let document = grobid::tei::parse_document(xml)?;
    let json = serde_json::to_string_pretty(&document)?;
    println!("{json}");
    Ok(())
}

§Reference completion against OpenAlex (optional)

With the openalex feature, parsed references can be completed against OpenAlex: a reference is looked up by DOI when one was parsed, otherwise by title, and missing fields — authors, journal, volume, pages, DOI, PubMed ID, … — are filled in from the matching work. Existing fields are never overwritten, and a search result is accepted only when title, publication year and first author are compatible with the parsed reference. Lookups require HTTPS access to api.openalex.org; they use the async reqwest client this crate already depends on, so no external OpenAlex client is required. Responses are parsed into a small, deliberately tolerant in-house model: unknown, missing and null fields are ignored, so unrelated OpenAlex schema changes do not break completion. Titles are sanitized for OpenAlex’s filter syntax before they are sent as search terms, so punctuation such as commas, pipes or wildcards does not break the lookup either. The matching heuristics are pinned by an extensive test suite, with known weaknesses documented as ignored tests. Completer::with_timeout() (or Completer::builder()) bounds each lookup, so an unreachable host cannot stall a batch.

Enable the feature in Cargo.toml:

grobid = { version = "0.6", features = ["openalex"] }
ⓘ
use grobid::openalex::Completer;
use grobid::Biblio;

async fn example(biblio: Biblio) -> Result<(), Box<dyn std::error::Error>> {
    let completer = Completer::new();
    if let Some(completion) = completer.complete(&biblio).await? {
        let completed: Biblio = completion.biblio;
        println!("matched {} by {:?}", completion.openalex_id, completion.matched_by);
    }
    Ok(())
}

§Coordinates

When tei_coordinates is requested, GROBID adds a @coords attribute to the selected structures and a <facsimile> section with the page dimensions. Both are parsed:

  • Coords/BoundingBox on sections, paragraphs, sentences, figures, formulas, notes, citations, authors, …
  • document.facsimile: Vec<Surface> with the width/height of every page.

The coordinate system is the native PDF one: origin at the upper-left corner of a page, the y-axis extending downward, values in abstract PDF units, and the first page numbered 1. See the GROBID documentation for details.

§Examples and command-line tools

The BibTeX examples moved to the companion crate grobid-bibtex: its pdf2bibtex binary processes the headers of all PDFs in a directory and its refs2bibtex binary extracts their references, both writing BibTeX/BibLaTeX files. That crate also hosts the BibTeX helpers (bibtex::format_entry, bibtex::suggest_key, the file-naming policies and merging into existing bibliographies) that were formerly grobid::bibtex.

§The GROBID server

This crate is a client only: all PDF parsing happens in a running GROBID server, which is reached over its REST API. GROBID is a machine learning system that turns PDFs of scholarly articles into structured TEI XML — header metadata, body structure, references and optional PDF coordinates. The server preloads its models at startup and listens on port 8070 by default.

The quickest way to get a server running is Docker, which is also the only supported option on Windows (GROBID itself is developed and tested on Linux and macOS). On arm64 hosts the images run emulated and are considerably slower:

PlatformInstallation instructions
Docker (any platform, recommended)Run with Docker
LinuxBuild from source (JDK 21, then ./gradlew run)
macOSBuild from source (includes Homebrew JDK instructions)
Windows (native)Not supported by GROBID; use Docker or WSL instead, see Windows related issues

To start a server with Docker — the -crf image is small (about 500 MB) and fast, the -full image (about 8 GB, GPU recommended) adds Deep Learning models that improve reference parsing:

# CRF models only
docker run -p 8070:8070 grobid/grobid:0.9.1-crf

# with Deep Learning models (add --gpus all on Linux to use a GPU)
docker run -p 8070:8070 grobid/grobid:0.9.1-full

Once the container is up, http://localhost:8070/api/isalive returns true and the examples above work as shown. See the GROBID documentation for the quick start, the REST API reference and the troubleshooting guide.

§Requirements

  • Rust 1.85+
  • A running GROBID server, see The GROBID server above

§License

MIT, see LICENSE.

Re-exports§

pub use tei::parse_citation;
pub use tei::parse_citation_list;
pub use tei::parse_document;
pub use tei::Address;
pub use tei::Affiliation;
pub use tei::Author;
pub use tei::Biblio;
pub use tei::Block;
pub use tei::BoundingBox;
pub use tei::Citation;
pub use tei::Coords;
pub use tei::Div;
pub use tei::Document;
pub use tei::Figure;
pub use tei::Formula;
pub use tei::Header;
pub use tei::List;
pub use tei::MarkerRef;
pub use tei::Note;
pub use tei::Paragraph;
pub use tei::RefKind;
pub use tei::Sentence;
pub use tei::Surface;

Modules§

tei
Parsing of GROBID TEI-XML into strongly typed structures.

Structs§

GrobidClient
A client for the GROBID REST API.
GrobidClientBuilder
Builder for GrobidClient.
ProcessOptions
Options for a GROBID processing request.
RetryPolicy
Retry policy for requests that hit a busy GROBID server (HTTP 503 or 429). Retries are spaced with exponential backoff: the delay after the n-th attempt is initial_delay * multiplier^n, capped at max_delay.

Enums§

CitationConsolidation
Consolidation level for citations/references.
CoordinateElement
TEI structures for which PDF coordinates can be requested.
Error
Errors produced by the grobid client and TEI parser.
HeaderConsolidation
Consolidation level for the document header.
InvalidDocument
Reasons why a well-formed XML document is not a usable GROBID TEI document.
PdfInput
A PDF to be processed by GROBID: either a path on disk, or the document bytes with a file name. In-memory inputs avoid a round-trip through the filesystem, e.g. when the PDF was downloaded or comes from an object store.

Constants§

DEFAULT_GROBID_URL
The default GROBID server URL (a locally running instance).
SERVICE_CITATION
The citation parsing service (/api/processCitation).
SERVICE_CITATION_LIST
The citation list parsing service (/api/processCitationList).
SERVICE_FULLTEXT
The fulltext processing service (/api/processFulltextDocument).
SERVICE_HEADER
The header processing service (/api/processHeaderDocument).
SERVICE_REFERENCES
The reference extraction service (/api/processReferences).