Skip to main content

Crate grobid_bibtex

Crate grobid_bibtex 

Source
Expand description

A reference management system for scientific writing.

grobid-bibtex combines GROBID for reference extraction and OpenAlex for reference completion with a data model organized around BibTeX/BibLaTeX. It is the foundation for user-facing tools such as editor plugins (citation keys, duplicate detection, pickers) and LLM- or RAG-based writing assistants (structured, deduplicated bibliography data).

The modules follow the pipeline of that workflow:

  • extract: turn PDFs into Biblio records with a GROBID server.
  • complete (feature openalex): fill gaps in extracted records from OpenAlex.
  • collection: parse, index and append .bib files; assign citation keys; detect and merge duplicates.
  • bibtex: render individual records as BibLaTeX and derive citation keys and file names.
  • files: work with the PDFs behind the records (discovery, renaming).
  • cite: resolve @key mentions in a query to documents in a corpus (cite_keys, cite::CiteState, cite::load_documents).
  • prepass (feature pipeline): the orchestration around those modules — fingerprint, extract, complete, rename and merge a corpus, with progress and cancellation hooks.

The pdf2bibtex and refs2bibtex binaries are front ends over these modules.

§Example

use std::path::PathBuf;

use grobid_bibtex::collection::Collection;
use grobid_bibtex::{Author, Biblio, FileLink};

let biblio = Biblio {
    authors: vec![Author {
        surname: Some("Kahle".to_string()),
        ..Author::default()
    }],
    date: Some("2000".to_string()),
    ..Biblio::default()
};
let collection = Collection::new();
let entries = collection.format_all(
    &[(PathBuf::from("paper.pdf"), biblio)],
    &FileLink::Absolute,
);
assert!(entries[0].1.starts_with("@misc{Kahle2000,"));

The full README is included below.

§grobid-bibtex

A reference management system for scientific writing, built on GROBID for reference extraction and OpenAlex for reference completion, with a data model organized around BibTeX/BibLaTeX.

The crate is the foundation for user-facing tools such as editor plugins (citation keys, duplicate detection, pickers) and LLM- or RAG-based writing assistants (structured, deduplicated bibliography data). The bundled pdf2bibtex and refs2bibtex binaries are front ends over the same API.

§Organization

ModuleRole
extractPDFs into records with GROBID: batch headers() and references(), plus collect_references() for flattening, DOI-deduplicating and ordering extracted reference lists.
completeSecond-tier completion of extracted records against OpenAlex (feature openalex).
collectionCollection parses, indexes and appends .bib files, assigns citation keys and merges records with duplicate detection (content, identifiers, PDF file name); merge_file() merges records into a file in one step, load_or_new() handles files that do not exist yet.
bibtexRendering and naming for individual records: format_entry(), format_all(), suggest_key(), unique_key() and the Author_Year_Title file-name policies.
filesThe PDFs behind the records: recursive discovery (collect_pdfs() and the generic collect_files()), the rename policy (rename_pdfs_with() with any file-stem style and collision style) and a Manifest that skips unchanged files on later runs and can cache each processed file’s extracted record.
cite@key mention scanning (cite_keys()), resolution against a bibliography and the active corpus directories (CiteState, Located), and load_documents() with an injected loader — the citation-attach pipeline of a chat host.
prepass (feature pipeline)The full corpus pre-pass: fingerprint manifest, GROBID extraction, OpenAlex completion, metadata renaming and BibTeX merge, with progress/cancellation hooks (PrepassObserver) and a PrepassReport.

The pipeline is: discover PDFs (files) → extract records (extract) → complete them (complete) → render and merge them into a collection (collection, bibtex).

§Citation attach and corpus maintenance

The cite module turns a query’s @Key mentions into documents: it scans the text, resolves each key against a .bib file (exact first, then ASCII case-insensitive) and the active corpus directories, and loads the located documents through a caller-supplied parser. Failures — an unknown key, a missing or ambiguous document, a path outside the corpora — are returned as values, never as errors.

With the pipeline feature, prepass orchestrates a whole corpus: it keeps a fingerprint manifest (.ragrig_grobid.json), parses new or changed PDFs with GROBID, completes them against OpenAlex, renames them to Key - Full authors - Full title - Year (shortened to the filesystem’s 255-byte file-name limit) and merges the records into a .bib file. Unchanged files are merged from the manifest cache without contacting the server. Hosts that need a UI pass a PrepassObserver for progress and cancellation (prepass_with); the manifest name and rename policy are published as constants so every host shares one format.

§Collection

use std::path::PathBuf;

use grobid_bibtex::collection::Collection;
use grobid_bibtex::{Author, Biblio};

let biblio = Biblio {
    authors: vec![Author {
        surname: Some("Kahle".to_string()),
        ..Author::default()
    }],
    date: Some("2000".to_string()),
    ..Biblio::default()
};
let mut collection = Collection::new();
let records = [(PathBuf::from("paper.pdf"), biblio)];
let link = grobid_bibtex::FileLink::Absolute;
let (entries, duplicates) = collection.merge_all(&records, &link).unwrap();
assert_eq!(duplicates, 0);
assert!(entries[0].1.starts_with("@misc{Kahle2000,"));
assert!(entries[0].1.contains("file = {paper.pdf},"));

// Merging the same records again detects the duplicate.
let (entries, duplicates) = collection.merge_all(&records, &link).unwrap();
assert_eq!(duplicates, 1);
assert!(entries.is_empty());

To keep a .bib file in sync with a corpus, collection::merge_file() combines the steps: it reads the file (a missing file starts empty), skips records that are already present, appends the new entries and reports what happened. Existing entries are never modified or removed.

use std::path::PathBuf;

use grobid_bibtex::collection;
use grobid_bibtex::{Author, Biblio};

let biblio = Biblio {
    authors: vec![Author {
        surname: Some("Kahle".to_string()),
        ..Author::default()
    }],
    date: Some("2000".to_string()),
    ..Biblio::default()
};
let path = dir.join("refs.bib");
let link = grobid_bibtex::FileLink::Absolute;
let report = collection::merge_file(&path, &[(PathBuf::from("paper.pdf"), biblio)], &link)
    .unwrap();
assert_eq!(report.entries.len(), 1);
assert_eq!(report.total, 1);

Beyond Collection, the bibtex module provides the single-record helpers: entry_type(), year(), suggest_key() and unique_key() for entry types and collision-free citation keys, format_entry(), format_entry_with_file() and format_entry_with_link() for typed BibLaTeX output, format_all() for batches, and suggest_file_name(), suggest_file_stem_with(), unique_path() and unique_path_with_year() for the file-naming policies (FileStemStyle::Full and FileStemStyle::Keyed).

bibtex::FileLink decides what the file field says: None writes no field, Absolute records the path as given, and Relative(base) records a path relative to the workspace (or any other base) when one exists. A document on another filesystem root — another drive on Windows — stays absolute.

§Binaries

Both binaries process all PDFs in a directory recursively through a running GROBID server and write their bibliographic metadata as BibTeX. They share the common options below; run either with --help for all of them.

§pdf2bibtex

Writes one entry per document (its header):

cargo run --release --bin pdf2bibtex -- ~/papers -s http://localhost:8070

With -r/--rename, each PDF is renamed after its metadata has been extracted to Author_Year_<first 10 title words>.pdf, e.g. Kahle_2000_The_Barc_model_for_continuous_variables.pdf. Parts GROBID could not extract (author, year or title) are dropped, non-ASCII characters are removed, and colliding names get a -2, -3, … suffix. Names longer than the filesystem’s 255-byte limit are shortened at a character boundary.

With -l/--link, each entry records the path of its PDF in a file field, so reference managers can open the document; combined with -r/--rename, the field points at the renamed file. Paths are recorded as passed on the command line, so relative input paths stay relative. Library users that want portable links pass bibtex::FileLink::Relative(base) to format_all()/merge_file(); that also makes CiteState resolve the bibliography when the project moves.

With -a/--append or -m/--merge, the entries go into an existing BibTeX file instead of a new one. Both load the target collection first and reserve its citation keys, so a new entry whose suggested key already exists gets a -2, -3, … suffix. --append adds every extracted entry; --merge skips records that are already there, where a record counts as present when its normalized field content, one of its identifiers (DOI, PMID, arXiv) or its PDF file name matches an existing entry. --output, --append and --merge are mutually exclusive.

With --openalex (requires building with --features openalex), each extracted header is completed against OpenAlex before entries are written and PDFs are renamed: missing authors, journal, volume, pages, DOI, … are filled in from the matching work. See Reference completion against OpenAlex in the grobid documentation.

§refs2bibtex

Extracts the bibliographic references of all PDFs in a directory (via /api/processReferences) and writes one BibTeX entry per reference:

cargo run --release --bin refs2bibtex -- ~/papers -s http://localhost:8070

Both binaries discover PDFs recursively and process them with a bounded number of concurrent requests (-w, default 4); per-document failures are reported on stderr and skipped, and an unresponsive server is detected by a liveness probe with bounded retries and a clear error message. Entry types (@article, @incollection, @techreport, @book, @misc) and keys (first author surname + year, deduplicated) are derived from the parsed metadata via the bibtex helpers. refs2bibtex skips empty parse results and drops references with a duplicate DOI, and supports reference consolidation against CrossRef with -c/--consolidate. Both binaries accept --openalex when built with --features openalex: refs2bibtex completes the collected references, pdf2bibtex each document header, before the entries are written:

cargo run --release --features openalex --bin refs2bibtex -- ~/papers --openalex
cargo run --release --features openalex --bin pdf2bibtex -- ~/papers --openalex

§Features

  • openalex: enables the OpenAlex completion tier of the grobid client, the complete module and the --openalex option of both binaries.

Re-exports§

pub use collection::Collection;

Modules§

bibtex
BibTeX rendering for parsed bibliographic records.
cite
Chat-side citation resolution: @key mentions become attached documents.
collection
The BibTeX-centered reference collection.
complete
Second-tier completion of extracted records against OpenAlex.
extract
Extraction of bibliographic records from PDFs with GROBID.
files
Working with the PDF files behind the records.
openalex
Second-tier reference completion against OpenAlex.
prepass
Reusable GROBID pre-pass orchestration.
tei
Parsing of GROBID TEI-XML into strongly typed structures.

Structs§

Author
A person (author or editor), parsed from a TEI <persName> element.
Biblio
A complete bibliographic record.
Citation
A single bibliographic reference (one <biblStruct> element), e.g. from the reference list of a processed document.
GrobidClient
A client for the GROBID REST API.
GrobidClientBuilder
Builder for GrobidClient.
ProcessOptions
Options for a GROBID processing request.
RetryPolicy
Retry policy for requests that hit a busy GROBID server (HTTP 503 or 429). Retries are spaced with exponential backoff: the delay after the n-th attempt is initial_delay * multiplier^n, capped at max_delay.

Enums§

CitationConsolidation
Consolidation level for citations/references.
Error
Errors produced by the grobid client and TEI parser.
HeaderConsolidation
Consolidation level for the document header.
PdfInput
A PDF to be processed by GROBID: either a path on disk, or the document bytes with a file name. In-memory inputs avoid a round-trip through the filesystem, e.g. when the PDF was downloaded or comes from an object store.

Constants§

DEFAULT_GROBID_URL
The default GROBID server URL (a locally running instance).