Expand description
A reference management system for scientific writing.
grobid-bibtex combines GROBID for reference extraction and OpenAlex
for reference completion with a data model organized around
BibTeX/BibLaTeX. It is the foundation for user-facing tools such as
editor plugins (citation keys, duplicate detection, pickers) and LLM- or
RAG-based writing assistants (structured, deduplicated bibliography
data).
The modules follow the pipeline of that workflow:
extract: turn PDFs intoBibliorecords with a GROBID server.complete(featureopenalex): fill gaps in extracted records from OpenAlex.collection: parse, index and append.bibfiles; assign citation keys; detect and merge duplicates.bibtex: render individual records as BibLaTeX and derive citation keys and file names.files: work with the PDFs behind the records (discovery, renaming).cite: resolve@keymentions in a query to documents in a corpus (cite_keys,cite::CiteState,cite::load_documents).prepass(featurepipeline): the orchestration around those modules — fingerprint, extract, complete, rename and merge a corpus, with progress and cancellation hooks.
The pdf2bibtex and refs2bibtex binaries are front ends over these
modules.
§Example
use std::path::PathBuf;
use grobid_bibtex::collection::Collection;
use grobid_bibtex::{Author, Biblio, FileLink};
let biblio = Biblio {
authors: vec![Author {
surname: Some("Kahle".to_string()),
..Author::default()
}],
date: Some("2000".to_string()),
..Biblio::default()
};
let collection = Collection::new();
let entries = collection.format_all(
&[(PathBuf::from("paper.pdf"), biblio)],
&FileLink::Absolute,
);
assert!(entries[0].1.starts_with("@misc{Kahle2000,"));The full README is included below.
§grobid-bibtex
A reference management system for scientific writing, built on GROBID for reference extraction and OpenAlex for reference completion, with a data model organized around BibTeX/BibLaTeX.
The crate is the foundation for user-facing tools such as editor plugins
(citation keys, duplicate detection, pickers) and LLM- or RAG-based writing
assistants (structured, deduplicated bibliography data). The bundled
pdf2bibtex and refs2bibtex binaries are front ends over the same API.
§Organization
| Module | Role |
|---|---|
extract | PDFs into records with GROBID: batch headers() and references(), plus collect_references() for flattening, DOI-deduplicating and ordering extracted reference lists. |
complete | Second-tier completion of extracted records against OpenAlex (feature openalex). |
collection | Collection parses, indexes and appends .bib files, assigns citation keys and merges records with duplicate detection (content, identifiers, PDF file name); merge_file() merges records into a file in one step, load_or_new() handles files that do not exist yet. |
bibtex | Rendering and naming for individual records: format_entry(), format_all(), suggest_key(), unique_key() and the Author_Year_Title file-name policies. |
files | The PDFs behind the records: recursive discovery (collect_pdfs() and the generic collect_files()), the rename policy (rename_pdfs_with() with any file-stem style and collision style) and a Manifest that skips unchanged files on later runs and can cache each processed file’s extracted record. |
cite | @key mention scanning (cite_keys()), resolution against a bibliography and the active corpus directories (CiteState, Located), and load_documents() with an injected loader — the citation-attach pipeline of a chat host. |
prepass (feature pipeline) | The full corpus pre-pass: fingerprint manifest, GROBID extraction, OpenAlex completion, metadata renaming and BibTeX merge, with progress/cancellation hooks (PrepassObserver) and a PrepassReport. |
The pipeline is: discover PDFs (files) → extract records (extract) →
complete them (complete) → render and merge them into a collection
(collection, bibtex).
§Citation attach and corpus maintenance
The cite module turns a query’s @Key mentions into documents: it scans the
text, resolves each key against a .bib file (exact first, then ASCII
case-insensitive) and the active corpus directories, and loads the located
documents through a caller-supplied parser. Failures — an unknown key, a
missing or ambiguous document, a path outside the corpora — are returned as
values, never as errors.
With the pipeline feature, prepass orchestrates a whole corpus: it keeps a
fingerprint manifest (.ragrig_grobid.json), parses new or changed PDFs with
GROBID, completes them against OpenAlex, renames them to
Key - Full authors - Full title - Year (shortened to the filesystem’s
255-byte file-name limit) and merges the records into a .bib
file. Unchanged files are merged from the manifest cache without contacting
the server. Hosts that need a UI pass a PrepassObserver for progress and
cancellation (prepass_with); the manifest name and rename policy are
published as constants so every host shares one format.
§Collection
use std::path::PathBuf;
use grobid_bibtex::collection::Collection;
use grobid_bibtex::{Author, Biblio};
let biblio = Biblio {
authors: vec![Author {
surname: Some("Kahle".to_string()),
..Author::default()
}],
date: Some("2000".to_string()),
..Biblio::default()
};
let mut collection = Collection::new();
let records = [(PathBuf::from("paper.pdf"), biblio)];
let link = grobid_bibtex::FileLink::Absolute;
let (entries, duplicates) = collection.merge_all(&records, &link).unwrap();
assert_eq!(duplicates, 0);
assert!(entries[0].1.starts_with("@misc{Kahle2000,"));
assert!(entries[0].1.contains("file = {paper.pdf},"));
// Merging the same records again detects the duplicate.
let (entries, duplicates) = collection.merge_all(&records, &link).unwrap();
assert_eq!(duplicates, 1);
assert!(entries.is_empty());To keep a .bib file in sync with a corpus, collection::merge_file()
combines the steps: it reads the file (a missing file starts empty), skips
records that are already present, appends the new entries and reports what
happened. Existing entries are never modified or removed.
use std::path::PathBuf;
use grobid_bibtex::collection;
use grobid_bibtex::{Author, Biblio};
let biblio = Biblio {
authors: vec![Author {
surname: Some("Kahle".to_string()),
..Author::default()
}],
date: Some("2000".to_string()),
..Biblio::default()
};
let path = dir.join("refs.bib");
let link = grobid_bibtex::FileLink::Absolute;
let report = collection::merge_file(&path, &[(PathBuf::from("paper.pdf"), biblio)], &link)
.unwrap();
assert_eq!(report.entries.len(), 1);
assert_eq!(report.total, 1);Beyond Collection, the bibtex module provides the single-record helpers:
entry_type(), year(), suggest_key() and unique_key() for entry types
and collision-free citation keys, format_entry(), format_entry_with_file()
and format_entry_with_link() for typed BibLaTeX output, format_all() for
batches, and suggest_file_name(), suggest_file_stem_with(), unique_path()
and unique_path_with_year() for the file-naming policies (FileStemStyle::Full
and FileStemStyle::Keyed).
bibtex::FileLink decides what the file field says: None writes no field,
Absolute records the path as given, and Relative(base) records a path
relative to the workspace (or any other base) when one exists. A document on
another filesystem root — another drive on Windows — stays absolute.
§Binaries
Both binaries process all PDFs in a directory recursively through a running
GROBID server and write their bibliographic metadata as BibTeX. They share
the common options below; run either with --help for all of them.
§pdf2bibtex
Writes one entry per document (its header):
cargo run --release --bin pdf2bibtex -- ~/papers -s http://localhost:8070With -r/--rename, each PDF is renamed after its metadata has been
extracted to Author_Year_<first 10 title words>.pdf, e.g.
Kahle_2000_The_Barc_model_for_continuous_variables.pdf. Parts GROBID could
not extract (author, year or title) are dropped, non-ASCII characters are
removed, and colliding names get a -2, -3, … suffix. Names longer
than the filesystem’s 255-byte limit are shortened at a character boundary.
With -l/--link, each entry records the path of its PDF in a file
field, so reference managers can open the document; combined with
-r/--rename, the field points at the renamed file. Paths are recorded
as passed on the command line, so relative input paths stay relative. Library
users that want portable links pass bibtex::FileLink::Relative(base) to
format_all()/merge_file(); that also makes CiteState resolve the
bibliography when the project moves.
With -a/--append or -m/--merge, the entries go into an existing
BibTeX file instead of a new one. Both load the target collection first and
reserve its citation keys, so a new entry whose suggested key already exists
gets a -2, -3, … suffix. --append adds every extracted entry;
--merge skips records that are already there, where a record counts as
present when its normalized field content, one of its identifiers (DOI,
PMID, arXiv) or its PDF file name matches an existing entry. --output,
--append and --merge are mutually exclusive.
With --openalex (requires building with --features openalex), each
extracted header is completed against OpenAlex before entries are written
and PDFs are renamed: missing authors, journal, volume, pages, DOI, … are
filled in from the matching work. See Reference completion against
OpenAlex in the grobid documentation.
§refs2bibtex
Extracts the bibliographic references of all PDFs in a directory (via
/api/processReferences) and writes one BibTeX entry per reference:
cargo run --release --bin refs2bibtex -- ~/papers -s http://localhost:8070Both binaries discover PDFs recursively and process them with a bounded
number of concurrent requests (-w, default 4); per-document failures are
reported on stderr and skipped, and an unresponsive server is detected by a
liveness probe with bounded retries and a clear error message. Entry types
(@article, @incollection, @techreport, @book, @misc) and keys
(first author surname + year, deduplicated) are derived from the parsed
metadata via the bibtex helpers. refs2bibtex skips empty parse results
and drops references with a duplicate DOI, and supports reference
consolidation against CrossRef with -c/--consolidate. Both binaries
accept --openalex when built with --features openalex: refs2bibtex
completes the collected references, pdf2bibtex each document header,
before the entries are written:
cargo run --release --features openalex --bin refs2bibtex -- ~/papers --openalex
cargo run --release --features openalex --bin pdf2bibtex -- ~/papers --openalex§Features
openalex: enables the OpenAlex completion tier of thegrobidclient, thecompletemodule and the--openalexoption of both binaries.
Re-exports§
pub use bibtex::FileLink;pub use collection::Collection;
Modules§
- bibtex
- BibTeX rendering for parsed bibliographic records.
- cite
- Chat-side citation resolution:
@keymentions become attached documents. - collection
- The BibTeX-centered reference collection.
- complete
- Second-tier completion of extracted records against OpenAlex.
- extract
- Extraction of bibliographic records from PDFs with GROBID.
- files
- Working with the PDF files behind the records.
- openalex
- Second-tier reference completion against OpenAlex.
- prepass
- Reusable GROBID pre-pass orchestration.
- tei
- Parsing of GROBID TEI-XML into strongly typed structures.
Structs§
- Author
- A person (author or editor), parsed from a TEI
<persName>element. - Biblio
- A complete bibliographic record.
- Citation
- A single bibliographic reference (one
<biblStruct>element), e.g. from the reference list of a processed document. - Grobid
Client - A client for the GROBID REST API.
- Grobid
Client Builder - Builder for
GrobidClient. - Process
Options - Options for a GROBID processing request.
- Retry
Policy - Retry policy for requests that hit a busy GROBID server (HTTP 503 or
429). Retries are spaced with exponential backoff: the delay after the
n-th attempt is
initial_delay * multiplier^n, capped atmax_delay.
Enums§
- Citation
Consolidation - Consolidation level for citations/references.
- Error
- Errors produced by the grobid client and TEI parser.
- Header
Consolidation - Consolidation level for the document header.
- PdfInput
- A PDF to be processed by GROBID: either a path on disk, or the document bytes with a file name. In-memory inputs avoid a round-trip through the filesystem, e.g. when the PDF was downloaded or comes from an object store.
Constants§
- DEFAULT_
GROBID_ URL - The default GROBID server URL (a locally running instance).