spectre-rs

A native Rust PDF extraction engine. Read-only drop-in for the surface most LLM/RAG pipelines actually call: text, structured metadata, AcroForm widgets, image bytes, encrypted PDFs, and markdown for ingestion. Built on a lazy parser that opens a PDF in 8 µs (median, ICDAR 2013). No C dependencies, no AGPL, full pure-Rust decryption for every Standard Security Handler revision Acrobat has shipped since 1996.

import spectre_rs
doc      = spectre_rs.Document(open("doc.pdf","rb").read())
text     = doc.text()
markdown = doc.markdown()        # 374× faster than pymupdf4llm on 46-PDF corpus, F1 0.816 vs 0.723
words    = doc.words()
blocks   = doc.blocks()
hits     = doc.search("New York")
toc      = doc.toc()
links    = doc.links()
images   = doc.images()
widgets  = doc.widgets()         # AcroForm fields, 14.3× faster than pymupdf
img      = doc.image_bytes(images[0].xref)
open(f"page0.{img.ext}", "wb").write(img.bytes)

# Encrypted? Pass the password — V=1..5 R=2..6 all supported.
encrypted = spectre_rs.Document(open("locked.pdf","rb").read(), password=b"secret")

use spectre_pdf::Document;
let bytes  = std::fs::read("doc.pdf")?;
let doc    = Document::open(&bytes)?;
let text   = doc.text()?;
let words  = doc.words(None)?;
let blocks = doc.blocks(None)?;
let toc    = doc.toc()?;

Headline

Five measurements that decide most evaluator comparisons. All numbers from this repository's bench scripts; full per-document data is committed under data/.

• Text extraction on the canonical PDF benchmark (ICDAR 2013 Table Competition, full 67-PDF corpus, strict mode):

  Tool	Total time	vs spectre
  spectre_rs extract_text (strict)	0.32 s	1.0× baseline
  pymupdf	0.37 s	1.17× slower
  pdfminer.six	8.45 s	26.53× slower
  pdfplumber	12.89 s	40.46× slower

  Char-count parity vs pymupdf: 60 of 65 PDFs within ±10% (median Δ +0.6%, mean Δ −0.1%). 3 of 67 PDFs raise ExtractError::PageExtractFailed on CID fonts missing /ToUnicode; the failure time is counted in spectre's total.

• Block segmentation on human-annotated ground truth (DocLayNet, 62 val pages, ~47 per metric where both libraries returned valid output):

  Metric	spectre-rs	pymupdf
  Token micro-F1	0.860	0.775
  Boundary F1	0.623	0.387
  V-measure	0.762	0.683
  pair-F1	0.545	0.607
  ARI	0.490	0.541

  Three user-facing metrics (token-F1, boundary-F1, V-measure) favor spectre; two pair-counting metrics (pair-F1, ARI) favor pymupdf.

• Encrypted PDFs (7 variants from RC4-40 through AES-256 R=5 and R=6, full Algorithm 2.B): every variant unlocks to byte-identical baseline text in pure Rust at 10.32× the end-to-end speed of pymupdf.authenticate + get_text.

• Markdown for LLM ingestion (46-PDF cross-corpus, mostly IRS Publications, every doc with a /Outline): spectre-rs Document.markdown() runs the full corpus in 2.7 s vs pymupdf4llm's 1020.6 s — 374× faster. Heading-detection F1 against each document's own /Outline: 0.816 vs pymupdf4llm 0.723 (spec wins 35 of 42 documents, ties on 4 degenerate fixtures).

• Persistent-handle pattern: opening the document once and pulling multiple surfaces beats pymupdf 5–130× across every measured API (words, blocks, search, toc, links, annotations, images, pages, info). See the per-surface table below.

Surfaces

Each row is a paired speed × accuracy claim against the closest pymupdf or pymupdf4llm equivalent. The reproducer column points at a script that prints the same numbers from a fresh clone.

Text + structural

Structural metadata

Forms, images, encryption

Other

Not in scope: OCR. For scanned-image PDFs, run ocrmypdf upstream and pipe into spectre-rs. Pure-Rust OCR via ocrs is on the v0.8 roadmap with a published accuracy benchmark.

Install

Python

When wheels are published to PyPI:

pip install spectre-rs
# or
uv pip install spectre-rs

Wheels are built for Linux (x86_64 + aarch64), macOS (x86_64 + arm64), and Windows x64 across CPython 3.10–3.13. Source distribution available for platforms without a prebuilt wheel.

To build from source today (until first PyPI publish):

pip install maturin
maturin build --release --features python
pip install target/wheels/spectre_rs-*.whl

Rust

[dependencies]
spectre_pdf = "1.0"

The Rust crate is published on crates.io as spectre_pdf because spectre-rs was already taken. The library name, GitHub repo, and Python package all remain spectre-rs.

Reproduce every number in this README

# 1. Build the parity example (release profile, needed by the ICDAR bench)
cargo build --release --example parity

# 2. Install the Python comparators
pip install pymupdf pymupdf4llm pdfminer.six pdfplumber

# 3. Install spectre-rs (development mode)
pip install maturin
maturin develop --release --features python

# 4. Build the test corpus (downloads ~30 IRS publications + ICDAR 2013)
bash scripts/expand_corpus.sh

# 5. Each headline number lives in one script:

# Text extraction speed + char-count parity vs pymupdf / pdfminer.six / pdfplumber
python scripts/bench_icdar2013.py

# Block segmentation accuracy on human ground truth
python scripts/doclaynet_metrics_compare.py /path/to/doclaynet/val/

# Markdown speed + heading-F1 (46-PDF corpus, vs pymupdf4llm)
python scripts/sweep_heading_f1.py > sweep_results.json
python scripts/summarize_sweep.py

# v0.7 surfaces (each prints its own row of the table above)
python scripts/bench_widgets_vs_pymupdf.py
python scripts/bench_image_bytes_vs_pymupdf.py
python scripts/bench_dict_vs_pymupdf.py
python scripts/bench_encrypted_vs_pymupdf.py

# 9-surface speed parity (words/blocks/search/toc/links/annotations/images/pages/info)
python scripts/bench_positional_vs_pymupdf.py

Per-PDF data is committed under data/: icdar2013-results.csv (67 rows), parity-icdar2013-final.csv (603), parity-pdfjs-final.csv (450), parity-irs-final.csv (9), doclaynet-metrics-compare.csv, plus the new v0.7 surfaces under data/launch-bench/. Readers who want to verify any aggregate claim can do so directly from the per-document data without running anything.

Methodology

• Speed: single-threaded, release build, warm cache, median of 3–5 calls per file. spectre cold call followed by warm calls; the bench reads the warm reading. The pymupdf and pymupdf4llm comparators are timed the same way in-process. All four corpora ship with raw per-PDF timings.
• DocLayNet accuracy is measured against human-annotated paragraph blocks. Five metrics because no single clustering metric captures both "which block is each token in" and "are the cluster counts right." Token first-occurrence alignment is the headline measurement; bbox-overlap alignment is documented in the per-page CSV as a sensitivity check (Δ +0.04 pair-F1).
• Markdown F1 uses the document's own /Outline TOC as ground truth. Detection rate = exact normalized match against the TOC entry title; precision/recall/F1 computed per document, then mean across the corpus. The 12-PDF sample we used early was unrepresentative of the broader corpus; the 46-PDF number is the load-bearing figure.
• Image-bytes accuracy is pixel-exact equality after decoding both libraries' output through Pillow to RGB. CMYK JPEGs are reported separately because pymupdf converts to RGB lossily without an ICC profile; spectre preserves the original CMYK bytes.
• Where spectre disagrees with pymupdf on text content, the per-PDF CSV row lets you inspect the specific characters. We don't suppress the disagreements — they're committed in data/.

Architecture

The spectre_parse lazy parser (crates/spectre-parse/) drives every surface — Document handle methods and the legacy free-function entrypoints:

• At open: locate the PDF header, walk the xref chain back through every /Prev entry, parse the trailer dictionary. Median work on the ICDAR corpus: 8 µs per document.
• On first access: materialize one object body, decode its filter chain (Flate / LZW / ASCII85 / ASCIIHex with predictor decoding), cache the result. Image-codec streams (DCT / JPX / JBIG2 / CCITT) keep their raw bytes for the image_bytes API to repackage.
• Decryption hooks into get_object and materialize_stream_body, decrypting per-object before the filter chain runs. RC4 40/128-bit + AES-128/256, with constant-time hash comparison on the R=5/R=6 password-derivation step.

Pure Rust top to bottom. No C dependencies. No lopdf.

Status

• v0.7 is the current line. Adds AcroForm widgets, image-pixel decoding, full Standard Security Handler decryption, dict/markdown surfaces with the bbox-merge heading detector.
• v0.5–0.6 introduced the lazy parser and positional/structural surface.
• v0.4 surfaced per-page extraction errors as ExtractError::PageExtractFailed. v0.2–0.1 are retained behind --features python as RustValidator for backward compatibility.

License

MIT

Contributing

cargo fmt
cargo clippy --all-targets --all-features -- -D warnings
cargo test --all-features
cargo bench --bench extract_text -- --quick

CI runs the same on every push and PR.

Built by Ryan Stewart