spectre_pdf 1.0.0

# spectre-rs

[![CI](https://github.com/RyanJamesStewart/spectre-rs/actions/workflows/ci.yml/badge.svg)](https://github.com/RyanJamesStewart/spectre-rs/actions/workflows/ci.yml)
[![Crate](https://img.shields.io/crates/v/spectre_rs.svg)](https://crates.io/crates/spectre_rs)
[![License](https://img.shields.io/crates/l/spectre_rs.svg)](LICENSE)

**A native Rust PDF extraction engine.** Read-only drop-in for the surface most LLM/RAG pipelines actually call: text, structured metadata, AcroForm widgets, image bytes, encrypted PDFs, and markdown for ingestion. Built on a lazy parser that opens a PDF in 8 µs (median, ICDAR 2013). No C dependencies, no AGPL, full pure-Rust decryption for every Standard Security Handler revision Acrobat has shipped since 1996.

```python
import spectre_rs
doc      = spectre_rs.Document(open("doc.pdf","rb").read())
text     = doc.text()
markdown = doc.markdown()        # 374× faster than pymupdf4llm on 46-PDF corpus, F1 0.816 vs 0.723
words    = doc.words()
blocks   = doc.blocks()
hits     = doc.search("New York")
toc      = doc.toc()
links    = doc.links()
images   = doc.images()
widgets  = doc.widgets()         # AcroForm fields, 14.3× faster than pymupdf
img      = doc.image_bytes(images[0].xref)
open(f"page0.{img.ext}", "wb").write(img.bytes)

# Encrypted? Pass the password — V=1..5 R=2..6 all supported.
encrypted = spectre_rs.Document(open("locked.pdf","rb").read(), password=b"secret")
```

```rust
use spectre_pdf::Document;
let bytes  = std::fs::read("doc.pdf")?;
let doc    = Document::open(&bytes)?;
let text   = doc.text()?;
let words  = doc.words(None)?;
let blocks = doc.blocks(None)?;
let toc    = doc.toc()?;
```

## Headline

Five measurements that decide most evaluator comparisons. All numbers from this repository's bench scripts; full per-document data is committed under `data/`.

- **Text extraction on the canonical PDF benchmark** (ICDAR 2013 Table Competition, full 67-PDF corpus, strict mode):

  | Tool | Total time | vs spectre |
  | --- | ---: | ---: |
  | **spectre_rs** `extract_text` (strict) | **0.32 s** | **1.0× baseline** |
  | `pymupdf` | 0.37 s | 1.17× slower |
  | `pdfminer.six` | 8.45 s | 26.53× slower |
  | `pdfplumber` | 12.89 s | 40.46× slower |

  Char-count parity vs pymupdf: 60 of 65 PDFs within ±10% (median Δ +0.6%, mean Δ −0.1%). 3 of 67 PDFs raise `ExtractError::PageExtractFailed` on CID fonts missing `/ToUnicode`; the failure time is counted in spectre's total.

- **Block segmentation on human-annotated ground truth** (DocLayNet, 62 val pages, ~47 per metric where both libraries returned valid output):

  | Metric | spectre-rs | pymupdf |
  | --- | ---: | ---: |
  | Token micro-F1 | **0.860** | 0.775 |
  | Boundary F1 | **0.623** | 0.387 |
  | V-measure | **0.762** | 0.683 |
  | pair-F1 | 0.545 | 0.607 |
  | ARI | 0.490 | 0.541 |

  Three user-facing metrics (token-F1, boundary-F1, V-measure) favor spectre; two pair-counting metrics (pair-F1, ARI) favor pymupdf.

- **Encrypted PDFs** (7 variants from RC4-40 through AES-256 R=5 and R=6, full Algorithm 2.B): every variant unlocks to byte-identical baseline text in pure Rust at **10.32× the end-to-end speed of `pymupdf.authenticate` + `get_text`**.

- **Markdown for LLM ingestion** (46-PDF cross-corpus, mostly IRS Publications, every doc with a `/Outline`): spectre-rs `Document.markdown()` runs the full corpus in **2.7 s vs pymupdf4llm's 1020.6 s — 374× faster**. Heading-detection F1 against each document's own `/Outline`: **0.816 vs pymupdf4llm 0.723** (spec wins 35 of 42 documents, ties on 4 degenerate fixtures).

- **Persistent-handle pattern**: opening the document once and pulling multiple surfaces beats pymupdf 5–130× across every measured API (`words`, `blocks`, `search`, `toc`, `links`, `annotations`, `images`, `pages`, `info`). See the per-surface table below.

## Surfaces

Each row is a paired speed × accuracy claim against the closest pymupdf or pymupdf4llm equivalent. The reproducer column points at a script that prints the same numbers from a fresh clone.

### Text + structural

| Surface | Speed | Accuracy | Reproducer |
| --- | ---: | --- | --- |
| `text()` | **1.17×** vs `pymupdf` (ICDAR 2013, 67 PDFs) | ±10% char-count parity on 60/65 docs vs pymupdf | `scripts/bench_icdar2013.py` |
| `text_lenient()` | 1.20× | Silent-skip behavior matching pymupdf/pdfminer | `scripts/bench_icdar2013.py` |
| `words()` | **6.13×** (persistent handle) / 2.41× (free function) | DocLayNet token-F1 **0.860 vs pymupdf 0.775** | `scripts/bench_positional_vs_pymupdf.py` |
| `blocks()` | **5.42×** (handle) | DocLayNet boundary-F1 **0.623 vs 0.387**; token-F1 0.860 vs 0.775 | `scripts/doclaynet_metrics_compare.py` |
| `search()` | **5.83×** (handle) | Same rect-per-match semantics, including cross-word matches like "New York" | `scripts/bench_positional_vs_pymupdf.py` |
| `dict()` | **2.33×** | Same hierarchy: page → blocks → lines → spans, same key set | `scripts/bench_dict_vs_pymupdf.py` |
| `markdown()` | **374×** vs `pymupdf4llm` (46-PDF corpus) | Heading-detection F1 **0.816 vs pymupdf4llm 0.723** on the same corpus; spec wins 35/42 docs | `scripts/sweep_heading_f1.py` |

### Structural metadata

| Surface | Speed (persistent handle) | Accuracy | Reproducer |
| --- | ---: | --- | --- |
| `toc()` | **134.76×** | Named-destination resolution via `/Names/Dests` name tree | `scripts/bench_positional_vs_pymupdf.py` |
| `links()` | **5.50×** | Includes resolved named destinations | `scripts/bench_positional_vs_pymupdf.py` |
| `annotations()` | **10.97×** | Read-only; emits `subtype`, `rect`, `contents`, `author` | `scripts/bench_positional_vs_pymupdf.py` |
| `images()` | **13.55×** | Inventory (no decoding): xref, dimensions, colorspace, filter chain | `scripts/bench_positional_vs_pymupdf.py` |
| `pages()` | **10.43×** | Per-page mediabox / cropbox / rotation / width / height | `scripts/bench_positional_vs_pymupdf.py` |

### Forms, images, encryption

| Surface | Speed | Accuracy | Reproducer |
| --- | ---: | --- | --- |
| `widgets()` (AcroForm) | **14.32×** (4 IRS forms, 382 widgets) | Field-name Jaccard **1.000**, field-value Jaccard **1.000** vs `pymupdf.Page.widgets()` | `scripts/bench_widgets_vs_pymupdf.py` |
| `image_bytes(xref)` | **2.63×** (10 PDFs, 50 images) | **100% extraction** (50/50 → viewable JPEG/PNG/JP2/JBIG2/TIFF); **100% pixel-exact** on RGB+grayscale (22/22); 1 CMYK image preserved with native colorspace (pymupdf converts to RGB) | `scripts/bench_image_bytes_vs_pymupdf.py` |
| `open_with_password()` | **10.32×** vs `pymupdf.authenticate + get_text` | **7/7 unlock to byte-identical baseline text**: V=1 R=2 RC4-40, V=2 R=3 RC4-128, V=4 R=4 AES-128 CBC, V=5 R=5 AES-256 (withdrawn Acrobat 9), V=5 R=6 AES-256 with full Algorithm 2.B (SHA-2 + AES feedback). Wrong password rejected; constant-time hash comparison via `subtle`. | `scripts/bench_encrypted_vs_pymupdf.py` |

### Other

| Surface | Notes |
| --- | --- |
| `info()` | Page count, version, encryption flag, linearized flag, trailer ID. Sub-millisecond on the persistent handle. |
| `extract_tables()` | **Experimental.** Whitespace-column strategy targets borderless layouts (10-Ks, prospectuses, municipal Official Statements). **41.5× faster than pdfplumber on ICDAR 2013, but F1 0.021 vs pdfplumber 0.419** on the same corpus — borderless heuristics don't match ruled-table ground truth. Ship it for speed-critical borderless-only workloads, not as a competitive table extractor. Fix is on the roadmap. |
| `score_text()`, `score_batch()` | Garbage-text detector, 0.0 (binary noise) → 1.0 (clean prose). Language-fair (CJK / accented Latin score correctly). Parallel via [`rayon`]. No pymupdf equivalent. |

**Not in scope:** OCR. For scanned-image PDFs, run `ocrmypdf` upstream and pipe into spectre-rs. Pure-Rust OCR via `ocrs` is on the v0.8 roadmap with a published accuracy benchmark.

## Install

### Python

When wheels are published to PyPI:

```sh
pip install spectre-rs
# or
uv pip install spectre-rs
```

Wheels are built for Linux (x86_64 + aarch64), macOS (x86_64 + arm64), and Windows x64 across CPython 3.10–3.13. Source distribution available for platforms without a prebuilt wheel.

To build from source today (until first PyPI publish):

```sh
pip install maturin
maturin build --release --features python
pip install target/wheels/spectre_rs-*.whl
```

### Rust

```toml
[dependencies]
spectre_pdf = "1.0"
```

The Rust crate is published on crates.io as `spectre_pdf` because `spectre-rs` was already taken. The library name, GitHub repo, and Python package all remain `spectre-rs`.

## Reproduce every number in this README

```sh
# 1. Build the parity example (release profile, needed by the ICDAR bench)
cargo build --release --example parity

# 2. Install the Python comparators
pip install pymupdf pymupdf4llm pdfminer.six pdfplumber

# 3. Install spectre-rs (development mode)
pip install maturin
maturin develop --release --features python

# 4. Build the test corpus (downloads ~30 IRS publications + ICDAR 2013)
bash scripts/expand_corpus.sh

# 5. Each headline number lives in one script:

# Text extraction speed + char-count parity vs pymupdf / pdfminer.six / pdfplumber
python scripts/bench_icdar2013.py

# Block segmentation accuracy on human ground truth
python scripts/doclaynet_metrics_compare.py /path/to/doclaynet/val/

# Markdown speed + heading-F1 (46-PDF corpus, vs pymupdf4llm)
python scripts/sweep_heading_f1.py > sweep_results.json
python scripts/summarize_sweep.py

# v0.7 surfaces (each prints its own row of the table above)
python scripts/bench_widgets_vs_pymupdf.py
python scripts/bench_image_bytes_vs_pymupdf.py
python scripts/bench_dict_vs_pymupdf.py
python scripts/bench_encrypted_vs_pymupdf.py

# 9-surface speed parity (words/blocks/search/toc/links/annotations/images/pages/info)
python scripts/bench_positional_vs_pymupdf.py
```

Per-PDF data is committed under [`data/`](./data/): `icdar2013-results.csv` (67 rows), `parity-icdar2013-final.csv` (603), `parity-pdfjs-final.csv` (450), `parity-irs-final.csv` (9), `doclaynet-metrics-compare.csv`, plus the new v0.7 surfaces under `data/launch-bench/`. Readers who want to verify any aggregate claim can do so directly from the per-document data without running anything.

## Methodology

- **Speed**: single-threaded, release build, warm cache, median of 3–5 calls per file. spectre cold call followed by warm calls; the bench reads the warm reading. The `pymupdf` and `pymupdf4llm` comparators are timed the same way in-process. All four corpora ship with raw per-PDF timings.
- **DocLayNet accuracy** is measured against human-annotated paragraph blocks. Five metrics because no single clustering metric captures both "which block is each token in" and "are the cluster counts right." Token first-occurrence alignment is the headline measurement; bbox-overlap alignment is documented in the per-page CSV as a sensitivity check (Δ +0.04 pair-F1).
- **Markdown F1** uses the document's own `/Outline` TOC as ground truth. Detection rate = exact normalized match against the TOC entry title; precision/recall/F1 computed per document, then mean across the corpus. The 12-PDF sample we used early was unrepresentative of the broader corpus; the 46-PDF number is the load-bearing figure.
- **Image-bytes accuracy** is pixel-exact equality after decoding both libraries' output through Pillow to RGB. CMYK JPEGs are reported separately because pymupdf converts to RGB lossily without an ICC profile; spectre preserves the original CMYK bytes.
- **Where spectre disagrees with pymupdf on text content**, the per-PDF CSV row lets you inspect the specific characters. We don't suppress the disagreements — they're committed in `data/`.

## Architecture

The `spectre_parse` lazy parser ([`crates/spectre-parse/`](./crates/spectre-parse)) drives every surface — `Document` handle methods and the legacy free-function entrypoints:

- **At open**: locate the PDF header, walk the xref chain back through every `/Prev` entry, parse the trailer dictionary. Median work on the ICDAR corpus: 8 µs per document.
- **On first access**: materialize one object body, decode its filter chain (Flate / LZW / ASCII85 / ASCIIHex with predictor decoding), cache the result. Image-codec streams (DCT / JPX / JBIG2 / CCITT) keep their raw bytes for the `image_bytes` API to repackage.
- **Decryption** hooks into `get_object` and `materialize_stream_body`, decrypting per-object before the filter chain runs. RC4 40/128-bit + AES-128/256, with constant-time hash comparison on the R=5/R=6 password-derivation step.

Pure Rust top to bottom. No C dependencies. No `lopdf`.

## Status

- v0.7 is the current line. Adds AcroForm widgets, image-pixel decoding, full Standard Security Handler decryption, dict/markdown surfaces with the bbox-merge heading detector.
- v0.5–0.6 introduced the lazy parser and positional/structural surface.
- v0.4 surfaced per-page extraction errors as `ExtractError::PageExtractFailed`. v0.2–0.1 are retained behind `--features python` as `RustValidator` for backward compatibility.

## License

[MIT](LICENSE)

## Contributing

```sh
cargo fmt
cargo clippy --all-targets --all-features -- -D warnings
cargo test --all-features
cargo bench --bench extract_text -- --quick
```

CI runs the same on every push and PR.

---

Built by Ryan Stewart