# pdf-untangle
Untangle garbled PDF text: layout-aware extraction that reconstructs reading
order, strips page furniture, repairs hyphenation and page-break flow, and —
with a declarative *profile* — splits a document into structured, citable
sections. A library and a CLI in one crate.
## The problem
A PDF stores text as thousands of individually positioned glyphs: *"put an 'R'
at (72, 640) in 11pt bold."* There are no paragraphs, headings, columns, or
reading order in the file. Naive extraction gives you the familiar mishmash —
lines interleaved out of order, page headers jammed mid-sentence, words split
by hyphens, two columns shuffled together. `pdf-untangle` reconstructs the
logical document, in three layers:
1. **extract** — PDF → positioned text runs (pluggable backends)
2. **layout** — runs → ordered lines/blocks: reading order, column detection,
header/footer stripping, de-hyphenation, page-flow merge
3. **structure** — blocks → discrete sections ("provisions"), driven by a TOML
**profile**: heading patterns, a citation template, history/date formats.
All document-family knowledge is data, not code.
Built for (and battle-tested on) legal PDFs — court rules and statutes that
many U.S. jurisdictions publish only as PDF — but the engine is generic.
## Status
**Phases 1–4: extraction + layout + structure + wave-1 profiles.** Layer 1 is
implemented with two feature-gated backends (see *Extraction backends*
below), and Layer 2 turns their positioned runs into reading-ordered lines
and blocks: baseline line assembly, two-column detection, repeated
header/footer/page-number stripping (with accounting — stripped text is
recorded, never silently dropped), conservative de-hyphenation, paragraph
merge across page and column breaks, and quantized indentation levels.
`pdf-untangle text` prints the result. Layer 3 applies a declarative TOML
**profile** (heading patterns, history/date parsing, skip regions and
blocks, stop markers, a citation template) to those lines and emits
structured provisions with full coverage accounting — `pdf-untangle extract`
and `pdf-untangle check` run it. See [docs/DESIGN.md](docs/DESIGN.md) for
the full design and phased plan with checkboxes, including the backend
decision-gate comparison and the Phase-4 corpus notes.
Ten built-in profiles are green against the real corpus at 100% line
coverage each:
| `cnmi-rules` | NMI R. Civ. P. | one rule-set PDF | 97 |
| `cnmi-statutes` | {title} CMC § | per-section PDFs | 1/file |
| `guam-gca` | {title} GCA § | per-chapter PDFs | 4–7/chapter |
| `guam-rules` | Guam R. Civ. P. | one rule-set PDF | 98 |
| `maine-rules` | M.R. Civ. P. | per-rule + consolidated PDFs | 163 |
| `connecticut-practice-book` | Conn. Practice Book § | one 699-page book | 1530 |
| `kentucky-krs` | KRS | per-section PDFs | 1/file |
| `pr-rules` | 32 L.P.R.A. Ap. V, R. | one Spanish rule-set PDF | 334 |
| `north-dakota-century-code` | N.D. Cent. Code § | per-chapter PDFs | 53 (ch. 28-01); 602 of 1529 chapters are recognized empty repealed-chapter stubs |
| `wyoming-statutes` | Wyo. Stat. § | per-title PDFs | 1034 (title 1) |
## CLI
```
pdf-untangle extract rules.pdf --profile cnmi-rules -o out.jsonl # structured provisions (JSONL)
pdf-untangle check rules.pdf --profile cnmi-rules # coverage + integrity dry-run
pdf-untangle check rules.pdf --profile my-profile.toml --json # machine-readable report
pdf-untangle text rules.pdf # clean reading-ordered text (no profile needed)
pdf-untangle text rules.pdf --keep-furniture # keep headers/footers inline
pdf-untangle text rules.pdf --json # the Line/Block layout model
pdf-untangle inspect rules.pdf --page 3 # positioned runs (profile-authoring aid)
pdf-untangle inspect rules.pdf --page 3 --json # same, machine-readable
pdf-untangle profiles # list built-ins
```
`check` prints a coverage and structure-integrity report and exits non-zero
when line coverage falls below 98%, no provisions were detected (except in a
document the profile recognizes as intentionally empty — a wholly repealed
chapter matching `empty_doc_patterns`), or a provision number appears twice. Every input line is accounted for — consumed
into a provision, skipped by a profile rule (committee notes, front matter),
stripped as page furniture, or listed as unconsumed:
```
tests/corpus-cache/cnmi-rules-r05.pdf — profile cnmi-rules
provisions: 97 (NMI R. Civ. P. 1 … NMI R. Civ. P. 86)
line coverage: 100.00% (6369/6369 accounted: 5315 consumed, 598 skipped, 456 furniture)
sequence: ok
PASS
```
## Extraction backends
Two Layer-1 backends hide behind one trait; cargo features pick what compiles
in:
- **`backend-pure`** *(default)* — pure Rust via
[`pdf-extract`](https://crates.io/crates/pdf-extract)/`lopdf`. Zero native
dependencies: `cargo install pdf-untangle` just works. Real positions and
font sizes; **no bold/italic flags or font names** (the underlying API does
not expose them), and mid-line x-positions can drift on fonts with missing
width tables.
- **`backend-pdfium`** — Google PDFium via
[`pdfium-render`](https://crates.io/crates/pdfium-render). Highest
fidelity: exact bounding boxes, font names, bold/italic flags. Requires the
native `libpdfium` at *runtime* (loaded dynamically; nothing extra at build
time) — see the development section for fetching it.
When both features are enabled, dispatch prefers PDFium at runtime if its
library loads and falls back to the pure backend with a logged warning.
## Development
**PDFium library (for `backend-pdfium`):** the backend searches the
`PDFIUM_DYNAMIC_LIB_PATH` environment variable (directory or file path), then
a git-ignored `.pdfium/` directory under the current directory, then the
system library path. For local work, fetch a prebuilt library from the
[bblanchon/pdfium-binaries](https://github.com/bblanchon/pdfium-binaries)
releases:
```sh
scripts/fetch-pdfium.sh # detects your platform; installs into .pdfium/
```
**Corpus tests:** integration tests exercise real jurisdiction PDFs cached in
git-ignored `tests/corpus-cache/`. They skip (with a message) unless opted in,
so offline/CI runs stay green:
```sh
# one-time, polite single fetches:
curl -o tests/corpus-cache/cnmi-t1-2051.pdf https://cnmilaw.org/pdf/cmc_section/T1/2051.pdf
curl -o tests/corpus-cache/cnmi-rules-r05.pdf https://cnmilaw.org/pdf/courtrules/R05.pdf
curl -o tests/corpus-cache/guam-1gc001.pdf https://col.guamcourts.gov/sites/default/files/1gc001.pdf
curl -o tests/corpus-cache/guam-7gc002.pdf https://col.guamcourts.gov/sites/default/files/07gc002.pdf
curl -o tests/corpus-cache/guam-grcp-2024-04-01.pdf 'https://guamcourts.gov/sites/default/files/GRCP%2020240401_Control.pdf'
curl -o tests/corpus-cache/maine-civ-rule11-2023-11-15.pdf https://www.courts.maine.gov/rules/text/MRCivPPlus/mr_civ_p_11_plus_2023-11-15.pdf
curl -o tests/corpus-cache/maine-civ-rule12-2023-11-15.pdf https://www.courts.maine.gov/rules/text/MRCivPPlus/mr_civ_p_12_plus_2023-11-15.pdf
curl -o tests/corpus-cache/maine-civ-rule14-2018-07-11.pdf https://www.courts.maine.gov/rules/text/MRCivPPlus/mr_civ_p_14_plus_2018-07-11.pdf
curl -o tests/corpus-cache/maine-civ-consolidated-2026-06-01.pdf https://www.courts.maine.gov/rules/text/mr_civ_p_only_2026-06-01.pdf
curl -o tests/corpus-cache/ct-practice-book.pdf https://www.jud.ct.gov/publications/PracticeBook/PB.pdf
curl -o tests/corpus-cache/ky-krs-362-150.pdf 'https://apps.legislature.ky.gov/law/statutes/statute.aspx?id=34289'
curl -o tests/corpus-cache/ky-krs-362-155.pdf 'https://apps.legislature.ky.gov/law/statutes/statute.aspx?id=36747'
curl -o tests/corpus-cache/ky-krs-362-160.pdf 'https://apps.legislature.ky.gov/law/statutes/statute.aspx?id=34291'
curl -o tests/corpus-cache/pr-reglas-proc-civil.pdf https://www.poderjudicial.pr/Documentos/SecretariadoConf/Reglas-de-Proc-Civil.pdf
curl -o tests/corpus-cache/nd-cencode-t28c01.pdf https://ndlegis.gov/cencode/t28c01.pdf
curl -o tests/corpus-cache/nd-cencode-t02c01.pdf https://ndlegis.gov/cencode/t02c01.pdf
curl -o tests/corpus-cache/wy-statutes-title01.pdf https://wyoleg.gov/statutes/compress/title01.pdf
PDF_UNTANGLE_CORPUS_TESTS=1 cargo test --all-features
```
Expected checksums for every cached PDF are pinned in
`tests/golden/layer3/MANIFEST.toml` (sources publish revisions in place; a
checksum change means the document moved on and the goldens need a reviewed
regeneration).
**Golden outputs:** `tests/golden/layer2/` holds committed `text` output and
`tests/golden/layer3/` committed `extract` JSONL for the corpus documents,
pinned to the pure backend (see each directory's `MANIFEST.toml`). After a
deliberate behavior change, regenerate and review the diff:
```sh
PDF_UNTANGLE_CORPUS_TESTS=1 UPDATE_GOLDEN=1 cargo test --test layout_corpus --test structure_corpus
```
**Backend comparison** (the Phase-1 decision-gate harness):
```sh
cargo run --release --example backend_compare --features backend-pdfium,backend-pure
```
Two invariants make output trustworthy: **line-coverage accounting** (every
input line is consumed, skipped by rule, or reported — silent text loss is
impossible) and **structure-integrity checks** (citation-sequence sanity:
gaps beyond the profile's `expected_gaps` and duplicate numbers are flagged).
Each record carries a `confidence` score with a human-readable
`confidence_basis`: document line coverage × a citation factor (every
template placeholder resolved) × a date factor (parsed from history >
document adoption date > sentinel). Effective dates are **never null**: a
date parsed from the provision's history lines, else the document-level
adoption date (profile-detected or declared), else the corpus-standard
`1800-01-01` sentinel.
## Writing a profile
All document-family knowledge is data: a profile is one TOML file, and
adding a jurisdiction means writing one — never forking the engine. The
workflow that produced every built-in:
1. **Look at the document.** `pdf-untangle text doc.pdf` shows the
reconstructed reading order (add `--json` for per-line positions, sizes,
and block structure; `inspect --page N` shows raw runs). Find the heading
shape, the history/credit lines, the non-provision content (TOC, front
matter, commentary, appendices), and where dates live.
2. **Declare the routing keys and the citation template.**
`jurisdiction`, `kind`, and `citation_template` are required; the
template is the single source of truth for the citation strings the
profile emits and must be agreed with whatever consumes them *before*
the profile ships (`draft = true` until then). `{num}` is the provision
number; `{title}` resolves via `title_patterns` (running heads are
usually where a statute code's title number lives).
3. **Write the heading patterns.** Regexes with a required `(?P<num>…)`
capture and optional `heading`/`title` captures, matched against trimmed
lines. Four guards reject look-alikes, all data-tunable:
the **x0 cluster** (`heading_x0_cluster`, on by default) keeps candidates
near the dominant heading left edge — disable it for two-column layouts
and for documents whose TOC out-clusters centered body headings; the
**monotonic guard** (`monotonic_numbers`, on by default) rejects
cross-references and TOC re-listings; **style hints** (`[heading_style]`)
add bold/size/centering requirements where the backend supports them; and
**`toc_restart`** handles the nasty case where TOC entries are
typographically identical to real headings — when the first number comes
around again, the first pass is reclassified as skipped front matter.
4. **Name the non-provision content.** `skip_patterns` open a skipped
region that runs to the next accepted heading (front matter, committee
notes, division heads); `skip_block_patterns` skip exactly one layout
block (content interleaved with body text, like a chapter index set in a
sidebar column); `stop_patterns` end detection for good (trailing form
appendices whose credit lines would otherwise attach to the final
provision). Skipped content is recorded and counted, never lost — do not
over-skip to inflate coverage.
5. **Wire the dates.** `history_patterns` mark credit lines; they are
captured as the provision's history and scanned with `date_formats`
(strftime-style; Spanish month names normalize through a built-in
table). The LAST parsed date wins, so if a family lists amendments
newest-first (Kentucky's "History:" blocks), skip that block instead and
let a cleaner line carry the date. `adoption_patterns`/`adoption_date`
supply the document-level fallback; the `1800-01-01` sentinel is the
never-null floor and is visible in `confidence_basis`.
6. **Declare the family's empty documents, if it has them.** Some sources
publish a PDF for every chapter even when the whole chapter is repealed
(North Dakota's "[Repealed by S.L. 1947, ch. 1, § 18]" stubs — 602 of
1529 chapter PDFs). `empty_doc_patterns` name the disposition line; a
provision-less document matching one is reported as a recognized empty
document (`empty_doc_marker`) and passes `check` instead of failing
"no provisions detected". The patterns are only consulted when zero
provisions were found, so they can never mask content in a normal
document.
7. **Gate on `pdf-untangle check`.** It must PASS: ≥98% line coverage,
sequence sane (declare genuinely reserved numbers in `expected_gaps` —
never paper over a real gap), no duplicates, and provisions found
(unless a declared empty-document marker matched). Unconsumed lines are
listed verbatim; each one is either a bug in the profile or content that
needs a skip rule naming it.
The built-ins under `profiles/` are worked examples, from the trivial
(`kentucky-krs`: one section per file) to the adversarial
(`connecticut-practice-book`: 699 pages, two columns, multiple embedded
rule sets, interleaved chapter indexes, a trailing forms appendix). Each
file's header comment documents the family's quirks and the reasoning
behind every pattern.
## Library
```rust
let profile = pdf_untangle::Profile::builtin("cnmi-rules")?;
let doc = pdf_untangle::extract(Path::new("rules.pdf"), &profile)?;
for p in doc.provisions { println!("{}: {}", p.citation, p.heading.unwrap_or_default()); }
assert!(doc.line_coverage >= 0.98, "lost lines: {:?}", doc.unconsumed_lines);
```
## Non-goals (v1)
Scanned/image PDFs (OCR is on the v2 roadmap), table extraction as structured
data, and fetching PDFs from the network — bring your own bytes.
## License
MIT OR Apache-2.0, at your option.