pdf-untangle
Untangle garbled PDF text: layout-aware extraction that reconstructs reading order, strips page furniture, repairs hyphenation and page-break flow, and — with a declarative profile — splits a document into structured, citable sections. A library and a CLI in one crate.
The problem
A PDF stores text as thousands of individually positioned glyphs: "put an 'R'
at (72, 640) in 11pt bold." There are no paragraphs, headings, columns, or
reading order in the file. Naive extraction gives you the familiar mishmash —
lines interleaved out of order, page headers jammed mid-sentence, words split
by hyphens, two columns shuffled together. pdf-untangle reconstructs the
logical document, in three layers:
- extract — PDF → positioned text runs (pluggable backends)
- layout — runs → ordered lines/blocks: reading order, column detection, header/footer stripping, de-hyphenation, page-flow merge
- structure — blocks → discrete sections ("provisions"), driven by a TOML profile: heading patterns, a citation template, history/date formats. All document-family knowledge is data, not code.
Built for (and battle-tested on) legal PDFs — court rules and statutes that many U.S. jurisdictions publish only as PDF — but the engine is generic.
Status
Phases 1–4: extraction + layout + structure + wave-1 profiles. Layer 1 is
implemented with two feature-gated backends (see Extraction backends
below), and Layer 2 turns their positioned runs into reading-ordered lines
and blocks: baseline line assembly, two-column detection, repeated
header/footer/page-number stripping (with accounting — stripped text is
recorded, never silently dropped), conservative de-hyphenation, paragraph
merge across page and column breaks, and quantized indentation levels.
pdf-untangle text prints the result. Layer 3 applies a declarative TOML
profile (heading patterns, history/date parsing, skip regions and
blocks, stop markers, a citation template) to those lines and emits
structured provisions with full coverage accounting — pdf-untangle extract
and pdf-untangle check run it. See docs/DESIGN.md for
the full design and phased plan with checkboxes, including the backend
decision-gate comparison and the Phase-4 corpus notes.
Ten built-in profiles are green against the real corpus at 100% line coverage each:
| profile | family | corpus documents | provisions |
|---|---|---|---|
cnmi-rules |
NMI R. Civ. P. | one rule-set PDF | 97 |
cnmi-statutes |
{title} CMC § | per-section PDFs | 1/file |
guam-gca |
{title} GCA § | per-chapter PDFs | 4–7/chapter |
guam-rules |
Guam R. Civ. P. | one rule-set PDF | 98 |
maine-rules |
M.R. Civ. P. | per-rule + consolidated PDFs | 163 |
connecticut-practice-book |
Conn. Practice Book § | one 699-page book | 1530 |
kentucky-krs |
KRS | per-section PDFs | 1/file |
pr-rules |
32 L.P.R.A. Ap. V, R. | one Spanish rule-set PDF | 334 |
north-dakota-century-code |
N.D. Cent. Code § | per-chapter PDFs | 53 (ch. 28-01); 602 of 1529 chapters are recognized empty repealed-chapter stubs |
wyoming-statutes |
Wyo. Stat. § | per-title PDFs | 1034 (title 1) |
CLI
pdf-untangle extract rules.pdf --profile cnmi-rules -o out.jsonl # structured provisions (JSONL)
pdf-untangle check rules.pdf --profile cnmi-rules # coverage + integrity dry-run
pdf-untangle check rules.pdf --profile my-profile.toml --json # machine-readable report
pdf-untangle text rules.pdf # clean reading-ordered text (no profile needed)
pdf-untangle text rules.pdf --keep-furniture # keep headers/footers inline
pdf-untangle text rules.pdf --json # the Line/Block layout model
pdf-untangle inspect rules.pdf --page 3 # positioned runs (profile-authoring aid)
pdf-untangle inspect rules.pdf --page 3 --json # same, machine-readable
pdf-untangle profiles # list built-ins
check prints a coverage and structure-integrity report and exits non-zero
when line coverage falls below 98%, no provisions were detected (except in a
document the profile recognizes as intentionally empty — a wholly repealed
chapter matching empty_doc_patterns), or a provision number appears twice. Every input line is accounted for — consumed
into a provision, skipped by a profile rule (committee notes, front matter),
stripped as page furniture, or listed as unconsumed:
tests/corpus-cache/cnmi-rules-r05.pdf — profile cnmi-rules
provisions: 97 (NMI R. Civ. P. 1 … NMI R. Civ. P. 86)
line coverage: 100.00% (6369/6369 accounted: 5315 consumed, 598 skipped, 456 furniture)
sequence: ok
PASS
Extraction backends
Two Layer-1 backends hide behind one trait; cargo features pick what compiles in:
backend-pure(default) — pure Rust viapdf-extract/lopdf. Zero native dependencies:cargo install pdf-untanglejust works. Real positions and font sizes; no bold/italic flags or font names (the underlying API does not expose them), and mid-line x-positions can drift on fonts with missing width tables.backend-pdfium— Google PDFium viapdfium-render. Highest fidelity: exact bounding boxes, font names, bold/italic flags. Requires the nativelibpdfiumat runtime (loaded dynamically; nothing extra at build time) — see the development section for fetching it.
When both features are enabled, dispatch prefers PDFium at runtime if its library loads and falls back to the pure backend with a logged warning.
Development
PDFium library (for backend-pdfium): the backend searches the
PDFIUM_DYNAMIC_LIB_PATH environment variable (directory or file path), then
a git-ignored .pdfium/ directory under the current directory, then the
system library path. For local work, fetch a prebuilt library from the
bblanchon/pdfium-binaries
releases:
Corpus tests: integration tests exercise real jurisdiction PDFs cached in
git-ignored tests/corpus-cache/. They skip (with a message) unless opted in,
so offline/CI runs stay green:
# one-time, polite single fetches:
PDF_UNTANGLE_CORPUS_TESTS=1
Expected checksums for every cached PDF are pinned in
tests/golden/layer3/MANIFEST.toml (sources publish revisions in place; a
checksum change means the document moved on and the goldens need a reviewed
regeneration).
Golden outputs: tests/golden/layer2/ holds committed text output and
tests/golden/layer3/ committed extract JSONL for the corpus documents,
pinned to the pure backend (see each directory's MANIFEST.toml). After a
deliberate behavior change, regenerate and review the diff:
PDF_UNTANGLE_CORPUS_TESTS=1 UPDATE_GOLDEN=1
Backend comparison (the Phase-1 decision-gate harness):
Two invariants make output trustworthy: line-coverage accounting (every
input line is consumed, skipped by rule, or reported — silent text loss is
impossible) and structure-integrity checks (citation-sequence sanity:
gaps beyond the profile's expected_gaps and duplicate numbers are flagged).
Each record carries a confidence score with a human-readable
confidence_basis: document line coverage × a citation factor (every
template placeholder resolved) × a date factor (parsed from history >
document adoption date > sentinel). Effective dates are never null: a
date parsed from the provision's history lines, else the document-level
adoption date (profile-detected or declared), else the corpus-standard
1800-01-01 sentinel.
Writing a profile
All document-family knowledge is data: a profile is one TOML file, and adding a jurisdiction means writing one — never forking the engine. The workflow that produced every built-in:
- Look at the document.
pdf-untangle text doc.pdfshows the reconstructed reading order (add--jsonfor per-line positions, sizes, and block structure;inspect --page Nshows raw runs). Find the heading shape, the history/credit lines, the non-provision content (TOC, front matter, commentary, appendices), and where dates live. - Declare the routing keys and the citation template.
jurisdiction,kind, andcitation_templateare required; the template is the single source of truth for the citation strings the profile emits and must be agreed with whatever consumes them before the profile ships (draft = trueuntil then).{num}is the provision number;{title}resolves viatitle_patterns(running heads are usually where a statute code's title number lives). - Write the heading patterns. Regexes with a required
(?P<num>…)capture and optionalheading/titlecaptures, matched against trimmed lines. Four guards reject look-alikes, all data-tunable: the x0 cluster (heading_x0_cluster, on by default) keeps candidates near the dominant heading left edge — disable it for two-column layouts and for documents whose TOC out-clusters centered body headings; the monotonic guard (monotonic_numbers, on by default) rejects cross-references and TOC re-listings; style hints ([heading_style]) add bold/size/centering requirements where the backend supports them; andtoc_restarthandles the nasty case where TOC entries are typographically identical to real headings — when the first number comes around again, the first pass is reclassified as skipped front matter. - Name the non-provision content.
skip_patternsopen a skipped region that runs to the next accepted heading (front matter, committee notes, division heads);skip_block_patternsskip exactly one layout block (content interleaved with body text, like a chapter index set in a sidebar column);stop_patternsend detection for good (trailing form appendices whose credit lines would otherwise attach to the final provision). Skipped content is recorded and counted, never lost — do not over-skip to inflate coverage. - Wire the dates.
history_patternsmark credit lines; they are captured as the provision's history and scanned withdate_formats(strftime-style; Spanish month names normalize through a built-in table). The LAST parsed date wins, so if a family lists amendments newest-first (Kentucky's "History:" blocks), skip that block instead and let a cleaner line carry the date.adoption_patterns/adoption_datesupply the document-level fallback; the1800-01-01sentinel is the never-null floor and is visible inconfidence_basis. - Declare the family's empty documents, if it has them. Some sources
publish a PDF for every chapter even when the whole chapter is repealed
(North Dakota's "[Repealed by S.L. 1947, ch. 1, § 18]" stubs — 602 of
1529 chapter PDFs).
empty_doc_patternsname the disposition line; a provision-less document matching one is reported as a recognized empty document (empty_doc_marker) and passescheckinstead of failing "no provisions detected". The patterns are only consulted when zero provisions were found, so they can never mask content in a normal document. - Gate on
pdf-untangle check. It must PASS: ≥98% line coverage, sequence sane (declare genuinely reserved numbers inexpected_gaps— never paper over a real gap), no duplicates, and provisions found (unless a declared empty-document marker matched). Unconsumed lines are listed verbatim; each one is either a bug in the profile or content that needs a skip rule naming it.
The built-ins under profiles/ are worked examples, from the trivial
(kentucky-krs: one section per file) to the adversarial
(connecticut-practice-book: 699 pages, two columns, multiple embedded
rule sets, interleaved chapter indexes, a trailing forms appendix). Each
file's header comment documents the family's quirks and the reasoning
behind every pattern.
Library
let profile = builtin?;
let doc = extract?;
for p in doc.provisions
assert!;
Non-goals (v1)
Scanned/image PDFs (OCR is on the v2 roadmap), table extraction as structured data, and fetching PDFs from the network — bring your own bytes.
License
MIT OR Apache-2.0, at your option.