pdf-untangle 0.3.0

Untangle garbled PDF text: layout-aware extraction that reconstructs reading order, strips page furniture, and splits documents into structured, citable sections
Documentation

pdf-untangle

Untangle garbled PDF text: layout-aware extraction that reconstructs reading order, strips page furniture, repairs hyphenation and page-break flow, and — with a declarative profile — splits a document into structured, citable sections. A library and a CLI in one crate.

The problem

A PDF stores text as thousands of individually positioned glyphs: "put an 'R' at (72, 640) in 11pt bold." There are no paragraphs, headings, columns, or reading order in the file. Naive extraction gives you the familiar mishmash — lines interleaved out of order, page headers jammed mid-sentence, words split by hyphens, two columns shuffled together. pdf-untangle reconstructs the logical document, in three layers:

  1. extract — PDF → positioned text runs (pluggable backends)
  2. layout — runs → ordered lines/blocks: reading order, column detection, header/footer stripping, de-hyphenation, page-flow merge
  3. structure — blocks → discrete sections ("provisions"), driven by a TOML profile: heading patterns, a citation template, history/date formats. All document-family knowledge is data, not code.

Built for (and battle-tested on) legal PDFs — court rules and statutes that many U.S. jurisdictions publish only as PDF — but the engine is generic.

Status

Phases 1–4: extraction + layout + structure + wave-1 profiles. Layer 1 is implemented with two feature-gated backends (see Extraction backends below), and Layer 2 turns their positioned runs into reading-ordered lines and blocks: baseline line assembly, two-column detection, repeated header/footer/page-number stripping (with accounting — stripped text is recorded, never silently dropped), conservative de-hyphenation, paragraph merge across page and column breaks, and quantized indentation levels. pdf-untangle text prints the result. Layer 3 applies a declarative TOML profile (heading patterns, history/date parsing, skip regions and blocks, stop markers, a citation template) to those lines and emits structured provisions with full coverage accounting — pdf-untangle extract and pdf-untangle check run it. See docs/DESIGN.md for the full design and phased plan with checkboxes, including the backend decision-gate comparison and the Phase-4 corpus notes.

Ten built-in profiles are green against the real corpus at 100% line coverage each:

profile family corpus documents provisions
cnmi-rules NMI R. Civ. P. one rule-set PDF 97
cnmi-statutes {title} CMC § per-section PDFs 1/file
guam-gca {title} GCA § per-chapter PDFs 4–7/chapter
guam-rules Guam R. Civ. P. one rule-set PDF 98
maine-rules M.R. Civ. P. per-rule + consolidated PDFs 163
connecticut-practice-book Conn. Practice Book § one 699-page book 1530
kentucky-krs KRS per-section PDFs 1/file
pr-rules 32 L.P.R.A. Ap. V, R. one Spanish rule-set PDF 334
north-dakota-century-code N.D. Cent. Code § per-chapter PDFs 53 (ch. 28-01); 602 of 1529 chapters are recognized empty repealed-chapter stubs
wyoming-statutes Wyo. Stat. § per-title PDFs 1034 (title 1)

CLI

pdf-untangle extract rules.pdf --profile cnmi-rules -o out.jsonl   # structured provisions (JSONL)
pdf-untangle check rules.pdf --profile cnmi-rules                  # coverage + integrity dry-run
pdf-untangle check rules.pdf --profile my-profile.toml --json      # machine-readable report
pdf-untangle text rules.pdf                      # clean reading-ordered text (no profile needed)
pdf-untangle text rules.pdf --keep-furniture     # keep headers/footers inline
pdf-untangle text rules.pdf --json               # the Line/Block layout model
pdf-untangle inspect rules.pdf --page 3          # positioned runs (profile-authoring aid)
pdf-untangle inspect rules.pdf --page 3 --json   # same, machine-readable
pdf-untangle profiles                            # list built-ins

check prints a coverage and structure-integrity report and exits non-zero when line coverage falls below 98%, no provisions were detected (except in a document the profile recognizes as intentionally empty — a wholly repealed chapter matching empty_doc_patterns), or a provision number appears twice. Every input line is accounted for — consumed into a provision, skipped by a profile rule (committee notes, front matter), stripped as page furniture, or listed as unconsumed:

tests/corpus-cache/cnmi-rules-r05.pdf — profile cnmi-rules
provisions: 97 (NMI R. Civ. P. 1 … NMI R. Civ. P. 86)
line coverage: 100.00% (6369/6369 accounted: 5315 consumed, 598 skipped, 456 furniture)
sequence: ok
PASS

Extraction backends

Two Layer-1 backends hide behind one trait; cargo features pick what compiles in:

  • backend-pure (default) — pure Rust via pdf-extract/lopdf. Zero native dependencies: cargo install pdf-untangle just works. Real positions and font sizes; no bold/italic flags or font names (the underlying API does not expose them), and mid-line x-positions can drift on fonts with missing width tables.
  • backend-pdfium — Google PDFium via pdfium-render. Highest fidelity: exact bounding boxes, font names, bold/italic flags. Requires the native libpdfium at runtime (loaded dynamically; nothing extra at build time) — see the development section for fetching it.

When both features are enabled, dispatch prefers PDFium at runtime if its library loads and falls back to the pure backend with a logged warning.

Development

PDFium library (for backend-pdfium): the backend searches the PDFIUM_DYNAMIC_LIB_PATH environment variable (directory or file path), then a git-ignored .pdfium/ directory under the current directory, then the system library path. For local work, fetch a prebuilt library from the bblanchon/pdfium-binaries releases:

scripts/fetch-pdfium.sh          # detects your platform; installs into .pdfium/

Corpus tests: integration tests exercise real jurisdiction PDFs cached in git-ignored tests/corpus-cache/. They skip (with a message) unless opted in, so offline/CI runs stay green:

# one-time, polite single fetches:
curl -o tests/corpus-cache/cnmi-t1-2051.pdf     https://cnmilaw.org/pdf/cmc_section/T1/2051.pdf
curl -o tests/corpus-cache/cnmi-rules-r05.pdf   https://cnmilaw.org/pdf/courtrules/R05.pdf
curl -o tests/corpus-cache/guam-1gc001.pdf      https://col.guamcourts.gov/sites/default/files/1gc001.pdf
curl -o tests/corpus-cache/guam-7gc002.pdf      https://col.guamcourts.gov/sites/default/files/07gc002.pdf
curl -o tests/corpus-cache/guam-grcp-2024-04-01.pdf 'https://guamcourts.gov/sites/default/files/GRCP%2020240401_Control.pdf'
curl -o tests/corpus-cache/maine-civ-rule11-2023-11-15.pdf https://www.courts.maine.gov/rules/text/MRCivPPlus/mr_civ_p_11_plus_2023-11-15.pdf
curl -o tests/corpus-cache/maine-civ-rule12-2023-11-15.pdf https://www.courts.maine.gov/rules/text/MRCivPPlus/mr_civ_p_12_plus_2023-11-15.pdf
curl -o tests/corpus-cache/maine-civ-rule14-2018-07-11.pdf https://www.courts.maine.gov/rules/text/MRCivPPlus/mr_civ_p_14_plus_2018-07-11.pdf
curl -o tests/corpus-cache/maine-civ-consolidated-2026-06-01.pdf https://www.courts.maine.gov/rules/text/mr_civ_p_only_2026-06-01.pdf
curl -o tests/corpus-cache/ct-practice-book.pdf https://www.jud.ct.gov/publications/PracticeBook/PB.pdf
curl -o tests/corpus-cache/ky-krs-362-150.pdf   'https://apps.legislature.ky.gov/law/statutes/statute.aspx?id=34289'
curl -o tests/corpus-cache/ky-krs-362-155.pdf   'https://apps.legislature.ky.gov/law/statutes/statute.aspx?id=36747'
curl -o tests/corpus-cache/ky-krs-362-160.pdf   'https://apps.legislature.ky.gov/law/statutes/statute.aspx?id=34291'
curl -o tests/corpus-cache/pr-reglas-proc-civil.pdf https://www.poderjudicial.pr/Documentos/SecretariadoConf/Reglas-de-Proc-Civil.pdf
curl -o tests/corpus-cache/nd-cencode-t28c01.pdf https://ndlegis.gov/cencode/t28c01.pdf
curl -o tests/corpus-cache/nd-cencode-t02c01.pdf https://ndlegis.gov/cencode/t02c01.pdf
curl -o tests/corpus-cache/wy-statutes-title01.pdf https://wyoleg.gov/statutes/compress/title01.pdf

PDF_UNTANGLE_CORPUS_TESTS=1 cargo test --all-features

Expected checksums for every cached PDF are pinned in tests/golden/layer3/MANIFEST.toml (sources publish revisions in place; a checksum change means the document moved on and the goldens need a reviewed regeneration).

Golden outputs: tests/golden/layer2/ holds committed text output and tests/golden/layer3/ committed extract JSONL for the corpus documents, pinned to the pure backend (see each directory's MANIFEST.toml). After a deliberate behavior change, regenerate and review the diff:

PDF_UNTANGLE_CORPUS_TESTS=1 UPDATE_GOLDEN=1 cargo test --test layout_corpus --test structure_corpus

Backend comparison (the Phase-1 decision-gate harness):

cargo run --release --example backend_compare --features backend-pdfium,backend-pure

Two invariants make output trustworthy: line-coverage accounting (every input line is consumed, skipped by rule, or reported — silent text loss is impossible) and structure-integrity checks (citation-sequence sanity: gaps beyond the profile's expected_gaps and duplicate numbers are flagged). Each record carries a confidence score with a human-readable confidence_basis: document line coverage × a citation factor (every template placeholder resolved) × a date factor (parsed from history > document adoption date > sentinel). Effective dates are never null: a date parsed from the provision's history lines, else the document-level adoption date (profile-detected or declared), else the corpus-standard 1800-01-01 sentinel.

Writing a profile

All document-family knowledge is data: a profile is one TOML file, and adding a jurisdiction means writing one — never forking the engine. The workflow that produced every built-in:

  1. Look at the document. pdf-untangle text doc.pdf shows the reconstructed reading order (add --json for per-line positions, sizes, and block structure; inspect --page N shows raw runs). Find the heading shape, the history/credit lines, the non-provision content (TOC, front matter, commentary, appendices), and where dates live.
  2. Declare the routing keys and the citation template. jurisdiction, kind, and citation_template are required; the template is the single source of truth for the citation strings the profile emits and must be agreed with whatever consumes them before the profile ships (draft = true until then). {num} is the provision number; {title} resolves via title_patterns (running heads are usually where a statute code's title number lives).
  3. Write the heading patterns. Regexes with a required (?P<num>…) capture and optional heading/title captures, matched against trimmed lines. Four guards reject look-alikes, all data-tunable: the x0 cluster (heading_x0_cluster, on by default) keeps candidates near the dominant heading left edge — disable it for two-column layouts and for documents whose TOC out-clusters centered body headings; the monotonic guard (monotonic_numbers, on by default) rejects cross-references and TOC re-listings; style hints ([heading_style]) add bold/size/centering requirements where the backend supports them; and toc_restart handles the nasty case where TOC entries are typographically identical to real headings — when the first number comes around again, the first pass is reclassified as skipped front matter.
  4. Name the non-provision content. skip_patterns open a skipped region that runs to the next accepted heading (front matter, committee notes, division heads); skip_block_patterns skip exactly one layout block (content interleaved with body text, like a chapter index set in a sidebar column); stop_patterns end detection for good (trailing form appendices whose credit lines would otherwise attach to the final provision). Skipped content is recorded and counted, never lost — do not over-skip to inflate coverage.
  5. Wire the dates. history_patterns mark credit lines; they are captured as the provision's history and scanned with date_formats (strftime-style; Spanish month names normalize through a built-in table). The LAST parsed date wins, so if a family lists amendments newest-first (Kentucky's "History:" blocks), skip that block instead and let a cleaner line carry the date. adoption_patterns/adoption_date supply the document-level fallback; the 1800-01-01 sentinel is the never-null floor and is visible in confidence_basis.
  6. Declare the family's empty documents, if it has them. Some sources publish a PDF for every chapter even when the whole chapter is repealed (North Dakota's "[Repealed by S.L. 1947, ch. 1, § 18]" stubs — 602 of 1529 chapter PDFs). empty_doc_patterns name the disposition line; a provision-less document matching one is reported as a recognized empty document (empty_doc_marker) and passes check instead of failing "no provisions detected". The patterns are only consulted when zero provisions were found, so they can never mask content in a normal document.
  7. Gate on pdf-untangle check. It must PASS: ≥98% line coverage, sequence sane (declare genuinely reserved numbers in expected_gaps — never paper over a real gap), no duplicates, and provisions found (unless a declared empty-document marker matched). Unconsumed lines are listed verbatim; each one is either a bug in the profile or content that needs a skip rule naming it.

The built-ins under profiles/ are worked examples, from the trivial (kentucky-krs: one section per file) to the adversarial (connecticut-practice-book: 699 pages, two columns, multiple embedded rule sets, interleaved chapter indexes, a trailing forms appendix). Each file's header comment documents the family's quirks and the reasoning behind every pattern.

Library

let profile = pdf_untangle::Profile::builtin("cnmi-rules")?;
let doc = pdf_untangle::extract(Path::new("rules.pdf"), &profile)?;
for p in doc.provisions { println!("{}: {}", p.citation, p.heading.unwrap_or_default()); }
assert!(doc.line_coverage >= 0.98, "lost lines: {:?}", doc.unconsumed_lines);

Non-goals (v1)

Scanned/image PDFs (OCR is on the v2 roadmap), table extraction as structured data, and fetching PDFs from the network — bring your own bytes.

License

MIT OR Apache-2.0, at your option.