pdf-untangle 0.0.1

Untangle garbled PDF text: layout-aware extraction that reconstructs reading order, strips page furniture, and splits documents into structured, citable sections
Documentation
# pdf-untangle — PDF provision extractor (design doc + build plan)

**Owner:** John (scraper-agent lane) · **Drafted:** 2026-07-02 · **Status:** FINAL — approved by John + reviewed/approved by the resolution agent (6 suggestions incorporated); ready to build, start at Phase 0
**Crate:** `pdf-untangle` (verified available on crates.io 2026-07-02; runner-up names also verified free: `pdf-reflow`, `pdf-unscramble`, `legible-pdf`)
**Repo:** `~/code/pdf-untangle` → `git@gitlab.com:john711/pdf-untangle.git` (new, standalone — NOT inside cite-check)
**License:** MIT OR Apache-2.0 (Rust convention; required for clean crates.io publication — no AGPL deps)
**Relationship to cite-check:** the ingester consumes pdf-untangle as a **library dependency** (git dep during development, crates.io version after publish). The CLI binary ships from the same crate for the world + debugging. This is the Wave-3 **PDF capability**; the headless-browser capability is a SEPARATE future tool (out of scope here).

## 1. Problem

~20 jurisdictions in the [state statutes & rules plan](2026-07-01-state-statutes-rules-plan.md) publish authoritative law only as PDFs: nearly all blocked court rules (ME, RI, CT, DE, NC-app, VA, TX, OH, WI-SCR, AL, FL, DC, MS, MI-MRE, WA, GA-Sup/USCR), Kentucky's statutes, and the territories (Guam, CNMI, USVI rules, PR rules). Character extraction from text-based PDFs is solved by existing crates; **what does not exist anywhere in open source is the layer that turns a legal PDF into structured provisions** — one record per rule/section with citation, heading, body, and effective date. That layer is this tool.

## 2. Goals / non-goals

**Goals (v1):**
- Text-based PDFs → structured provisions (JSONL/JSON), deterministic and testable.
- Profile-driven: all jurisdiction knowledge lives in **TOML profiles (data), not code**. Adding a new document family = writing a profile, not forking the engine.
- Library API (typed, in-process, used by the cite-check ingester) + CLI binary (`cargo install pdf-untangle`), same crate.
- Publishable quality: docs.rs docs, README with examples, golden-file test suite, semver discipline.

**Non-goals (v1) — explicitly deferred:**
- OCR / scanned-image PDFs (v2; all verified Wave-3 targets embed real text).
- Translation (Spanish PR/other text is extracted as published — controlling text stays controlling).
- Headless-browser fetching (separate tool), PDF *fetching* at all — pdf-untangle reads local files/bytes; the ingester's crawlers do the downloading.
- Annotations/forms/tables-as-data. Tables render as text lines in v1.

## 3. Architecture

```
pdf-untangle (crate)
├── src/lib.rs          — public API
├── src/extract/        — Layer 1: PDF → positioned text runs (per page)
├── src/layout/         — Layer 2: runs → logical lines/blocks (reading order,
│                          columns, header/footer strip, hyphenation & page-flow merge)
├── src/structure/      — Layer 3: blocks → provisions (profile-driven heading
│                          detection, section splitting, history/date parsing)
├── src/profile.rs      — TOML profile schema + loader (built-in + user-supplied)
├── src/output.rs       — Provision model + JSON/JSONL serialization
├── src/bin/pdf-untangle.rs   — CLI (thin wrapper over the library)
└── profiles/*.toml     — built-in jurisdiction profiles (shipped with the crate)
```

**Layer 1 — extraction backend.** Primary: `pdfium-render` (Google PDFium, BSD-licensed, best-in-class text + position extraction; bundles or dlopens libpdfium). Fallback/pure-Rust: `lopdf`/`pdf-extract` behind a cargo feature (`backend-pdfium` default, `backend-pure` for no-native-dep builds). Both hide behind one internal `TextRun` trait so Layers 2-3 never know the backend. Decision gate in Phase 1: if pdfium's packaging friction is too high for a publishable crate, flip the default to pure-Rust and keep pdfium optional.

**Layer 2 — layout reconstruction (backend-agnostic, the hard 40%).**
Reading order from run positions; column detection; strip repeated page furniture (headers/footers/page numbers) by cross-page repetition + position; de-hyphenate line-break splits; merge paragraphs across page boundaries; preserve indentation levels (they carry meaning in rules: subdivisions (a)(1)(A)).

**Layer 3 — structure detection (profile-driven, the novel 40%).**
A profile declares, as data: **`jurisdiction` slug + `kind`** (the two resolution routing keys — e.g. `cnmi-rules` → `northern_mariana_islands` / `rule`; declared in the TOML so the mapping is data, never integration-code convention), heading patterns (regex + optional style hints like bold/size/centered), what starts a new provision, the **citation template** (`"Guam R. Civ. P. {num}"` — the single source of truth the resolution patterns must reproduce byte-for-byte; each PDF jurisdiction's template is **jointly signed off with the resolution agent before Phase 4**, since territory Bluebook abbreviations are non-obvious), history-line patterns and date formats (incl. Spanish month names for PR), stop markers (committee notes, appendices), skip blocks. The engine applies the profile to Layer-2 blocks and emits provisions. Unmatched-content accounting: every input line is either consumed or reported, so silent text loss is impossible.

**Output model** (superset of what cite-check's `load.rs` needs; ingester maps it to `ProvisionRec`):
```json
{"jurisdiction":"northern_mariana_islands","kind":"rule",
 "citation":"NMI R. Civ. P. 12","number":"12","heading":"Defenses and Objections",
 "text":"…","history":"…","effective_date":"2019-01-01","language":"en",
 "confidence":0.93,"confidence_basis":"coverage=99.4%; citation+date parsed clean",
 "source_file":"R05.pdf","pages":[11,14],"profile":"cnmi-rules","warnings":[]}
```
- `jurisdiction`/`kind` come from the profile (routing keys for `provisions` resolution — a wrong slug silently breaks resolution, hence declared data).
- `effective_date` is **never null**: real parsed date, else the document's stated adoption date, else the project's standard `1800-01-01` sentinel (matches the HTML scrapers; `provisions.effective_date` is NOT NULL and drives point-in-time lookups — the sentinel resolves as "in force now").
- `confidence`/`confidence_basis` map to the existing `provisions` columns (default 0.9): computed per provision from line-coverage %, clean-citation parse, and clean-date parse — so shakier PDF extractions surface as lower-confidence in the verifier instead of masquerading as clean HTML ones. (The example citation string is illustrative — the real form per jurisdiction is fixed at the Phase-4 template sign-off.)

**Library API (sketch):**
```rust
let profile = Profile::builtin("cnmi-rules")?;            // or Profile::from_path(...)
let doc = pdf_untangle::extract(bytes_or_path, &profile)?;       // -> Document
for p in doc.provisions { ... }                            // typed records
// low-level access: doc.pages (runs/blocks) for debugging + profile authoring
```

**CLI (sketch):**
```
pdf-untangle extract rules.pdf --profile cnmi-rules -o out.jsonl     # structured provisions
pdf-untangle text rules.pdf                                          # Layer-2 clean text (no profile)
pdf-untangle inspect rules.pdf --page 3                              # runs/blocks dump for profile authoring
pdf-untangle profiles                                                # list built-ins
pdf-untangle check rules.pdf --profile x                             # dry-run: coverage report, warnings, exit≠0 on loss
```

## 4. Testing strategy

- **Golden-file suite:** `tests/corpus/<profile>/{input.pdf, expected.jsonl}` — small real PDFs (or excerpted pages) committed to the repo; full-size PDFs cached by URL in CI. An `UPDATE_GOLDEN=1` regeneration flow.
- **Coverage invariant:** `pdf-untangle check` must report ≥98% line consumption on every corpus document; unconsumed lines listed.
- **Structure-integrity invariant** (complements coverage — catches mis-splitting that line-coverage can't see): `pdf-untangle check` also asserts citation-sequence sanity per document (monotonic rule/section numbering; flag gaps and duplicates, with a profile-declared allowlist for intentional gaps like HCR's reserved numbers) and, where the PDF has a TOC, cross-checks provision count + numbers against it.
- **Backend pinning:** the golden suite pins the exact extraction-backend version (pdfium build number / crate versions) in a committed manifest; pdfium run-ordering can shift across versions, so a dependency bump is a deliberate `UPDATE_GOLDEN` regen with a diff review, never a mystery CI failure.
- **Property tests** on Layer 2 (merge/hyphenation round-trips).
- **First corpus (in build order):** CNMI per-section PDFs (cleanest: one section per file, deterministic URLs) → Guam per-chapter GCA → Maine rules (`RULE N.pdf` per rule) → CT Practice Book (single 1.2MB book — stress test) → KY per-section statute PDFs → PR Spanish rules (language + Spanish dates).

## 5. Ingester integration (cite-check side, after v1)

Each blocked `rules_<state>.rs` / `statutes_kentucky.rs` stub becomes a real module: crawler downloads the PDFs (existing Pacer politeness), calls `pdf_untangle::extract` with that state's profile, maps `Provision` → `ProvisionRec`, writes the standard JSONL. Profiles for cite-check states live in pdf-untangle's built-ins (they're public-law knowledge — publishing them IS the point). Table cells flip ⛔ → 🧱 → 📥 per the normal pipeline; resolution agent gets the standard load-time pings.

## 6. Risks

- **pdfium packaging** (native lib) vs pure-Rust extraction quality — resolved by the Phase-1 decision gate + feature flags.
- **PDF diversity:** some court PDFs are Word-exports (clean), some are typeset (columns, odd run ordering). Layer 2 must be tuned against the real corpus early — hence corpus-first development.
- **Maintenance burden of a public crate** — keep the public API tiny (one entry point + types); profiles carry the churn.
- **Name/brand:** `pdf-untangle` reserved at first publish (Phase 0 does a `0.0.1` placeholder publish to hold the name).

## 7. Build phases

### Phase 0 — Scaffold & name (small)
- [ ] Create `~/code/pdf-untangle` repo (workspace: lib + bin in one crate), MIT/Apache-2.0 dual license files
- [ ] Push to `gitlab.com:john711/pdf-untangle`; basic CI (fmt, clippy, test)
- [ ] Publish `0.0.1` placeholder to crates.io to reserve `pdf-untangle` (needs `cargo login` — John's crates.io token)
- [ ] README skeleton: what/why, the profile concept, roadmap

### Phase 1 — Extraction core (Layer 1)
- [ ] `TextRun` model (text, bbox, font size/weight flags, page)
- [ ] `backend-pdfium` implementation (pdfium-render), feature-gated
- [ ] `backend-pure` implementation (lopdf/pdf-extract), feature-gated
- [ ] **Decision gate:** pick the default backend by comparing both on 3 corpus PDFs (CNMI section, Guam chapter, CT Practice Book); record the call here
- [ ] `pdf-untangle inspect` CLI subcommand (runs dump) — needed for everything after

### Phase 2 — Layout reconstruction (Layer 2)
- [ ] Reading order + line assembly from runs; column detection
- [ ] Page-furniture stripping (repeated headers/footers/page numbers)
- [ ] De-hyphenation + paragraph merge across page breaks
- [ ] Indentation-level preservation (subdivision structure)
- [ ] `pdf-untangle text` subcommand; property tests; golden Layer-2 outputs for 3 corpus docs

### Phase 3 — Profiles & structure (Layer 3)
- [ ] Profile TOML schema + loader (+ `Profile::builtin` embedding via include_dir) — schema includes `jurisdiction`, `kind`, `citation_template` as required fields
- [ ] Heading/provision-start detection (regex + style hints); provision assembly
- [ ] History-line + effective-date parsing (multi-format incl. Spanish months; never-null fallback chain: parsed date → document adoption date → `1800-01-01` sentinel)
- [ ] Per-provision `confidence`/`confidence_basis` computation (coverage % + clean citation parse + clean date parse)
- [ ] Coverage accounting + structure-integrity checks (sequence sanity, TOC cross-check) + warnings (`pdf-untangle check`)
- [ ] `Provision` output model + JSON/JSONL; `pdf-untangle extract` subcommand
- [ ] First two profiles green on goldens: `cnmi-rules`, `cnmi-statutes`

### Phase 4 — Corpus profiles, wave 1 (prove generality)
- [ ] **Citation-template sign-off with the resolution agent** for every wave-1 profile BEFORE building it (the one seam where the two halves can drift; territory abbreviations especially) — signed-off templates recorded in each profile TOML header
- [ ] `guam-gca` (per-chapter statutes) + `guam-rules`
- [ ] `maine-rules` (per-rule files)
- [ ] `connecticut-practice-book` (single-book stress test)
- [ ] `kentucky-krs` (per-section statute PDFs)
- [ ] `pr-rules` (Spanish; language tag + Spanish date parsing proven)
- [ ] Golden tests all green; coverage ≥98% each; profile-authoring guide in README

### Phase 5 — Publish v0.1
- [ ] API polish + rustdoc on all public items; examples/ dir
- [ ] README full: install, CLI walkthrough, library example, profile guide, corpus table
- [ ] CI release workflow (tag → test → `cargo publish`)
- [ ] Publish `pdf-untangle 0.1.0`; announce decision (blog/HN/r/rust) left to John

### Phase 6 — cite-check integration (in the cite-check repo, my lane)
- [ ] Add pdf-untangle dep to the ingester (git rev → crates.io version at 0.1)
- [ ] Flip CNMI + Guam: real `rules_*`/`statutes_*` modules (crawl PDFs → pdf-untangle → ProvisionRec) — first ⛔→🧱 conversions
- [ ] Flip the state-rules queue in corpus order: ME, RI, DE, VA, TX, OH, FL, AL, DC, NC-app, WI-SCR, MI-MRE, MS-rules, WA, GA-Sup/USCR, CT
- [ ] Flip `statutes_kentucky` (per-section KRS PDFs)
- [ ] PR rules (Spanish) — coordinate extractor patterns with the resolution agent (Spanish citation forms)
- [ ] Update the state plan table per state; loads + mirror pings per the normal handoff

### Phase 7 — v2 backlog (not scheduled)
- [ ] OCR feature (`backend-ocr`: tesseract or a Rust OCR) for scanned PDFs
- [ ] Table extraction as structured data
- [ ] USVI rules (needs the headless-browser tool first to enumerate the CMS ids — cross-tool dependency)

## 8. Coordination notes
- Lanes: pdf-untangle + its cite-check integration are **scraper-agent lane**. The resolution agent consumes the resulting provisions exactly as with HTML states (pattern from branch, mirror on ping). Relay this doc's existence + the Phase-6 state order to them so patterns can be pre-staged for the PDF states.
- **Reviewed by the resolution agent 2026-07-02 — approved**, with 6 suggestions all folded in above: profile-declared `jurisdiction`+`kind` routing keys (§3 Layer 3, output model), joint citation-template sign-off before Phase 4 (§3, Phase 4 gate), never-null effective_date with the corpus-standard 1800-01-01 sentinel (§3 output model, Phase 3), per-provision `confidence`/`confidence_basis` emission (§3 output model, Phase 3), structure-integrity checks beside line-coverage (§4, Phase 3), pinned extraction-backend version in the golden suite (§4).
- The **headless-browser capability** (unblocks MS/TN/AR/GA/PR/VI statutes, NM, LA rules, VT rules) remains a separate, undesigned tool — do not fold it into pdf-untangle.