pdf2 1.1.0

A PDF 2.0 (ISO 32000-2) reader, writer and text extractor.
Documentation
# How a PDF claim is proven here

The ADR (`docs/adr/0001-differentiators.md`) says a feature without a fixture and
an oracle is not done. This file defines what that means: which corpora we use
and under which licence, which engines are the oracles, which metrics decide, and
how a tolerance is set without inventing a number. It is the contract; the
corpus itself arrives in Q4 and the first calibration in Q4 too.

## Three tiers of evidence

| Tier | What it is | What it can prove | Where it lives |
| --- | --- | --- | --- |
| **Fixtures with arithmetic** | `tests/fixtures/*.pdf`, generated by `make_fixtures.py`: `calibration.pdf` (11 pages), `book.pdf` (book prose in an embedded serif), `paths.pdf`, `images.pdf` | that a rule is implemented the way the specification says — pixel values are computed, not blessed | committed; documented in `docs/RENDERING.md` |
| **Tier A corpus** | a small, curated set of third-party files covering constructs we do not generate: CJK/Identity-H, Arabic CID, Type 3, CID-CFF, `/ToUnicode`, JPX, CCITT, CMYK JPEG, shading/pattern/transparency | that real files with those constructs open, draw and extract | committed under `corpus/tier-a/` with a manifest and licence texts |
| **Tier B corpus** | whole upstream trees, pinned by commit and fetched in CI, cached | robustness and scale on hundreds of files, run nightly, not per pull request | not committed |

## Candidate corpora, licences and sizes

Checked 2026-11-14. Sizes are the sum of the directory listing from the GitHub
API, not estimates.

| Corpus | Licence | Size / files | Use | Notes |
| --- | --- | --- | --- | --- |
| this repository's fixtures | Unlicense OR MIT | ~15 KB | committed | generated, zero licence risk, no real-world constructs |
| [pdf.js `test/pdfs`](https://github.com/mozilla/pdf.js/tree/master/test/pdfs) | Apache-2.0 (repo metadata) | 79 645 896 B / 690 PDFs | **Tier B**, and **Tier A** as a curated subset | includes `90ms_rksj_h_sample.pdf` (CJK, 1 KB), `ArabicCIDTrueType.pdf` (38 KB), `cid_cff.pdf` (19 KB), `IdentityToUnicodeMap_charCodeOf.pdf` (<1 KB), `bug_jpx.pdf` (5 KB), `ccitt_EndOfBlock_false.pdf` (3 KB), `cmykjpeg.pdf` (365 KB), Type 3 and shading files of 2–4 KB. No per-file provenance beyond the repository licence; the manifest records name, sha256 and upstream commit. |
| [veraPDF corpus](https://github.com/veraPDF/veraPDF-corpus) | **CC BY 4.0** (stated in its README) | 74 427 KB | Tier B | PDF/A-1–4, PDF/UA, ISO 32000-1; atomic, self-documenting files. Attribution required wherever results are published. |
| [PDFBox `src/test/resources/input`](https://github.com/apache/pdfbox/tree/trunk/pdfbox/src/test/resources/input) | Apache-2.0 (repo metadata) | 1 921 581 B / 40 PDFs | optional Tier A addition | small and broad; structural rather than visual |
| [qpdf test files](https://github.com/qpdf/qpdf) | Apache-2.0 | 43 MB repository | not chosen | mostly structural, overlapping veraPDF, and its corpus is not a separate directory |
| `mozilla/pdf.js.comparator` | README declares `SPDX-License-Identifier: MIT`; GitHub reports NOASSERTION | n/a | methodology only | not a corpus: it is the oracle set and diff-metric reference |

**Decision.** Tier A = our fixtures plus a curated subset of pdf.js files (target
under 1 MB) and, if the Q4 run shows it earns its 1.9 MB, the PDFBox input set.
Tier B = the full pdf.js `test/pdfs` and the veraPDF corpus, pinned by commit and
fetched in CI. Everything committed carries the upstream licence text and the
attribution in `corpus/LICENSES/`; everything fetched is pinned to a commit SHA
and cached by that SHA.

**Packaging consequence.** The corpus is repository evidence, not shipped code:
`corpus/` must be added to `exclude` in the root `Cargo.toml` so the published
`pdf2` crate does not grow by megabytes of third-party PDFs.

## Oracles

| Oracle | How it runs here | Role |
| --- | --- | --- |
| poppler (`pdftoppm`, splash) | native CLI, package `poppler-utils` | in the gate |
| mupdf (`mutool draw`) | native CLI, package `mupdf-tools` | in the gate |
| Ghostscript (`gs`) | native CLI | in the gate when installed |
| pdfium | C library; the comparator runs it as wasm | manual cross-check only — no C dependency in CI |
| pdf.js | Node | manual cross-check only |
| PDFBox | Java | manual cross-check only |

The comparator runs seven engines and six image diffs (pixelmatch, Resemble,
SSIM, butteraugli, DSSIM, FLIP). We borrow its *method* — several independent
oracles plus a ranked diff — not its toolchain: three native CLIs cover the CI
case without Java, wasm or a C library.

## Metrics and how a tolerance is set

Raster comparisons happen at **144 DPI**, one page at a time. Three numbers are
recorded per page:

1. **page geometry** — page count and page size in points after `/Rotate`; must
   match exactly.
2. **text** — extracted text compared page by page (exact for fixtures,
   whitespace-normalised for corpus files).
3. **pixels** — fraction of pixels differing by more than 32/255 in any channel,
   and mean absolute channel error, against each oracle.

Tolerances are **derived from oracle disagreement, not decreed**:

- First measure the spread *between the oracles* on the same page — pdfium,
  poppler, mupdf and Ghostscript do not agree pixel-for-pixel, and
  [JBIG2 decoders disagree visibly](https://nico.github.io/jbig2-bench/).
- Our tolerated distance from a page is the largest distance any two oracles show
  on that page, plus 10%. Where the oracles disagree, a renderer cannot be
  wrong by more than they are.
- **Fallback bound until calibration lands**: 1% of pixels beyond 32/255 at
  144 DPI against the median oracle, and no page may differ in geometry or text
  at all. The Q4 calibration replaces this with measured numbers per construct.

A file outside its tolerance is a **finding**, recorded in `docs/CORPUS.md` with
the three numbers and the oracle spread — never re-blessed silently. A golden
image in `tests/fixtures/golden/` is a regression net, not a correctness proof;
the correctness claim is the arithmetic fixture or the oracle comparison.

## The procedure

```sh
# structure and text, one file at a time (today's CLI: info, pages, text, check)
cargo run -q -p pdf2-cli -- check corpus/tier-a/<file>.pdf
cargo run -q -p pdf2-cli -- text  corpus/tier-a/<file>.pdf

# one page to PNG, today through the renderer's golden binary
cargo run -q -p pdf2-render --bin pdf2-render-golden -- \
    --fixture corpus/tier-a/<file>.pdf --page 1 --scale 2 --bless --out /tmp/ours.png

# the oracles
pdftoppm -r 144 -png -f 1 -l 1 corpus/tier-a/<file>.pdf /tmp/poppler
mutool draw -r 144 -o /tmp/mupdf.png corpus/tier-a/<file>.pdf 1
```

The CLI has no `render` subcommand yet; if the Q4 batch run needs one, it is
added there rather than documented here as if it existed.

## What this does not prove

- It does not prove the absence of bugs outside the corpus, and it says nothing
  about pages the corpus does not contain.
- It does not cover print-precision behaviour — CMYK blending, overprint,
  knockout groups — which `hayro` excludes too and Acrobat alone implements.
- Third-party corpus files are artefacts with their own licences: the manifest
  and `corpus/LICENSES/` are the attribution record, and results published from
  the veraPDF corpus must credit it (CC BY 4.0).
- No user files: the proof set is public and redistributable, by construction.

## Status

- Committed: this strategy, the three tiers, the licences, the oracle set, the
  metric definitions and the calibration rule.
- Committed (Q4): Tier A — 23 files, 827 249 bytes, hashed in
  `corpus/manifest.json`, with the Apache-2.0 text in `corpus/LICENSES/`, and
  `corpus/` excluded from the crate.
- **First run stopped at the door**: all 23 files were rejected at the header
  (`malformed only PDF 2.x is supported`), which is itself the finding — see
  [`docs/CORPUS.md`](CORPUS.md).
- **Second run, after opening the gate** (`pdf2 1.1.0`, `cargo xtask corpus`):
  21 of 23 parse and render, 14 are fully readable, and the nine failures fall
  into three classes — composite (Type 0) extraction on 7 files, one xref edge
  case, one 1.x-era encryption case. The measurement is reproducible with
  `cargo xtask corpus`.
- Oracle calibration is still pending: the files reach a rasteriser now, but the
  oracle spread has not been measured yet (no poppler/mupdf on this machine;
  the CI job that runs them is part of the roadmap).