Module redact 

Opt-in PII redaction of a DoclingDocument (#621).

A converted document feeds search and RAG pipelines — Markdown, chunks, embeddings, vector stores, LLM prompts — and personal data in the source flows into every one of them. Redacting each export format after the fact misses the text that lives elsewhere in the model: the item tree behind the JSON export, the link table, a code block’s orig, the key-value graph’s cells, a caption’s hyperlink, the pixels of an embedded image. This pass runs once, on the model, before any serializer, chunker or stream reads it, replaces every detected span in place with a placeholder and returns a RedactionReport of counts — never the original values (unless the caller asks for the mapping explicitly).

Detection is pluggable through PiiDetector. The built-in PatternDetector is pure Rust and compiles for wasm: e-mail, phone, Luhn-validated card numbers, mod-97-validated IBANs, IPv4/IPv6, URL credentials and a few national IDs with their own check digits (US SSN, UK NINO, Indian Aadhaar — Verhoeff), plus the caller’s own patterns and deny terms. A name/organization/location detector (NER) lives in the docling crate behind its ner feature and plugs into the same trait; several detectors compose through CompositeDetector, overlapping spans resolved longest-first.

Python docling has no redaction stage, so this is a docling.rs extension (recorded in docs/MIGRATION.md); it is off by default and, unused, changes nothing.

What is walked: every user-visible string of the flat nodes, recursively (groups, furniture wrappers, picture children, rich table cells), the item tree (tree), the link table, code text and orig, a formula’s orig (the LaTeX is left alone — it is a rendering, not text, and the pipeline’s own orig is what a reader could recover PII from), tables (grid, first-class cells, rich-cell blocks), field regions, key-value cells, comments, hrefs, VTT cue voices. An inline group is matched on the concatenation of its runs and the spans mapped back onto the runs. The document’s name (the file name) is not touched. Image handling is ImageRedaction: Drop removes every embedded image and page render here; BoxOut needs the OCR models and is the docling crate’s job (this pass then leaves the pixels alone, as Keep does).

Non-goals: no compliance guarantee (GDPR/HIPAA/PCI), no rewriting of the source file, no cross-node entity detection, no pseudonym persistence across documents.

Structs

AllowList

What is never redacted: the allow terms (case-insensitive, the whole span) and the allow patterns (a regex the whole span matches). Built once per pass from the options and asked about every span, whichever detector produced it.

CompositeDetector

Several detectors as one; their spans are merged by the pass’s overlap rule (longest first, then kind precedence).

CustomPattern

A caller-supplied pattern: name is the placeholder label ([CASE_ID] for case_id), regex the expression (the regex crate’s syntax — no look-around); the whole match is the span, or capture group 1 when the expression has one.

PatternDetector

The built-in detector: patterns with check-digit validation where the format has one, so a random 16-digit order number is not a card and 2024-01-15 is not a phone.

RedactionError

A rejected option (an invalid custom regex).

RedactionOptions

The pass’s settings. Default = every kind, labels, no custom patterns, images dropped, no mapping.

RedactionReport

What was redacted: one count per label (EMAIL, PHONE, a custom pattern’s name, REDACTED for deny terms), the total, and — only with RedactionOptions::return_mapping — the original → placeholder pairs. A backend that also builds docling’s item tree (DOCX, HTML) has its text redacted in both representations but counted once.

Redactor

The pass’s state across one document: the pseudonym numbering and the report. DoclingDocument::redact drives it over a whole document; the streaming converter drives it over each page batch so the Markdown it emits matches the buffered export byte for byte.

Span

One detected span: byte offsets into the text it was detected on, the kind, the detector’s confidence (1.0 for a pattern) and, for a custom pattern, its name (the label).

Enums

ImageRedaction

What happens to embedded images (pictures, page renders) when the pass runs. An image can carry anything the text does — a scanned letter, a screenshot of a profile — and the pass cannot read it without OCR.

PiiKind

The kinds of personal data a detector can report. The order is the precedence when two detectors’ spans overlap at equal length — the check-digit formats (card, IBAN, national ID) over a phone number, whose digit groups they also look like.

Replacement

What a detected span becomes.

Traits

PiiDetector

A span detector. detect returns the spans of one text; the pass calls it once per string of the document, so a detector keeps no per-document state (pseudonym numbering is the pass’s own).

Functions

iban_valid

ISO 13616 mod-97 check over a compact (no spaces) IBAN.

luhn_valid

Luhn (ISO/IEC 7812-1) check over a digit string.

verhoeff_valid

Verhoeff check (the Aadhaar check digit).