unrefine
A lossless, idempotent markdown shape-refinement pass for the un*
document-extraction family (unpdf,
undoc,
unhwp).
unrefine is not a general markdown formatter, and it is not cleanup (the
noise-removal stage each renderer above already ships). cleanup deletes —
page-number lines, repeated headers/footers, mojibake. unrefine never
deletes: it enforces two invariants across every pass it runs.
- Lossless — every visible text token present before a pass is still present after it. A pass may add text (a recovered table separator row, for instance), but never remove any.
- Idempotent —
refine(refine(x)) == refine(x).
Because of those two invariants, unrefine composes with a renderer's
existing cleanup stage in either order, and never changes output that has
already shipped to a consumer.
Install
[]
= "0.1"
Requires Rust 1.87. Builds for wasm32-unknown-unknown with no feature flags.
Status
0.1.0 implements all five planned passes:
table shape normalization (missing GFM separator-row recovery, and
widening a table's header when a body row has more cells than it),
ordered-list renumbering (out-of-order or repeated list numbers
normalized to CommonMark's own sequential rule), link/image path
normalization (backslashes converted to forward slashes, duplicate
slashes collapsed — real URLs and protocol-relative references are left
untouched), frontmatter normalization (canonical key: value
spacing, and dropping an entry or list item that exactly repeats an
earlier one — key order is left as authored), and section anchors (a
GitHub-compatible slug assigned to every heading via CommonMark's own
heading-attributes extension — a heading that already has one is left
alone).
let output = refine;
Before / after, table missing its separator row:
| a | b | | a | b |
| 1 | 2 -> | --- | --- |
| 1 | 2 |
Before / after, out-of-order list numbers:
1. one 1. one
5. two -> 2. two
3. three 3. three
Before / after, a Windows-style image path:
 -> 
Before / after, a duplicate frontmatter entry:
title :"x" title: "x"
title: "x" ->
Before / after, a heading with no anchor:
## Getting Started -> ## Getting Started { #getting-started }
License
MIT