unrefine 0.1.0

Lossless, idempotent markdown shape-refinement pass for the un* document extraction family
Documentation

unrefine

crates.io docs.rs CI license

A lossless, idempotent markdown shape-refinement pass for the un* document-extraction family (unpdf, undoc, unhwp).

unrefine is not a general markdown formatter, and it is not cleanup (the noise-removal stage each renderer above already ships). cleanup deletes — page-number lines, repeated headers/footers, mojibake. unrefine never deletes: it enforces two invariants across every pass it runs.

  • Lossless — every visible text token present before a pass is still present after it. A pass may add text (a recovered table separator row, for instance), but never remove any.
  • Idempotent — refine(refine(x)) == refine(x).

Because of those two invariants, unrefine composes with a renderer's existing cleanup stage in either order, and never changes output that has already shipped to a consumer.

Install

[dependencies]
unrefine = "0.1"

Requires Rust 1.87. Builds for wasm32-unknown-unknown with no feature flags.

Status

0.1.0 implements all five planned passes: table shape normalization (missing GFM separator-row recovery, and widening a table's header when a body row has more cells than it), ordered-list renumbering (out-of-order or repeated list numbers normalized to CommonMark's own sequential rule), link/image path normalization (backslashes converted to forward slashes, duplicate slashes collapsed — real URLs and protocol-relative references are left untouched), frontmatter normalization (canonical key: value spacing, and dropping an entry or list item that exactly repeats an earlier one — key order is left as authored), and section anchors (a GitHub-compatible slug assigned to every heading via CommonMark's own heading-attributes extension — a heading that already has one is left alone).

let output = unrefine::refine(markdown, &unrefine::RefineOptions::default());

Before / after, table missing its separator row:

| a | b |          | a | b |
| 1 | 2       ->    | --- | --- |
                    | 1 | 2 |

Before / after, out-of-order list numbers:

1. one             1. one
5. two        ->   2. two
3. three           3. three

Before / after, a Windows-style image path:

![x](img\sub\a.png)   ->   ![x](img/sub/a.png)

Before / after, a duplicate frontmatter entry:

---                  ---
title  :"x"          title: "x"
title: "x"      ->
---                  ---

Before / after, a heading with no anchor:

## Getting Started   ->   ## Getting Started { #getting-started }

License

MIT