Module normalize

Expand description

§Input Normalization

Preprocessing for markdown content before parsing. Handles invisible Unicode control characters (especially from copy-paste) that interfere with delimiter recognition, and HTML comment fences that would silently drop trailing text.

Double chevrons (<< and >>) are passed through unchanged.

§Why normalize bidi characters?

Unicode bidirectional formatting characters (LRO, RLO, LRE, RLE, etc.) are invisible and when placed adjacent to markdown delimiters like ** prevent parsers from recognizing them:

**bold** or <U+202D>**(1234**
            ^^^^^^^^ invisible LRO prevents second ** from being recognized as bold

These appear commonly when copying from web pages with mixed LTR/RTL content, PDFs, and word processors.

Functions§

fix_html_comment_fences: Inserts a newline after --> when followed by non-whitespace content.
normalize_document: Primary entry point for normalizing a crate::document::Document after parsing.
normalize_field_name: Normalize field name to Unicode NFC, so visually identical keys (e.g., composed "café" vs decomposed "cafe\u{0301}") are treated as equal.
normalize_markdown: Applies all markdown normalizations in order: CRLF → LF, bidi strip, HTML comment fence repair.
strip_bidi_formatting: Strips Unicode bidirectional formatting characters that can interfere with markdown parsing.