Module markdown_compact 

Deterministic Markdown/documentation compaction for LLM-agent reads.

Keeps heading topology intact, treats fenced code blocks as atomic units, and selects high-signal body units with a small IDF-style scorer. Intentionally std-only and byte-stable (#498): per-line token sets are ordered (BTreeSet), so the f64 score summation order — and therefore the selected lines and omission markers — are a pure function of the input bytes.

Functions

compact_markdown

Compact Markdown while preserving all headings, whole fenced code blocks, and high-signal details. Returns None when the document is too small, not markdown-shaped, or compaction would not actually shrink it.

has_markdown_headings

True when the document carries at least one ATX heading — the structural signal a plain .txt file must show before the lossy compactor may touch it (hyphen lists alone are not enough to call a text file “markdown”).