Module html_crush 

Deterministic HTML content extractor — extracts article/main content from web pages, converts to clean markdown, discards boilerplate (#1124).

Web pages fetched by agents (documentation, issue trackers, Stack Overflow) contain ~90% non-informational tokens (navigation, ads, scripts, footers). This module extracts only the meaningful article content and converts it to markdown — the format agents work best with.

Determinism (#498): output is a pure function of the input HTML — no timestamps, counters, or randomness. Same HTML always produces same markdown.

Structs

ArticleMeta

Article metadata found in document <meta>, <link>, and <time> nodes.

CodeBlock

A code block extracted from the selected article body.

CrushResult

ExtractionResult

Deterministic article extraction result.

Constants

KEEP_DATA_DIVISOR

Functions

crush_if_beneficial

extract_article

extract_article_content

Extract selected article content, metadata, and sacred code blocks.

is_html_content