Skip to main content

Module parse

Module parse 

Source
Expand description

HTML → denoised Markdown units.

extract is the whole parsing story: parse once with tl, index the arena, pick the content root, and write Markdown while dropping boilerplate. It never touches the network and never allocates a second copy of the document, so it is cheap enough to run over a whole result set in parallel via extract_many.

Structs§

Article
A cleaned document.
ExtractOptions
Knobs for extract.
Link
One harvested link.
Meta
Everything we can learn about a page without reading its body.
RenderOptions
Options that change what ends up in the Markdown.
Unit
One addressable block of an article.

Enums§

ArticleKind
Whether a page turned out to be an article or a listing.
IndexMode
When to fall back to index-page extraction.
UnitKind
What kind of block a Unit came from.

Functions§

extract
Turn a raw HTML document into denoised Markdown units.
extract_many
Extract many documents in parallel across the rayon pool.
extract_with
extract with explicit options.