Crate oratos_html 

HTML loading, parsing, and page extraction.

Structs

CrawlOptions

Heading

HtmlPage

ImageInfo

JsonLdBlock

LinkInfo

LoadOptions

Functions

disallowed_paths

Paths or prefixes disallowed for a given user-agent block (simplified: merges all Disallow).

is_disallowed

Returns true if url_path should not be fetched per robots disallow rules.

load_pages

normalize_url

parse_html

resolve_internal_path

urls_from_sitemap_xml

Collect page URLs from a sitemap XML document (urlset or sitemap index, one level).