Module sitemap 

Sitemap parsing (streamed, gzip-aware) and discovery through sitemap indexes.

Parsing streams through gunzip and a lossy UTF-8 decoder, so memory stays flat however large a file is. Discovery is bounded three ways: MAX_SITEMAP_FILES fetch attempts (failures count), a URL cap, and a deadline.

Structs

SitemapDiscovery

SitemapDoc

Enums

SitemapError

SitemapKind

Constants

MAX_SITEMAP_FILES

Fetch attempts per discovery, successful or not.

Functions

discover

Fetches the seed sitemaps, follows indexes up to depth 2, and returns up to max_urls unique page URLs. Stops at deadline with what it has. With a limiter, every sitemap fetch waits for a permit first.

parse_sitemap