Module crawl 

Crawl configuration and the small per-crawl records stored alongside pages.

Structs

CrawlConfig

CrawlLimits

Politeness

RobotsFile

robots.txt as fetched at the start of a crawl (kept for change detection).

SitemapSummary

What we keep about a site’s sitemaps (the URL list itself is not stored).

Enums

AddressPolicy

Whether the crawler may connect to private and internal addresses. The cloud uses Public; self-hosted, the CLI and local MCP use AllowPrivate.

Constants

USER_AGENT