Skip to main content Crate web2md Copy item path Source Browser Minimal HTTP client: fetches raw HTML only.
No rendering engine—intentionally lightweight for MCP token efficiency. BrowserOptions Configuration for the HTTP client ConvertOptions Options for HTML → Markdown conversion. Feed A parsed RSS, Atom, or JSON Feed. McpRequest MCP tool request schema McpServer MCP server wrapping the Browser PageMetadata Metadata extracted from an HTML page.
Used by both the MCP server and the CLI --format json output. PageToMarkdown Convert raw HTML to clean Markdown.
Strips scripts, styles, and non-essential markup to minimize token output. content_fingerprint 64-bit simhash fingerprint of normalized text (hex). Stable for near-duplicate detection. detect_content_language Detect language from extracted text when HTML metadata has no language.
Returns an ISO 639-3 code using a lightweight stopword heuristic (no extra crate). extract_feed_links Extract feed URLs (RSS/Atom/JSON Feed) from HTML extract_metadata Extract metadata (title, description, author, publication date, image, headline, site name, keywords, categories, excerpt, canonical URL, language) from HTML. extract_page_metadata Extract metadata from HTML, then attach extraction quality and page type using Markdown. feed_to_markdown Convert a feed to Markdown (feed title as H1, each entry as H2 with metadata). language_matches True when actual language matches target (ISO 639-1 or 639-3, with region tags). normalize_crawl_url Resolve and normalize a link for crawl deduplication (drops fragment). parse_feed Parse RSS 2.0, Atom, or JSON Feed content into a FeedNone when the document is not a recognized feed. parse_sitemap_urls Parse URLs from sitemap XML content.
Extracts all <loc> tag values from sitemap.xml format. same_origin_links Returns crawl targets on the same origin as root, excluding blacklisted URLs. truncate_with_marker Truncate text to max bytes and append a [truncated] marker.