Skip to main content

Module docs_rs

Module docs_rs 

Source
Expand description

docs.rs operations: readme, get-item, search-in-crate.

§Extraction posture (Rules Rust — web scraping / data extraction)

  • Transport is HTTP GET (HttpClient); this module only parses bytes.
  • Structure uses scraper (html5ever) + CSS selectors — never regex for field extraction from HTML.
  • Sanitization may use process-static regex only to strip on* handlers and javascript: URLs after DOM script/style removal (XSS hygiene, not content discovery).
  • Markdown conversion via htmd on the sanitized fragment.
  • Provenance: every success payload includes source_url (final URL).
  • Encoding: UTF-8 only after BOM strip (crate::http::decode_utf8); docs.rs / doc.rust-lang.org ship UTF-8. encoding_rs multi-charset is OOS.
  • Not a crawl: no link frontier, no sitemap, no RSS/Atom, no robots meta index policy (agent one-shot read, not dataset indexing).

§Workload classification (Rules Rust — parallelism)

  • I/O stage: HTTPS GET via HttpClient on multi-thread Tokio workers.
  • CPU stage: HTML scrape + markdown conversion via [ConcurrencyBudget::run_cpu_bound] (spawn_blocking + Semaphore).
  • Hit scan: large all.html candidate lists use rayon after a size threshold; small lists stay sequential to avoid pool overhead.

Structs§

GetItemData
Typed item documentation payload.
ReadmeData
Crate overview (rustdoc crate docblock, not git README).
SearchInCrateData
search-in-crate result set (truncated is true when total > emitted).
SearchInCrateHit
Single all.html hit.

Functions§

all_html_url
Build all.html index URL.
all_html_url_on_origin
Build all.html URL against a custom origin (wiremock tests).
extract_item_markdown_from_html
Extract get-item markdown from raw HTML (offline tests / pure path).
extract_method_markdown_from_html
Extract markdown for a single associated method by rustdoc id="method.X".
extract_method_markdown_scoped
Like extract_method_markdown_from_html but reports extraction scope.
extract_readme_markdown_from_html
Extract readme markdown from raw HTML (offline tests / pure path).
fetch_item
Fetch and convert a typed rustdoc item page (production docs.rs).
fetch_item_at
Fetch item from a prebuilt URL (wiremock).
fetch_item_on_origin
Fetch item against a configurable origin (offline mocks).
fetch_readme
Fetch and convert crate overview docblock (production docs.rs).
fetch_readme_at
Fetch readme from a prebuilt URL (wiremock).
fetch_readme_on_origin
Fetch readme against a configurable origin (offline mocks).
get_item_url
Build rustdoc item or module URL.
get_item_url_on_origin
Build get-item URL against a custom origin (wiremock tests).
get_item_url_on_origin_with_parent_kind
Like get_item_url_on_origin but forces the parent type kind for methods.
join_href
Join relative/absolute/full href against crate rustdoc base.
parse_all_html_hits
Pure parse of all.html body for offline tests and CPU workers.
readme_url
Build crate index URL with rustc hyphen→underscore segment.
readme_url_on_origin
Build readme URL against a custom origin (wiremock tests).
sanitize_html_fragment
Remove script/style nodes and on* handlers before Markdown conversion.
scrub_rustdoc_chrome
Strip rustdoc UI chrome that pollutes LLM context (§, “Copy item path”, …).
search_in_crate
Parse all.html and filter symbols (production docs.rs).
search_in_crate_at
search-in-crate against a prebuilt all.html URL (wiremock).
search_in_crate_from_html
Build SearchInCrateData from HTML body without network (offline tests).
search_in_crate_on_origin
search-in-crate against a configurable origin (offline mocks).