Expand description
docs.rs operations: readme, get-item, search-in-crate.
§Extraction posture (Rules Rust — web scraping / data extraction)
- Transport is HTTP GET (
HttpClient); this module only parses bytes. - Structure uses
scraper(html5ever) + CSS selectors — never regex for field extraction from HTML. - Sanitization may use process-static regex only to strip
on*handlers andjavascript:URLs after DOM script/style removal (XSS hygiene, not content discovery). - Markdown conversion via
htmdon the sanitized fragment. - Provenance: every success payload includes
source_url(final URL). - Encoding: UTF-8 only after BOM strip (
crate::http::decode_utf8); docs.rs / doc.rust-lang.org ship UTF-8.encoding_rsmulti-charset is OOS. - Not a crawl: no link frontier, no sitemap, no RSS/Atom, no robots meta index policy (agent one-shot read, not dataset indexing).
§Workload classification (Rules Rust — parallelism)
- I/O stage: HTTPS GET via
HttpClienton multi-thread Tokio workers. - CPU stage: HTML scrape + markdown conversion via
[
ConcurrencyBudget::run_cpu_bound] (spawn_blocking+ Semaphore). - Hit scan: large
all.htmlcandidate lists userayonafter a size threshold; small lists stay sequential to avoid pool overhead.
Structs§
- GetItem
Data - Typed item documentation payload.
- Readme
Data - Crate overview (rustdoc crate docblock, not git README).
- Search
InCrate Data - search-in-crate result set (
truncatedis true whentotal > emitted). - Search
InCrate Hit - Single all.html hit.
Functions§
- all_
html_ url - Build all.html index URL.
- all_
html_ url_ on_ origin - Build all.html URL against a custom origin (wiremock tests).
- extract_
item_ markdown_ from_ html - Extract get-item markdown from raw HTML (offline tests / pure path).
- extract_
method_ markdown_ from_ html - Extract markdown for a single associated method by rustdoc
id="method.X". - extract_
method_ markdown_ scoped - Like
extract_method_markdown_from_htmlbut reports extraction scope. - extract_
readme_ markdown_ from_ html - Extract readme markdown from raw HTML (offline tests / pure path).
- fetch_
item - Fetch and convert a typed rustdoc item page (production docs.rs).
- fetch_
item_ at - Fetch item from a prebuilt URL (wiremock).
- fetch_
item_ on_ origin - Fetch item against a configurable origin (offline mocks).
- fetch_
readme - Fetch and convert crate overview docblock (production docs.rs).
- fetch_
readme_ at - Fetch readme from a prebuilt URL (wiremock).
- fetch_
readme_ on_ origin - Fetch readme against a configurable origin (offline mocks).
- get_
item_ url - Build rustdoc item or module URL.
- get_
item_ url_ on_ origin - Build get-item URL against a custom origin (wiremock tests).
- get_
item_ url_ on_ origin_ with_ parent_ kind - Like
get_item_url_on_originbut forces the parent type kind for methods. - join_
href - Join relative/absolute/full href against crate rustdoc base.
- parse_
all_ html_ hits - Pure parse of all.html body for offline tests and CPU workers.
- readme_
url - Build crate index URL with rustc hyphen→underscore segment.
- readme_
url_ on_ origin - Build readme URL against a custom origin (wiremock tests).
- sanitize_
html_ fragment - Remove script/style nodes and on* handlers before Markdown conversion.
- scrub_
rustdoc_ chrome - Strip rustdoc UI chrome that pollutes LLM context (
§, “Copy item path”, …). - search_
in_ crate - Parse all.html and filter symbols (production docs.rs).
- search_
in_ crate_ at - search-in-crate against a prebuilt all.html URL (wiremock).
- search_
in_ crate_ from_ html - Build SearchInCrateData from HTML body without network (offline tests).
- search_
in_ crate_ on_ origin - search-in-crate against a configurable origin (offline mocks).