Expand description
docs.rs operations: readme, get-item, search-in-crate.
§Extraction posture (Rules Rust — web scraping / data extraction)
- Transport is HTTP GET (
HttpClient); this module only parses bytes. - Structure uses
scraper(html5ever) + CSS selectors — never regex for field extraction from HTML (version scrape uses path segments ona[href]). - Sanitization may use process-static regex only to strip
on*handlers andjavascript:URLs after DOM script/style removal (XSS hygiene, not content discovery). - Markdown conversion via
htmdon the sanitized fragment. - Provenance: every success payload includes
source_url(final URL). Hit URLs fromall.htmljoin against thatsource_urlbase (stdlib / mock / docs.rs) — never a hardcoded docs.rs host. Absolute hit hrefs must stay same-origin assource_url(SCRAPE-S-001); off-origin are skipped. - Encoding: UTF-8 only after BOM strip (
crate::http::decode_utf8); docs.rs / doc.rust-lang.org ship UTF-8.encoding_rsmulti-charset is OOS. - Not a crawl: no link frontier, no sitemap, no RSS/Atom.
- robots.txt: PROIBIDO — this product never fetches, parses, or enforces REP (operator mandate + ADR 0003). One-shot allowlisted GETs only.
§Workload classification (Rules Rust — parallelism)
- I/O stage: HTTPS GET via
HttpClienton multi-thread Tokio workers. - CPU stage: HTML scrape + markdown conversion via
ConcurrencyBudget::run_cpu_bound(spawn_blocking+ Semaphore). - Hit scan:
all.htmlcandidate lists are scanned sequentially at every size. Per-item work is one URL resolve plus a string match — too small to amortize thread fan-out and join. Arayonpath was benchmarked from 16 to 32768 candidates, lost at every size (best case 0.85x), and was removed.
Structs§
- GetItem
Data - Typed item documentation payload.
- Method
Extract - A located associated-item anchor plus its rendered markdown.
- Readme
Data - Crate overview (rustdoc crate docblock, not git README).
- Search
InCrate Data - search-in-crate result set (
truncatedis true when hits were cut by--limitand/or by--max-output-bytesoutput budget). - Search
InCrate Hit - Single all.html hit.
Enums§
- Assoc
Anchor Kind - Which rustdoc anchor family a
Parent::memberpath resolves through.
Constants§
- ASSOC_
ANCHOR_ MISS_ PREFIX - Sentinel prefix of the “anchor absent on the parent page” error message.
- METHOD_
ANCHOR_ PREFIXES - Anchor id prefixes for an associated function, in probe order.
Functions§
- all_
html_ url - Build all.html index URL.
- all_
html_ url_ on_ origin - Build all.html URL against an allowlisted origin (production or wiremock).
- assoc_
from_ fragment - Recover the anchor family and bare member name from a planned URL fragment.
- associated_
item_ path - Detect a parent-qualified associated item:
Parent::member. - extract_
item_ markdown_ from_ html - Extract get-item markdown from raw HTML (offline tests / pure path).
- extract_
method_ markdown_ from_ html - Extract markdown for a single associated method by rustdoc
id="method.X". - extract_
method_ markdown_ scoped - Like
extract_method_markdown_from_htmlbut reports scope and anchor id. - extract_
readme_ markdown_ from_ html - Extract readme markdown from raw HTML (offline tests / pure path).
- fetch_
item - Fetch and convert a typed rustdoc item page (production docs.rs).
- fetch_
item_ at - Fetch item from a prebuilt URL (wiremock).
- fetch_
item_ at_ with_ echo - Like
fetch_item_atbut echoes a requested kind string (e.g.methodvsfn). - fetch_
item_ on_ origin - Fetch item against a configurable origin (offline mocks).
- fetch_
item_ on_ origin_ with_ echo - Like
fetch_item_on_originwith a wireitem_typeecho (methodvsfn). - fetch_
readme - Fetch and convert crate overview docblock (production docs.rs).
- fetch_
readme_ at - Fetch readme from a prebuilt URL (wiremock).
- fetch_
readme_ on_ origin - Fetch readme against a configurable origin (offline mocks).
- get_
item_ url - Build rustdoc item or module URL.
- get_
item_ url_ on_ origin - Build get-item URL against a custom origin (wiremock tests).
- get_
item_ url_ on_ origin_ with_ parent_ kind - Like
get_item_url_on_originbut forces the parent type kind for methods. - is_
method_ path - Detect associated method / inherent method path:
Type::methodwhere Type is UpperCamel and method starts lowercase (or kind forced as method via parse alias). - join_
href - Join relative/absolute href against crate rustdoc base (same-origin).
- list_
assoc_ anchor_ names_ from_ html - Parse HTML and list one family’s anchor names (CPU-bound helper for ops recovery).
- list_
method_ anchor_ names - List rustdoc associated-method anchor names (
method.X/tymethod.X→X). - list_
method_ anchor_ names_ from_ html - Parse HTML and list method anchor names (CPU-bound helper for ops recovery).
- method_
segments_ from_ parent - Build method path segments from a resolved parent type path + method leaf.
- parse_
all_ html_ hits - Pure parse of all.html body for offline tests and CPU workers.
- pick_
unique_ type_ path - Pick a unique parent type path for
Type::methodshort forms (e.g.Runtime→runtime::Runtime). - readme_
url - Build crate index URL with rustc hyphen→underscore segment.
- readme_
url_ on_ origin - Build readme URL against an allowlisted origin (production or wiremock).
- resolve_
hit_ url - Resolve an all.html
hrefagainstbase(source_url), same-origin only. - sanitize_
html_ fragment - Remove script/style nodes and on* handlers before Markdown conversion.
- scrub_
rustdoc_ chrome - Strip rustdoc UI chrome that pollutes LLM context (
§, “Copy item path”, …). - search_
in_ crate - Parse all.html and filter symbols (production docs.rs).
- search_
in_ crate_ at - search-in-crate against a prebuilt all.html URL (wiremock).
- search_
in_ crate_ from_ html - Build SearchInCrateData from HTML body without network (offline tests).
- search_
in_ crate_ on_ origin - search-in-crate against a configurable origin (offline mocks).
- strip_
assoc_ anchor_ prefix - Strip any prefix of
prefixesfrom a rustdoc anchor id. - strip_
crate_ prefix_ segments - Strip leading crate name / rustc root from item path segments (same rules as URL builder).
- strip_
method_ anchor_ prefix - Strip any
METHOD_ANCHOR_PREFIXESentry from a rustdoc anchor id.