Skip to main content

Module docs_rs

Module docs_rs 

Source
Expand description

docs.rs operations: readme, get-item, search-in-crate.

§Extraction posture (Rules Rust — web scraping / data extraction)

  • Transport is HTTP GET (HttpClient); this module only parses bytes.
  • Structure uses scraper (html5ever) + CSS selectors — never regex for field extraction from HTML (version scrape uses path segments on a[href]).
  • Sanitization may use process-static regex only to strip on* handlers and javascript: URLs after DOM script/style removal (XSS hygiene, not content discovery).
  • Markdown conversion via htmd on the sanitized fragment.
  • Provenance: every success payload includes source_url (final URL). Hit URLs from all.html join against that source_url base (stdlib / mock / docs.rs) — never a hardcoded docs.rs host. Absolute hit hrefs must stay same-origin as source_url (SCRAPE-S-001); off-origin are skipped.
  • Encoding: UTF-8 only after BOM strip (crate::http::decode_utf8); docs.rs / doc.rust-lang.org ship UTF-8. encoding_rs multi-charset is OOS.
  • Not a crawl: no link frontier, no sitemap, no RSS/Atom.
  • robots.txt: PROIBIDO — this product never fetches, parses, or enforces REP (operator mandate + ADR 0003). One-shot allowlisted GETs only.

§Workload classification (Rules Rust — parallelism)

  • I/O stage: HTTPS GET via HttpClient on multi-thread Tokio workers.
  • CPU stage: HTML scrape + markdown conversion via ConcurrencyBudget::run_cpu_bound (spawn_blocking + Semaphore).
  • Hit scan: all.html candidate lists are scanned sequentially at every size. Per-item work is one URL resolve plus a string match — too small to amortize thread fan-out and join. A rayon path was benchmarked from 16 to 32768 candidates, lost at every size (best case 0.85x), and was removed.

Structs§

GetItemData
Typed item documentation payload.
MethodExtract
A located associated-item anchor plus its rendered markdown.
ReadmeData
Crate overview (rustdoc crate docblock, not git README).
SearchInCrateData
search-in-crate result set (truncated is true when hits were cut by --limit and/or by --max-output-bytes output budget).
SearchInCrateHit
Single all.html hit.

Enums§

AssocAnchorKind
Which rustdoc anchor family a Parent::member path resolves through.

Constants§

ASSOC_ANCHOR_MISS_PREFIX
Sentinel prefix of the “anchor absent on the parent page” error message.
METHOD_ANCHOR_PREFIXES
Anchor id prefixes for an associated function, in probe order.

Functions§

all_html_url
Build all.html index URL.
all_html_url_on_origin
Build all.html URL against an allowlisted origin (production or wiremock).
assoc_from_fragment
Recover the anchor family and bare member name from a planned URL fragment.
associated_item_path
Detect a parent-qualified associated item: Parent::member.
extract_item_markdown_from_html
Extract get-item markdown from raw HTML (offline tests / pure path).
extract_method_markdown_from_html
Extract markdown for a single associated method by rustdoc id="method.X".
extract_method_markdown_scoped
Like extract_method_markdown_from_html but reports scope and anchor id.
extract_readme_markdown_from_html
Extract readme markdown from raw HTML (offline tests / pure path).
fetch_item
Fetch and convert a typed rustdoc item page (production docs.rs).
fetch_item_at
Fetch item from a prebuilt URL (wiremock).
fetch_item_at_with_echo
Like fetch_item_at but echoes a requested kind string (e.g. method vs fn).
fetch_item_on_origin
Fetch item against a configurable origin (offline mocks).
fetch_item_on_origin_with_echo
Like fetch_item_on_origin with a wire item_type echo (method vs fn).
fetch_readme
Fetch and convert crate overview docblock (production docs.rs).
fetch_readme_at
Fetch readme from a prebuilt URL (wiremock).
fetch_readme_on_origin
Fetch readme against a configurable origin (offline mocks).
get_item_url
Build rustdoc item or module URL.
get_item_url_on_origin
Build get-item URL against a custom origin (wiremock tests).
get_item_url_on_origin_with_parent_kind
Like get_item_url_on_origin but forces the parent type kind for methods.
is_method_path
Detect associated method / inherent method path: Type::method where Type is UpperCamel and method starts lowercase (or kind forced as method via parse alias).
join_href
Join relative/absolute href against crate rustdoc base (same-origin).
list_assoc_anchor_names_from_html
Parse HTML and list one family’s anchor names (CPU-bound helper for ops recovery).
list_method_anchor_names
List rustdoc associated-method anchor names (method.X / tymethod.X → X).
list_method_anchor_names_from_html
Parse HTML and list method anchor names (CPU-bound helper for ops recovery).
method_segments_from_parent
Build method path segments from a resolved parent type path + method leaf.
parse_all_html_hits
Pure parse of all.html body for offline tests and CPU workers.
pick_unique_type_path
Pick a unique parent type path for Type::method short forms (e.g. Runtime → runtime::Runtime).
readme_url
Build crate index URL with rustc hyphen→underscore segment.
readme_url_on_origin
Build readme URL against an allowlisted origin (production or wiremock).
resolve_hit_url
Resolve an all.html href against base (source_url), same-origin only.
sanitize_html_fragment
Remove script/style nodes and on* handlers before Markdown conversion.
scrub_rustdoc_chrome
Strip rustdoc UI chrome that pollutes LLM context (§, “Copy item path”, …).
search_in_crate
Parse all.html and filter symbols (production docs.rs).
search_in_crate_at
search-in-crate against a prebuilt all.html URL (wiremock).
search_in_crate_from_html
Build SearchInCrateData from HTML body without network (offline tests).
search_in_crate_on_origin
search-in-crate against a configurable origin (offline mocks).
strip_assoc_anchor_prefix
Strip any prefix of prefixes from a rustdoc anchor id.
strip_crate_prefix_segments
Strip leading crate name / rustc root from item path segments (same rules as URL builder).
strip_method_anchor_prefix
Strip any METHOD_ANCHOR_PREFIXES entry from a rustdoc anchor id.