Module scrape_local 

Local scrape/crawl/map/search/parse (HTTP + files; one-shot). Local scrape/crawl/map/search/parse (one-shot HTTP and file extract; no SaaS).

Engines:

• http — reqwest + scraper HTML (no Chrome)
• browser — chromiumoxide via crate::browser::OneShotSession

Workload

• HTTP engine: I/O-bound network + CPU HTML parse off-async. Fan-out via tokio::task::JoinSet + Arc<Semaphore>::acquire_owned gated by crate::concurrency::effective_limit (or per-command --concurrency, 0 = process budget). Hard cap crate::concurrency::HARD_CAP. Parse uses spawn_blocking (docsrs: CPU work must not starve Tokio workers; bound by the same permit).
• Browser engine: I/O-bound CDP (subprocess Chrome). Sequential per shared session (product law one residual); use --engine http for fan-out.
• Crawl: BFS with bounded parallel frontier (JoinSet + budget); link discovery stays inside the permit; abort_all at limit; cancel token.
• Shared crate::robots::shared_http_client avoids per-request Client build.
• Compiled Regex patterns live in OnceLock (never recompile per call).

Structs

ScrapeOpts

Shared scrape options.

Enums

ScrapeFormat

Output formats for local scrape.

Constants

HTTP_USER_AGENT

Identifiable product User-Agent for HTTP scrapes (PRD politeness).

Functions

batch_scrape_http

Batch scrape N URLs (HTTP engine, bounded JoinSet + Semaphore fan-out).

build_scrape_payload

Build agent envelope data from HTML.

crawl_http

BFS crawl from seed URL (HTTP) with bounded parallel frontier.

map_http

Map site: collect unique URLs via BFS without full content (links format).

parse_file

Parse local file (html/md/txt/csv/json/xml/pdf/docx/xlsx) one-shot, no Chrome.

parse_file_opts

Parse local file with optional PII redaction.

read_urls_file

Read URLs file (one URL per line, # comments).

redact_pii

Redact common PII patterns in text (email, phone, card-like digits).

reject_non_http_scheme_for_http_engine

Reject file://, bare paths, and other non-HTTP(S) targets for the HTTP engine (GAP-A004).

scrape_http

HTTP static scrape (no Chrome).

search_http

Local search: fetch a public HTML search page or treat query as URL list seed. MVP: if query looks like URL, map it; else use DuckDuckGo HTML (optional network).

sorted_keys

Stable sorted map helper for tests.