Module scrape_local 

Local scrape/crawl/map/search/parse (HTTP + files; one-shot). Local scrape/crawl/map/search/parse (one-shot HTTP and file extract; no SaaS).

Engines:

• http — reqwest + scraper HTML (no Chrome)
• browser — chromiumoxide via crate::browser::OneShotSession

Structs

ScrapeOpts

Shared scrape options.

Enums

ScrapeFormat

Output formats for local scrape.

Constants

HTTP_USER_AGENT

Identifiable product User-Agent for HTTP scrapes (PRD politeness).

Functions

batch_scrape_http

Batch scrape N URLs (HTTP engine, sequential by default).

build_scrape_payload

Build agent envelope data from HTML.

crawl_http

BFS crawl from seed URL (HTTP), limited by limit and max_depth.

map_http

Map site: collect unique URLs via BFS without full content (links format).

parse_file

Parse local file (html/md/txt/csv/json/xml/pdf/docx/xlsx) one-shot, no Chrome.

parse_file_opts

Parse local file with optional PII redaction.

read_urls_file

Read URLs file (one URL per line, # comments).

redact_pii

Redact common PII patterns in text (email, phone, card-like digits).

reject_non_http_scheme_for_http_engine

Reject file://, bare paths, and other non-HTTP(S) targets for the HTTP engine (GAP-A004).

scrape_http

HTTP static scrape (no Chrome).

search_http

Local search: fetch a public HTML search page or treat query as URL list seed. MVP: if query looks like URL, map it; else use DuckDuckGo HTML (optional network).

sorted_keys

Stable sorted map helper for tests.