Crate mara 

mara is a scraper that fetches over a rotating pool of egress IPs, clearing bot-protection challenges along the way.

Today that means Cloudflare: mara solves the interactive challenge in a real headed browser once to bank a cf_clearance cookie, then serves every subsequent request to that host browser-free — a plain HTTP client replaying the cookie and user-agent. The browser is the fallback; the slim replay is the hot path. Routing is fully explicit: every host you fetch must be registered (exact match) as either a solve host (Cloudflare) or a raw one (fetched over the same rotating pool, never touching a browser). An unregistered host fails fast — mara never silently guesses a route.

Quick start

use futures::StreamExt;
use mara::{Client, Config, Domain};

let client = Client::new(Config {
    domains: vec![Domain::solve("example.com")],
    ..Default::default()
})
.await?;

// One result per input URL, in completion order. Bare URL strings work directly.
let mut results = client.fetch_all(["https://example.com/a", "https://example.com/b"]);
while let Some(item) = results.next().await {
    match item.result {
        Ok(page) => println!("{} → {} bytes", item.url, page.value.len()),
        Err(err) => eprintln!("{} failed: {err}", item.url),
    }
}

Client is the entry point; Client::fetch_all is the browser-free bulk path. See Config and Domain for routing (which hosts take the solve path) and Policy for timeouts, pacing, and probe tuning. Client::fetch_browser hands you a live headed chromiumoxide::Page when you need one; pair it with wait_full_load.

The public surface is deliberately small — the entry points above plus the doctor and store read-models. The orchestration internals (the exit pool, the per-exit workers, the introspection dashboard, the browser solver) are private: they’re the how, not the contract.

Modules

doctor

Startup diagnostics: environment checks (Xvfb, Chrome, the fingerprint triple, GPU) with a pass/warn/fail verdict. Backs the mara doctor command.

store

The persisted, read-only view: per-exit store::Stats and the clearance/cooldown state surfaced by Client::snapshot. These are report types — the store itself is internal.

Structs

Client

The entry point. Owns the exit pool, the per-exit serving workers, and the background maintainer that keeps the catalog warm. Cheaply cloneable (an Arc inside) — clone it to share one pool across tasks. Build one with Client::new; end a run with Client::shutdown.

Config

How a Client is built: the exit catalog, the browser/display knobs for the headed solver, timeouts, persistence, and the routing/pacing domains. Start from Config::default and override what you need. There is no client-concurrency dial — serving width is the exit count (one worker per exit).

Domain

A configured domain, matched by exact host — example.com does not cover www.example.com; register each host you fetch. Two orthogonal rate ceilings, both in requests-per-minute and both optional (compose freely):

FetchAll

The stream returned by Client::fetch_all: an unordered, completion-order stream of FetchResult, exactly one per input URL, that ends once every URL has settled.

FetchResult

One settled item of a FetchAll stream: the input slot it came from, the URL, the caller’s key (echoed from the input Resource, None if none was set), and the fetch result. index identifies the slot because URLs can repeat (e.g. --repeat) — it is assigned in strict input order, so an index → your-data map is always trivial even without a key. T is the body type: String for Client::fetch_all (text-decoded), Vec<u8> for fetch_all_bytes.

Outcome

A successful fetch: the body plus metadata about how it was obtained.

Policy

Tunables for retry/cooldown behaviour, the headed-solve deadlines, pacing, and probing. Read off Config::policy; Policy::default is sensible for most runs.

Resource

One thing to fetch: a URL plus optional request customization (method/headers/body, for non-HTML traffic like an Algolia POST) and an optional caller key echoed back on the result. From<&str>/From<String>/From<Url> give a bare GET with no key, so fetch_all(urls) still takes plain strings. Whether a resource is solved or fetched raw is decided by its host (the solve-set), never per-resource.

Enums

FetchError

Why a fetch gave up. A resource never fails on a winnable obstacle (rate-limit, block, timeout, a stale-clearance challenge on a registered host) — those retry forever across exits. These variants are the genuinely-unwinnable cases only.

Method

The HTTP method a Resource uses. A dependency-free mirror of the handful we need, so the public fetch API doesn’t leak wreq/http types. Defaults to Get (the HTML bulk path).

Reason

Why a fetch didn’t reach the origin — the payload of FetchError::GaveUp.

Functions

host_of

The host component of a URL, or None if it doesn’t parse or has no host. The same host derivation the router uses to match a URL against the configured Domains.

wait_full_load

Wait for a headed Page to finish loading — network to settle plus lazy content triggered by scrolling — bounded by budget, then return its HTML. Pair with Client::fetch_browser when you need the fully-rendered page.