Crate crw_renderer 

HTTP and headless-browser rendering engine for the CRW web scraper.

Provides a FallbackRenderer that fetches pages via plain HTTP and optionally re-renders them through a CDP-based headless browser when SPA content is detected.

• http_only — Simple HTTP fetcher using reqwest
• detector — Heuristic SPA shell detection (empty body, framework markers)
• cdp — Chrome DevTools Protocol renderer (LightPanda, Playwright, Chrome) (requires cdp feature)
• traits — PageFetcher trait for pluggable backends

Feature flags

Flag	Description
cdp	Enables CDP WebSocket rendering via tokio-tungstenite

Example

use crw_core::config::RendererConfig;
use crw_renderer::FallbackRenderer;
use std::collections::HashMap;

use crw_core::config::StealthConfig;
let config = RendererConfig::default();
let stealth = StealthConfig::default();
let renderer = FallbackRenderer::new(&config, "crw/0.1", None, &stealth)?;
let deadline = crw_core::Deadline::from_request_ms(8000);
let result = renderer.fetch("https://example.com", &HashMap::new(), None, None, None, deadline).await?;
println!("status: {}", result.status_code);

Modules

blocklist

Resource blocklist for CDP Fetch.requestPaused interception.

breaker

Sliding-window circuit breaker with multi-probe half-open and linear-back-off cooldown. See plans/breaker-cascade-fix.md Iter 3.

detector

host_limiter

Process-wide per-host rate-limiter and concurrency cap.

http_only

preference

Per-host renderer preference learning.

traits

Structs

FallbackRenderer

Composite renderer that tries multiple backends in order.

ScreenshotReq

Per-request screenshot capture parameters. Carried via a task-local rather than the PageFetcher::fetch signature (mirrors REQUEST_PROXY) so the trait + its ~30 call sites stay untouched. Some ⇒ capture a PNG via CDP Page.captureScreenshot after the wait window; None ⇒ no screenshot.

Statics

REQUEST_COUNTRY

Per-request country code (ISO 3166-1 alpha-2, lowercase) for the chrome_proxy tier’s CDP auth pump. Set by FallbackRenderer::fetch when a ScrapeRequest.country is present; read in cdp.rs while composing DataImpulse credentials. Task-local so child tasks spawned by the pool inherit it without trait-signature churn.

REQUEST_PROXY

Resolved proxy entry for the current request, picked from the active rotator by host. Set by the scrape/crawl entry points (via FallbackRenderer::pick_proxy); read in cdp.rs to drive the per-request Chrome proxyServer (a fresh proxied browser context) and the Fetch.authRequired pump. None = no proxy → existing behaviour.

REQUEST_SCREENSHOT

Resolved screenshot request for the current scrape. Set by the scrape/crawl entry point ([crw_crawl::single::scrape_url]) when formats contains Screenshot; read in cdp.rs to drive Page.captureScreenshot and in FallbackRenderer::fetch to force the vanilla-Chrome CDP path. None = no screenshot → existing behaviour.

Functions

current_screenshot_req

The resolved screenshot params for the current task, if any.

screenshot_requested

Whether a screenshot was requested for the current task (reads the REQUEST_SCREENSHOT task-local). false when unset / outside a scope.