Struct CrawlConfig 

pub struct CrawlConfig {

    pub max_depth: Option<usize>,
    pub max_pages: Option<usize>,
    pub max_concurrent: Option<usize>,
    pub respect_robots_txt: bool,
    pub soft_http_errors: bool,
    pub user_agent: Option<String>,
    pub stay_on_domain: bool,
    pub allow_subdomains: bool,
    pub include_paths: Vec<String>,
    pub exclude_paths: Vec<String>,
    pub custom_headers: HashMap<String, String>,
    pub request_timeout: Duration,
    pub rate_limit_ms: Option<u64>,
    pub max_redirects: usize,
    pub retry_count: usize,
    pub retry_codes: Vec<u16>,
    pub cookies_enabled: bool,
    pub auth: Option<AuthConfig>,
    pub max_body_size: Option<usize>,
    pub remove_tags: Vec<String>,
    pub content: ContentConfig,
    pub map_limit: Option<usize>,
    pub map_search: Option<String>,
    pub download_assets: bool,
    pub asset_types: Vec<AssetCategory>,
    pub max_asset_size: Option<usize>,
    pub browser: BrowserConfig,
    pub proxy: Option<ProxyConfig>,
    pub user_agents: Vec<String>,
    pub capture_screenshot: bool,
    pub follow_document_urls: bool,
    pub document_url_depth: Option<u32>,
    pub download_documents: bool,
    pub document_max_size: Option<usize>,
    pub document_mime_types: Vec<String>,
    pub warc_output: Option<PathBuf>,
    pub browser_profile: Option<String>,
    pub save_browser_profile: bool,
    pub ssrf: SsrfPolicy,
    pub dispatch: Option<DispatchProfile>,
    pub proxy_provider: Option<Arc<dyn ProxyProvider>>,
}

Configuration for crawl, scrape, and map operations.

Fields

max_depth: Option<usize>

Maximum crawl depth (number of link hops from the start URL).

max_pages: Option<usize>

Maximum number of pages to crawl.

max_concurrent: Option<usize>

Maximum number of concurrent requests.

respect_robots_txt: bool

Whether to respect robots.txt directives.

soft_http_errors: bool

When true, HTTP-level error responses (404 NotFound, 403 Forbidden, WAF blocks) are surfaced as ScrapeResult records with the matching status_code rather than raised as CrawlError. Default false preserves the historical throw-on-error contract for direct fetches. Independently of this flag, 404s reached at the end of a redirect chain are always surfaced softly — the user opted into redirect-following, so receiving a 404 there is part of the normal flow rather than an unexpected error.

user_agent: Option<String>

Custom user-agent string.

stay_on_domain: bool

Whether to restrict crawling to the same domain.

allow_subdomains: bool

Whether to allow subdomains when stay_on_domain is true.

include_paths: Vec<String>

Regex patterns for paths to include during crawling.

exclude_paths: Vec<String>

Regex patterns for paths to exclude during crawling.

custom_headers: HashMap<String, String>

Custom HTTP headers to send with each request.

request_timeout: Duration

Timeout for individual HTTP requests (in milliseconds when serialized).

rate_limit_ms: Option<u64>

Per-domain rate limit in milliseconds. When set, enforces a minimum delay between requests to the same domain. Defaults to 200ms when None.

max_redirects: usize

Maximum number of redirects to follow.

retry_count: usize

Number of retry attempts for failed requests.

retry_codes: Vec<u16>

HTTP status codes that should trigger a retry.

cookies_enabled: bool

Whether to enable cookie handling.

auth: Option<AuthConfig>

Authentication configuration.

max_body_size: Option<usize>

Maximum response body size in bytes.

remove_tags: Vec<String>

CSS selectors for tags to remove from HTML before processing.

content: ContentConfig

Content extraction and conversion configuration.

map_limit: Option<usize>

Maximum number of URLs to return from a map operation.

map_search: Option<String>

Search filter for map results (case-insensitive substring match on URLs).

download_assets: bool

Whether to download assets (CSS, JS, images, etc.) from the page.

asset_types: Vec<AssetCategory>

Filter for asset categories to download.

max_asset_size: Option<usize>

Maximum size in bytes for individual asset downloads.

browser: BrowserConfig

Browser configuration.

proxy: Option<ProxyConfig>

Proxy configuration for HTTP requests.

user_agents: Vec<String>

List of user-agent strings for rotation. If non-empty, overrides user_agent.

capture_screenshot: bool

Whether to capture a screenshot when using the browser.

follow_document_urls: bool

Re-enqueue discovered LinkType::Document URLs into the crawl frontier so the crawl follows links from document pages (PDFs, etc.) as it would from HTML pages. Default: false (documents terminate at materialisation).

document_url_depth: Option<u32>

Maximum document-depth (from the seed URL through document links only) when follow_document_urls is true. None means inherit max_depth. Independent of max_depth: a document URL is enqueued only if BOTH the outer max_depth and (if set) document_url_depth permit it.

download_documents: bool

Whether to download non-HTML documents (PDF, DOCX, images, code, etc.) instead of skipping them.

document_max_size: Option<usize>

Maximum size in bytes for document downloads. Defaults to 50 MB.

document_mime_types: Vec<String>

Allowlist of MIME types to download. If empty, uses built-in defaults.

warc_output: Option<PathBuf>

Path to write WARC output. If None, WARC output is disabled.

browser_profile: Option<String>

Named browser profile for persistent sessions (cookies, localStorage).

save_browser_profile: bool

Whether to save changes back to the browser profile on exit.

ssrf: SsrfPolicy

SSRF policy for outbound network requests. Default: deny private networks, allow http/https only, max 5 redirects.

Phase 1: deny_private and max_redirects are exposed to all language bindings. allowlist is skipped (see SsrfPolicy fields) and will be added in a follow-up when HostMatcher’s tagged-enum FFI form is decided.

dispatch: Option<DispatchProfile>

Pluggable dispatch components: bypass provider, escalation strategy, retry policy, WAF classifier, domain state, escalation budget, and max_total_attempts.

When None, the engine uses its built-in defaults (no bypass, BrowserOnly strategy, SimpleRetryPolicy, built-in WAF classifier, no domain state, unlimited budget, 10 total attempt cap).

Rust-only advanced field. Generated language bindings do not expose pluggable dispatch components; language clients use the built-in dispatch defaults configured by the Rust engine.

Not serializable — Rust callers construct this at runtime and skip it in TOML/JSON configs.

proxy_provider: Option<Arc<dyn ProxyProvider>>

Optional crate::ProxyProvider for per-request proxy rotation on the reqwest HTTP path. Takes precedence over the static ProxyConfig in proxy when set. Not serializable — Rust callers inject at runtime.

Implementations

impl CrawlConfig

pub fn builder() -> CrawlConfigBuilder

Start a fluent builder for CrawlConfig. See crate::CrawlConfigBuilder.

pub fn validate(&self) -> Result<(), CrawlError>

Validate the configuration, returning an error if any values are invalid.

Trait Implementations

impl Clone for CrawlConfig

fn clone(&self) -> CrawlConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for CrawlConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for CrawlConfig

fn default() -> Self

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for CrawlConfig

where CrawlConfig: Default,

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Serialize for CrawlConfig

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.