Struct CrawlConfig 

pub struct CrawlConfig {
    pub start_url: Url,
    pub max_pages: usize,
    pub max_time: Option<Duration>,
    pub max_depth: Option<usize>,
    pub request_delay: Duration,
    pub concurrency: usize,
    pub request_timeout: Duration,
    pub user_agent: String,
    pub max_redirects: usize,
    pub respect_robots_txt: bool,
    pub allowed_patterns: Vec<String>,
    pub disallowed_patterns: Vec<String>,
}

Configuration for a crawl session.

Controls the starting URL, crawl limits, politeness settings, and URL filtering patterns. Implements Default with sensible values for most use cases.

Examples

use crawlkit_engine::CrawlConfig;
use std::time::Duration;

let config = CrawlConfig {
    max_pages: 500,
    request_delay: Duration::from_millis(200),
    concurrency: 8,
    ..Default::default()
};
assert_eq!(config.max_pages, 500);

Fields

start_url: Url

The starting URL to crawl.

max_pages: usize

Maximum number of pages to crawl.

max_time: Option<Duration>

Maximum crawl duration. None means no time limit.

max_depth: Option<usize>

Maximum crawl depth from the starting URL. None means no depth limit.

request_delay: Duration

Delay between requests to the same domain.

concurrency: usize

Number of concurrent fetchers per domain.

request_timeout: Duration

HTTP request timeout.

user_agent: String

User-Agent string to send with requests.

max_redirects: usize

Maximum number of redirects to follow.

respect_robots_txt: bool

Whether to respect robots.txt directives.

allowed_patterns: Vec<String>

Allowed URL patterns (glob-style).

disallowed_patterns: Vec<String>

Disallowed URL patterns (glob-style).

Trait Implementations

impl Clone for CrawlConfig

fn clone(&self) -> CrawlConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for CrawlConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for CrawlConfig

fn default() -> Self

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for CrawlConfig

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl From<&CrawlConfig> for HttpClientConfig

fn from(config: &CrawlConfig) -> Self

Converts to this type from the input type.

impl From<&CrawlConfig> for ScopeConfig

fn from(config: &CrawlConfig) -> Self

Converts to this type from the input type.

impl Serialize for CrawlConfig

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.