Struct HttpSourceConfig 

pub struct HttpSourceConfig {
    pub urls: Vec<String>,
    pub sitemap: Option<String>,
    pub crawl_depth: u32,
    pub allow_external: bool,
    pub request_delay_seconds: f64,
    pub respect_robots: bool,
    pub max_pages: u64,
    pub user_agent: String,
}

Fields

urls: Vec<String>

sitemap: Option<String>

crawl_depth: u32

RM-B Task 4 / Python fcbad65: Depth-bounded link crawl. 0 = current behavior (fetch only listed URLs + sitemap entries). >=1 follows that many link-hops from each seed via <a href> extraction. Capped at 5 to match Python’s ge=0, le=5 Field constraint.

allow_external: bool

By default the crawler only follows same-host links. Flip to true to follow off-host links too (still rate-limited, still subject to max_pages and respect_robots).

request_delay_seconds: f64

Minimum delay between outbound requests (per source, not per host). Default 0.5s matches Python.

respect_robots: bool

Enforce robots.txt. One fetch per host, cached. Default true.

max_pages: u64

Hard runaway cap on number of pages fetched per call. Default 1000.

user_agent: String

User-Agent header. Default matches Python’s chunkshop/0.6 (+https://…).

Trait Implementations

impl Clone for HttpSourceConfig

fn clone(&self) -> HttpSourceConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for HttpSourceConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for HttpSourceConfig

fn default() -> Self

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for HttpSourceConfig

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.